SpaceScale

SpaceScale will soon be launching the MVP of Ignite, our stateless serverless compute engine for containers. Ignite allows you to run containerized workloads, APIs, Apps, and heavy data processing tasks instantly, without managing any underlying servers. For this initial MVP, the functionality will be fully accessible via CLI and API. We are looking to get early feedback and customer input to zero in on exactly what you need. Under the hood, we run entirely on microVMs. We intentionally focused on building the invisible, hard parts you cant see: Dynamic Routing Distributed Scheduling and reconciliation High-Density Execution Lifecycle Management Because this bedrock is cleared, our foundation gives us the extensibility to build exactly what the market needs on top of Ignite whether that’s an ephemeral AI Agent Sandbox, Persistent Machines, minimal stateful services or custom workloads. We're opening up a small number of early access spots to test the runtime. If you're building something cool and want early access, my DMs are open.

Every microVM needs its own IP address.The first step was hardcoding one IP to get it working. How do we scale beyond one guest? In our latest post, I break down how we went from a hardcoded /30 subnet to a dynamic subnet allocator that is capable of 16,384 subnets per host, hands each VM its own isolated network and claws it back when the VM dies using first principles. https://lnkd.in/g4_3dNPK

At SpaceScale, every VM our scale daemon launches gets a separate per-VM workspace for the tenant with the dependencies required for boot. The guest EXT4 rootfs was included in each workspace. While this worked, I discovered it was inefficient because the same data was duplicated across multiple VM workspaces. If the rootfs is 20 MB and we have 100 active VMs, that is 2 GB of duplicated data wasted just for the root filesystem alone. Those copies also waste disk I/O, pollute page cache, and increase boot latency. The base rootfs is tiny enough to be shared by every tenant because it contains only the guest daemon. We needed something immutable and fast to achieve the density we wanted. We used EROFS (Enhanced Read-Only File System), a general purpose high-performance read-only filesystem created by a Huawei engineer to solve space and lag problems in Android phones. In our latest article, I go much deeper on how we use EROFS, build the images in CI, and use Linux hard links to let Firecracker jailer see the rootfs inside each workspace where the chroot happens. Link in the comments. #SystemsEngineering #Linux #Virtualization #CloudInfrastructure #Firecracker

SpaceScale is building something really interesting for both humans and agents to run workloads. A lot of infrastructure platforms focus on abstraction, but not enough focus on making execution understandable. The idea that runtime signals, placement, lifecycle events, and routing should all tell one coherent story feels like the right direction for modern cloud infrastructure

we just released the first version of SpaceScale website to communicate our vision, making it plain so that people can understand what we are building towards. i have gotten a lot of questions on what spacescale is and this version currently satisfies that requirement. We will keep writing about the problems we are solving, the architecture and design decisions behind the platform, and what makes our approach different, while we build towards our first design partner testing. link below https://spacescale.io

Kubernetes uses centralized bin-packing to schedule workloads. At SpaceScale, we took a different road and pushed scheduling directly to the edge. When our stateless control plane (ScaleCP) needs to deploy a microVM in a specific region, it doesn't query a massive, state database like etcd. Instead, we use a Scatter-Gather pattern over NATS.io. ScaleCP creates a temporary ephemeral inbox and broadcasts an auction to a regional NATS subject. The edge nodes receive the broadcast, calculate their own physical capacity, and reply with a bid to take the workload. Because the nodes only reply if they have space, the control plane doesn't need to run complex scheduling algorithms. It just picks the fastest bid and discards the rest. But this creates a massive distributed systems bottleneck commonly known as The Thundering Herd Problem. If we have 1,000 active nodes, a single broadcast will trigger 1,000 simultaneous bids at the exact same millisecond. That will overwhelm the network and we will be wasting CPU cycles trying to process 999 useless bids. We solved this using a Capacity Weighted Jitter, Our bare-metal edge daemon (scaleD) artificially rate-limits itself based on utilization. 90-100% Free Capacity: 0ms delay (Bids instantly) 40-70% Free Capacity: 50ms delay <40% Free Capacity: 150ms delay The emptiest node fires first. The Control Plane gets the bid and closes the auction. When the busy nodes wake up from their 150ms sleep, they see the auction is already over and silently abort. Network traffic drops by 90%. ScaleCP stays 100% stateless and lightning fast. And the fleet naturally load-balances itself without a single centralized lock. If you are building distributed systems, Always push the algorithm to the autonomous edge. #DistributedSystems #PlatformEngineering #BareMetal #Golang #NATS #SpaceScale

We started building the global control plane for SpaceScale using gRPC. It was a mistake. Our bare-metal daemon (scaleD) running on worker nodes needed a reliable way to talk to our control plane, scalecp. gRPC is fast for point-to-point, but when you are managing bare-metal worker nodes around the world, point-to-point becomes a liability. Whenever our control plane went offline during testing, nodes got stuck in retry loops. We found ourselves writing ridiculous amounts of state management and complex backoff logic just to keep the network from eating itself. gRPC also limits horizontal scaling. We would be looking at deploying complex load balancers and running a heavy, dedicated distributed key-value store like etcd. We ripped it out and are currently rewriting our entire communication layer using NATS.io , and it feels like a cheat code. With NATS, we get message streaming, a distributed KV store, and state management in a single 20MB Go binary. For a lean team, the operational simplicity was our biggest advantage. If we have 3 NATS instances in the US and 2 in Europe, and we need to add capacity to the EU cluster, we just drop in a new instance. The existing cluster gossips the entire infrastructure topology to the new instance. Our worker nodes in Europe won't waste latency crossing the Atlantic to talk to a US control plane. They just talk to the local EU NATS cluster, and NATS handles the state replication in the background. Our North Star right now is letting developers deploy stateless container image apps to microVMs with a single click. You can't do that if your own infrastructure is choking on its own complexity. We are building it right so we won't have to rewrite our infrastructure later on!

well said 👏