[Feature] MicroVM support

For Alpha, we need to support running actors using a MicroVM hypervisor, preferably without regressing on gVisor support.

Open Questions:

Should we dispatch to Kata, or directly interface with Cloud Hypervisor? Kata apparently does not currently implement the checkpoint/restore parts of the containerd shim API.
Should we revive NoVM, if we think we will need to be patching the hypervisor?
MicroVM checkpoints will be larger, since the hypervisor doesn't have a granular understanding of which pages are in use.
- One alternative is to launch MicroVMs without checkpoint/restore. The actor always boots from scratch, and we just restore a data directory (or mount it via NFS or similar protocol).

Taahir Ahmed (@ahmedtd)

Taahir Ahmed (@ahmedtd) are we going to have say a ateom-firecracker similar to ateom-gvisor? (one for every micro vm we support?)

Can we come up with a check list of here are things we need from this microvm for us to have BEFORE we consider supporting it?

Taahir Ahmed (@ahmedtd) are we going to have say a ateom-firecracker similar to ateom-gvisor? (one for every micro vm we support?)

Can we come up with a check list of here are things we need from this microvm for us to have BEFORE we consider supporting it?

This is a great question

I think generally having different ateom-* means that the actual ateom impl can be pluggable, but as you say that also means we need to come up with a clean API and checklist of features which it needs to implement. The first thing that comes to mind is snapshotting, firecracker can do it, but I don't know off the top of my head which others can. Or do we do the dreaded gvisor in microvm 😆

The pluggability may also become complicated with things like #126 and other networking related features

Taahir Ahmed (@ahmedtd) are we going to have say a ateom-firecracker similar to ateom-gvisor? (one for every micro vm we support?)

I think this is yet undecided. If we have multiple types of ateoms, then we will need to do capacity management between them. If we have one ateom that understands how to handle all backends, then capacity management is much simpler. But probably, no matter what, people are going to want the ateom interface to be pluggable.

One other dimension that's still undecided is how much communication goes through atelet, vs how much is direct from ate-api-server to ateoms.

Like, a very clean ateom interface might just be to have the current atelet Run/Checkpoint/Restore workload RPCs surfaced on ateom. And ateom is responsible for everything, including image pulling (probably still sharing the cache in the a shared host folder).

MicroVM checkpoints will be larger, since the hypervisor doesn't have a granular understanding of which pages are in use.
One alternative is to launch MicroVMs without checkpoint/restore. The actor always boots from scratch, and we just restore a data directory (or mount it via NFS or similar protocol).

This is a very interesting idea, and especially prudent for agents which store things in the fileystem anyway.

Should we dispatch to Kata, or directly interface with Cloud Hypervisor? Kata apparently does not currently implement the checkpoint/restore parts of the containerd shim API.

This actually brings up a core question for me. Do we want the core "workload" semantic here to be containers. The current actor template re-uses the k8s-api Container shape, but is that out of expediency, or because containers are the end-state of these workloads? Happy to try and elaborate further if that doesn't quite make sense

In one of my original proof-of-concepts with Cloud Hypervisor, I just punted on container support and had the user specify a qcow disk image that supported cloud-init as their application.

I think we probably want the container to be how we specify the application, just so we're not breaking totally new ground. Individual backends can then repack the container image into a different format (qcow or whatever) if they need.

Using the K8s shape is mostly just because it's likely well thought through.

Makes sense to me. I'm more familiar with the firecracker side of the house, but from what I've seen it's mostly the same. The main question that I've had when thinking about this is multiple containers and what the boundary is between them within the VM, if there is any. However, maybe that's up to the implementation here.

Do we want the core "workload" semantic here to be containers.

There's a decent ecosystem around OCI images, but I have less direct experience with Firecracker.

If a developer of an agent wants to create an ActorTemplate using a microVM, do they want to specify just an app which runs in that VM (in which case OCI is AOK) or do they need to specify a boot disk with a kernel and other things? In other words - who owns the kernel version of the VM?

Let me clarify a bit, my issue is not with the shape of an OCI image, or a container style API, it is rather the number of fields in the k8s Container struct. There are a lot of kubernetesisms in there which may or may not be immediately relevant to a VM workload.

However, the ActorTemplate already uses a paired down version of it, so as long as we are careful/intentional about what gets added I think this is the correct shape.

Also in the future other microvm implementations could use different OCI specs to download the specific filesystem artifacts as has been mentioned before.

There are a lot of kubernetesisms in there which may or may not be immediately relevant to a VM workload

Oh, for sure. In fact, we need to dial down the kubernetes-isms which are exposed. If we expose everything we will be forced to implement all of it eventually. If we expose a subset, we can argue each feature on its own.

My question regarding uVM packaging remains -- who decides the kernel version? The person who installs the ateoms (workerpool) or the developer of the agent itself (actortemplate)? Or both?

However, the ActorTemplate already uses a paired down version of it, so as long as we are careful/intentional about what gets added I think this is the correct shape.

Yes. We should borrow a subset of fields and expand as needed for these types.

My question regarding uVM packaging remains -- who decides the kernel version? The person who installs the ateoms (workerpool) or the developer of the agent itself (actortemplate)? Or both?

For gVisor the actors are aware of the runsc version because of compatibility of the snapshots. I think the same issue will dictate versioning this. A typical VMM snapshot will include the kernel state so it kinda has to be associated with the actor snapshots at least ...

For gVisor the actors are aware of the runsc version because of compatibility of the snapshots. I think the same issue will dictate versioning this. A typical VMM snapshot will include the kernel state so it kinda has to be associated with the actor snapshots at least ...

Yes. Depending on the precise path we take for MicroVM support, Substrate will need to ship either

(If we repack OCI images to root filesystems): A linux kernel vmlinux image and an init binary/script
(If we run a standardized root filesystem, and use docker within the guest for container support): A linux kernel vmlinux, a root filesystem image, and maybe a cloud-init handoff volume.

All of these need to be kept constant in order to properly restore snapshots, which means that natural place to keep them is pinned in the ActorTemplate.

Alternatively, we can stamp them out into the individual actor database entries, if we don't want the user to be responsible for specifying them.

Yes. Depending on the precise path we take for MicroVM support, Substrate will need to ship either

(If we repack OCI images to root filesystems): A linux kernel vmlinux image and an init binary/script

(If we run a standardized root filesystem, and use docker within the guest for container support): A linux kernel vmlinux, a root filesystem image, and maybe a cloud-init handoff volume.

I personally prefer the former, the latter is A LOT of indirection for not very much value. Especially when we can already snapshot the VM state itself. If we want the heavily optimized snapshotting of gvisor than we can just use gvisor.

All of these need to be kept constant in order to properly restore snapshots, which means that natural place to keep them is pinned in the ActorTemplate.

Alternatively, we can stamp them out into the individual actor database entries, if we don't want the user to be responsible for specifying them.

I feel strongly that this should not be exposed directly to the user in the ActorTemplate spec. Stamping it in the status can be ok, but requiring configuration of runsc, or a hypervisor impl in every ActorTemplate alongside actual actor config feels like a muddying of the experience.

Also, if you want you can assign this to me as well, I will be POCing/working on a firecracker implementation. The biggest user facing choice that needs to be answered ASAP is how these different backends are exposed via the ActorTemplate API itself. I know there was brief talk about it in #121, but we need to come up with a solution there. An explicit oneof provides a slightly clearer API experience, but less extensibility and more difficult adding of implementations in the future. In other words the classic tradeoff.

For my POC I will probably bypass this hard question and just work on the tech itself, but definitely interested in reaching consensus there.

Eitan Yarmush (@EItanya)

We've been discussing this in slack (incl Eitan Yarmush (@EItanya) Davanum Srinivas (@dims) and others).

I feel strongly that this should not be exposed directly to the user in the ActorTemplate spec. Stamping it in the status can be ok, but requiring configuration of runsc, or a hypervisor impl in every ActorTemplate alongside actual actor config feels like a muddying of the experience.

Agreed, we resolved this via #253 across sandboxes -- decoupled the sandbox asset configuration from the actortemplate to the workerpool(s) and store it alongside the snapshot so it is pinned but ActorTemplate authors don't have to deal with this.

TLDR on the current direction:

Cloud-hypervisor > firecracker for this purpose. They have the same rust VMM crate from crosVM, but cloud-hypervisor is a Linux Foundation project with support for e.g. device passthrough which we will want later. Recent CHV has some excellent optimizations we can leverage and performance is pretty close.
Partial kata adoption (at least to start), kata doesn't support snapshot / restore, but we can still leverage at minimum the guest kernel / agent bits. We're also not stuck with that long-term though just the rough shape. The guest agent is cleaner than say, cloud-init ...
We need sparse file support in the backends for good performance. ondemand vs copy restore mode in CHV has some real tradeoffs but the biggest one is on repeated roundtrip copy mode will "densify" the memfd ... so ondemand + sparse support is tablestakes for avoiding massive snapshots when VMs are over-provisioned (As they probably will be often ...)

I built around a dozen variations on this ... fairly satisfied with that shape, sending PRs today (currently rebasing and retesting locally).

One tiny prefactor: #285

One outstanding problem ... CI coverage (no official support for nested virt in hosted GHA runners), will have to investigate that one still.
We'd like to avoid standing up a bunch of infrastructure at this stage ... but we're going to have to be really careful with PRs if we don't get it running in CI. I've booted VMs in GHA before but ... not nested (a hack using linux VMs on the old intel macOS runners).

feat: add SandboxClass support to ActorTemplate (#285)

microvm MVP cleanup: align rootfs behavior to gvisor, ... (#313)

Navigation Menu

Repository navigation

[Feature] MicroVM support #123

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature] MicroVM support #123

Description

Activity

dims commented on May 29, 2026

EItanya commented on May 29, 2026

ahmedtd commented on May 29, 2026

EItanya commented on May 29, 2026

ahmedtd commented on May 29, 2026

EItanya commented on May 29, 2026

thockin commented on May 31, 2026

EItanya commented on Jun 1, 2026

thockin commented on Jun 1, 2026

BenTheElder commented on Jun 1, 2026

ahmedtd commented on Jun 1, 2026

EItanya commented on Jun 3, 2026

EItanya commented on Jun 3, 2026

BenTheElder commented on Jun 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions