bug(vm): a MicroVM sandbox stopped with a full overlay disk cannot start again or be read #5149

User Story

I run coding agents in OpenShell MicroVM sandboxes on macOS and on Linux hosts with KVM. An agent filled the sandbox's own disk to 100 percent (a build wrote large files), and then the session ended and the sandbox stopped. I reproduced this on v0.1.6 on 2026-11-02 by filling the disk from inside a sandbox and stopping it (steps below).

When I started the sandbox again to collect its work, it went to Error. After that, start and stop were both refused, and I could not download anything from it. The only command left was delete, which discards the work. I first saw this with overlay_disk_mib = 2048 on macOS on the 0.1.1/0.1.2 MicroVM driver. Reproduced on v0.1.6 (2026-11-02) with overlay_disk_mib = 1024 on macOS 15.8 (Apple silicon), on three fresh sandboxes, with the same first failure each time.

Problem Statement

When the overlay disk (/dev/vdb) is full at boot, guest init cannot finish:

• sandbox start fails with ProcessExited: VM process exited with status 0 while waiting for Ready;
• the guest console shows setting up writable overlay root and then touch: cannot touch '/newroot/etc/passwd': Read-only file system (also group, shadow and gshadow);
• the sandbox is in Error with no reason that names the disk (Ready: False (ProcessExited) - VM process exited with status 0);
• on v0.1.6 sandbox start from Error is refused with "sandbox must be Stopped, Suspended, Completed, or a failed main-process Error to start (current phase: Error); stop the sandbox before retrying start". sandbox stop is accepted from Error and returns the sandbox to Stopped (my notes from 0.1.1/0.1.2 say it was refused there);
• the next sandbox start then fails differently: ProvisioningFailed: prepare guest overlay disk failed ... e2fsck failed with status exit status: 4 ... recovering journal ... Unattached inode 24 ... UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. The sandbox is back in Error, and the cycle repeats;
• sandbox download and sandbox exec need a Ready sandbox ("sandbox is not ready"), so the work cannot be read.

Nothing warns while the disk fills either: the workload gets ENOSPC only at 100 percent, and OpenShell emits no event before that.

Impact / Why This Matters

• Unsynced and unpulled work in the sandbox is lost. Delete is the only way out.
• The status gives no reason, so users do not know that the disk is the problem or that overlay_disk_mib sizes it.
• The workaround on my side is to watch df through sandbox exec and to keep a stopped sandbox running when its last pull failed, which is fragile.

Acceptance Criteria

• A MicroVM sandbox whose overlay disk is full can be started into a state where sandbox download and sandbox exec work (or it boots normally), so the work can be read.
• Its status and the start error name the cause (for example a condition reason OverlayDiskFull) and the setting that sizes the disk.
• Stop and delete work from that state.
• Optionally, an OCSF event or a status field reports overlay usage above a threshold before the disk is full.

Reproduction Steps

1. Configure a vm gateway with [openshell.drivers.vm] overlay_disk_mib = 1024 (any size works) and restart it.
2. openshell sandbox create --name full --keep --from ubuntu:24.04 -- sleep infinity
3. openshell sandbox exec full -- sh -c 'yes AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > /sandbox/fill; df -h /' until / is at 100 percent. (dd if=/dev/zero fails in the default policy with "Permission denied" on /dev/zero, so I used yes.)
4. openshell sandbox stop full
5. openshell sandbox start full
6. openshell sandbox stop full, then openshell sandbox start full again, then openshell sandbox download full /sandbox/fill .

Expected: the sandbox starts (or starts in a recovery state), and the download works.
Actual on v0.1.6: step 5 is ProcessExited, the sandbox is in Error; step 6 stop works, the second start is ProvisioningFailed (e2fsck exit 4), download answers "sandbox is not ready". Only delete works.

Control on the same gateway: a sandbox whose overlay is not full stops, starts and downloads normally.

Reproduced on v0.1.6 (2026-11-02)

Commands as above, on a stock v0.1.6 install (see Environment). Observed output, from evidence/vm-overlay-disk-full/07-run2.txt:

$ openshell sandbox exec d1 -- sh -c 'yes AAAA... > /sandbox/fill; df -h /; ls -la /sandbox/fill'
yes: standard output: No space left on device
overlay         974M  958M     0 100% /
-rw-r--r-- 1 ubuntu ubuntu 1003556864 /sandbox/fill
$ openshell sandbox exec d1 -- sh -c 'echo more > /sandbox/more.txt'
sh: 1: echo: echo: I/O error
$ openshell sandbox stop d1            -> Stopped
$ openshell sandbox start d1
Error: ProcessExited: VM process exited with status 0 while waiting for Ready     (phase Error)
$ openshell sandbox stop d1            -> Stopped
$ openshell sandbox download d1 /sandbox/fill ./dl
Error: ... "sandbox is not ready"
$ openshell sandbox start d1
Error: ProvisioningFailed: prepare guest overlay disk failed: ... e2fsck failed with status exit status: 4
  ... overlay.ext4: recovering journal / Unattached inode 24 / UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
$ openshell sandbox start d1
Error: sandbox must be Stopped, Suspended, Completed, or a failed main-process Error to start (current phase: Error); stop the sandbox before retrying start
$ openshell sandbox exec d1 -- whoami
Error: sandbox 'd1' is not ready (phase: Error); wait for it to reach Ready state

A third run (08-run3-fsck-before-start.txt) shows the overlay image is clean after the stop (e2fsck -fn and dumpe2fs: state clean, only the journal still to replay), and the first start still ends in ProcessExited. A first run on another sandbox (02-stop-start.txt, 05-c1-fsck.txt) left the overlay in "clean with errors" after the failed boot (unattached inodes 23 and 24, that is orphaned files).

Suggested UX (if applicable)

$ openshell sandbox start full
error: sandbox full did not start: its overlay disk is full (2048 MiB, [openshell.drivers.vm] overlay_disk_mib).
It is in recovery mode: download or remove files, then start it again.

Environment

• OpenShell: first seen on 0.1.1/0.1.2; re-run on stock v0.1.6 (openshell, openshell-gateway, openshell-driver-vm darwin-aarch64 release tarballs, sha256 match openshell-checksums-sha256.txt), 2026-11-02.
• Driver: vm (MicroVM), macOS 15.8 on Apple silicon, gateway on loopback with mTLS, image ubuntu:24.04, overlay_disk_mib = 1024, mem_mib = 1024. Host e2fsck is Homebrew e2fsprogs 1.47.4. RHEL 9 with KVM was not re-run.
• Guest init on main 0e96e8e: crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh setup_overlay_root (L220 onward) mounts /dev/vdb read-write and builds the overlay with no free-space check.

Logs

[0.001s] setting up writable overlay root
touch: cannot touch '/newroot/etc/passwd': Read-only file system
touch: cannot touch '/newroot/etc/group': Read-only file system
touch: cannot touch '/newroot/etc/shadow': Read-only file system
touch: cannot touch '/newroot/etc/gshadow': Read-only file system

That is the whole rootfs-console.log of the failed start on v0.1.6. The ProvisioningFailed condition on the second start:

Ready: False (ProvisioningFailed) - prepare guest overlay disk failed: code: 'The system is not in a state required for the operation's execution',
message: "/opt/homebrew/Cellar/e2fsprogs/1.47.4/sbin/e2fsck failed with status exit status: 4
stdout: .../sandboxes/<id>/overlay.ext4: recovering journal
.../sandboxes/<id>/overlay.ext4: Unattached inode 24
stderr: .../sandboxes/<id>/overlay.ext4: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

Notes for maintainers

Two possible directions, for discussion:

• keep headroom that only guest init can use (ext4 reserved blocks for root, while the workload runs as the sandbox user), so init can always finish;
• or detect the full upper layer in init, skip the workload, and bring up only the access plane (this fits feat: consistent sandbox access after the main process exits #4949).

Related: #4949 (access after the main process exits), #3804 (lifetimes). I found no existing issue for a full MicroVM overlay disk.