Repository navigation
Description
Not #1330 — different layer
I wrote the retracted comment on #1330. That comment proposed a relay data-path truncation theory; I retracted it after showing the published-port relay transfers bytes completely and repeatably, and that my ERR_INCOMPLETE_CHUNKED_ENCODING was a client codec artifact.
This is a different layer: process lifecycle, not the data path. Nothing here claims the relay corrupts or truncates anything. When this bug fires the relay works perfectly — it relays to a VM generation that no longer exists. Also distinct from #914 (throughput), #745 (accept queue), #1705 (FIN/EOF), and #1201 (guest-side zombie reaping).
Summary
A create-time -p HOST:GUEST publish binds its host listener inside the runtime process (msb machine). I observed a runtime process that still held 127.0.0.1:6767 (LISTEN) after the sandbox it served had been stopped, removed, and recreated — 7 days older than the VM then running under that name, and from a different msb installation than the one driving the commands. Only kill <pid> released it.
Because the port stayed bound, the replacement sandbox's own forwarder could not take it, and host clients kept reaching the old daemon. Both were healthy, so every signal looked fine; the symptom was an application behaving like a different machine.
Environment
- Host: macOS 15 (arm64), Homebrew
- Leaked process:
microsandbox 0.7.2 - Active install driving the lifecycle commands:
microsandbox-acq 0.6.18(our version-pinned rebuild of the same upstream release artifact) - Guest workload: a daemon on
0.0.0.0:6767, published at create time with-p 6767:6767 - No
MSB_HOMEoverride — verified; both installs resolve the default~/.microsandbox
Observed
Two msb processes from different installation prefixes, concurrently:
leaked PID 75399 /opt/homebrew/Cellar/microsandbox/0.7.2/libexec/msb machine …
healthy PID 19966 /opt/homebrew/Cellar/microsandbox-acq/0.6.18/libexec/msb sandbo…
The leaked one held the port:
$ lsof -nP -iTCP:6767 -sTCP:LISTEN
COMMAND PID USER FD TYPE ... NAME
msb 75399 ... 45u IPv4 ... TCP 127.0.0.1:6767 (LISTEN)Start time 2026-09-23T23:42:03Z — days before the VM then serving that name (guest /proc/uptime well under a day).
Restarting restarted the guest daemon but not the host listener:
guest daemon pid: 990 -> 698 (restarted)
host listener pid: 75399 -> 75399 (unchanged)
host-visible identity: (unchanged)
Stop and remove both left it bound. Only this worked:
$ kill 75399
$ lsof -nP -iTCP:6767 -sTCP:LISTEN # empty
$ # recreate the sandbox; its own forwarder binds, host and guest now agreePPID was 1, which on macOS is expected (launchd adopts it) and is not a leak signal. The discriminating signals were age (listener older than the VM it serves) and build (listener prefix ≠ active install).
Evidence boundary — please read before reproducing
I drove the lifecycle through a wrapper (acq) that shells out to msb; I did not type msb stop / msb rm myself, and I have not re-run this as a from-scratch bare-msb reproduction. The process observations above are measured. The reproduction below is proposed. I am happy to run any sequence you specify on the affected host.
Where (reading main @ 553e9e8)
1. The listener's lifetime is the runtime process's lifetime.
crates/network/lib/engine/ports/publisher.rs:440PortPublisher::spawn_listeners→:624tcp_listener_task→:604bind_listener(SocketAddr::new(port.host_bind, port.host_port), backlog)- constructed from the netstack poll loop,
crates/network/lib/engine/netstack/poll.rs:286 - which runs under
msb machine, whose doc comment says it never returns: "This command never returns — the VMM calls_exit()on guest shutdown" (crates/cli/lib/machine_cmd.rs:4)
Ending the listener therefore requires ending that process.
2. stop never signals the runtime PID.
stop_sandbox → request_stop_owned (sdk/rust/lib/backend/local/sandbox/mod.rs:299, :309) does not call kill_pid. It calls request_agent_shutdown (:561) and marks the row Draining. Shutdown travels as a MessageType::Shutdown over the name-addressed agent socket, and the code comments that the transport is name-addressed and identity is re-verified before sending.
So a graceful stop depends on the guest agent receiving a message and the VMM then exiting of its own accord. Only two paths signal PIDs directly:
kill_sandbox(:342) — SIGKILL, frommsb killstop_sandbox_for_replacement(sdk/rust/lib/backend/local/sandbox/create.rs:1817) — SIGTERM then SIGKILL, oncreate(replace: true)
Both take the PID from load_active_run(...).pid filtered by pid_is_alive. Both are therefore only as good as the run row.
3. After rm, the PID is unrecorded anywhere.
crates/db/lib/entity/run.rs:93 declares on_delete = "Cascade" from run to sandbox. remove_local_persisted_sandbox (sdk/rust/lib/sandbox/mod.rs:1799) deletes the sandbox row — so the run rows, and with them the only recorded PID of the runtime that owns the listener, are deleted too.
This is the part I find most concerning, and it is independent of the foreign-install detail. Once a rm completes while a runtime process is still alive, nothing in the system knows that PID any more. There is no later reconciliation that can find it, because the record it would need is gone. lsof is the only way to discover it, and kill the only way to clear it.
4. What I can not explain, stated plainly.
remove_local_persisted_sandbox requires status Stopped/Crashed before deleting, and reconcile_sandbox_runtime_state_owned (:717) returns early while the recorded PID is alive — so on the documented path a rm should have been refused while PID 75399 lived. It was not. Two candidate explanations, neither of which I have confirmed:
- the
runrow never named 75399 (sopid_is_alivechecked a different or absent PID), or - the row was already terminal for an unrelated reason while the process survived.
I would rather hand you the gap than paper over it. I am not claiming a specific code defect in the rm guard — only that the end state is reachable, was reached, and is unrecoverable through the CLI.
What the different installation prefixes do and do not explain
The two installs did not resolve different databases. crates/utils/lib/lib.rs:169 resolve_home() honours MSB_HOME and otherwise returns ~/.microsandbox; our wrapper sets no MSB_HOME (verified: zero occurrences across the wrapper and its backends). Both builds shared one database. The foreign install explains why the binary differs, not why the row lookup failed — so the cascade-delete observation above is the load-bearing one.
Prior art in-tree — this class is already recognized, on Windows only
sdk/rust/lib/sandbox/reap.rs:36 implements reap_leaked_runtime_process with an identity-checked LeakedReapVerdict (NoProcess / RecycledPid / Unverifiable), documented as handling "a stopped database row can still have a … VM process serving the sandbox's named pipes" and "older Windows runtimes did not own a lifecycle lock and can outlive their terminal DB row." It is gated #[cfg(windows)] (sdk/rust/lib/sandbox/mod.rs:1753).
My failure is the same shape — a runtime outliving its terminal row — on Unix, where that reaper does not run. I am not claiming that removing the gate is the fix: it reads the PID from the run row, which in my case is exactly what is missing.
Related: #1642 (signals a recorded PID without verifying it is still the runtime) is the same family from the other direction — signalling the wrong process vs never finding the right one. #1558 is the closest precedent in shape: a host-side resource leaking into an unrelated supervisor, surviving host-process restart, releasable only by killing the holder.
One rebuttal pre-empted
bind_listener sets SO_REUSEADDR on Unix (publisher.rs:614). That does not help here — SO_REUSEADDR does not permit binding over a socket in LISTEN, so the replacement forwarder genuinely cannot take the port while the old one lives.
How it is reachable in ordinary use
Switching the host's msb version while a sandbox has a published port. Replacing the active binary does not stop runtimes already running from the outgoing build.
On Homebrew this is not exotic, because of #1672: the tap carries a single always-latest formula, so anyone needing a specific version ends up with a second, separately-prefixed keg. Both own bin/msb and conflict, so a "switch" is a real swap of which binary is active.
Strength of this attribution: circumstantial. The leaked runtime's start time precedes our version-switching work by about 1h43m. I am labelling it the likely trigger, not a proven one. The load-bearing facts remain the two installation prefixes and the cascade-deleted run rows.
Proposed reproduction (not yet run bare)
The variant I would try first, because it isolates the mechanism from the two-installs detail and needs only one install:
msb run -d --name repro -p 6767:6767 <image> …; confirmlsof -nP -iTCP:6767 -sTCP:LISTENshows anmsb machine.kill -9the runtime process, or otherwise prevent the guest agent from acting on shutdown, so the row is not updated by the graceful path.msb stop repro && msb rm repro.- Re-check
lsof, and check whether anyrunrow naming that PID still exists.
Expected: either the rm is refused while the process lives, or the process is reaped. Reported behaviour: a listener survives with no record of its owner.
Then the two-install variant, matching what I actually hit:
- Install two msb versions under distinct prefixes so
bin/msbcan be swapped. - With A active, create the sandbox with a published port; confirm the A-prefixed listener.
- Make B active. Do not touch the sandbox.
msb stop repro && msb rm repro; re-checklsof.
Impact
- Silent misrouting. Both daemons are healthy; only an application-level identity reveals the split. Every restart appears to succeed.
- Unrecoverable through the CLI. After
rm, the owning PID is unrecorded;lsof+killis the only route. - Lifecycle commands report success while the port stays bound to a dead generation.
- For a sandbox used as a security boundary, a host client reaching an out-of-boundary generation is not traversing the intended interception/egress path at all. In my case that surfaced two layers away as a provider rejecting a credential, which cost real time to trace back to routing.
What would have helped
- Do not lose the PID. Whatever the reaping policy, the record needed to find a surviving runtime should outlive the sandbox row — or
rmshould refuse while a published host bind is still held. - Make lifecycle failure visible. On
stop/rm, if the host bind is stillLISTENing afterwards, say so and name the holding PID. A warning would have ended this immediately. - Identity-check the port, not just the PID. Before declaring a published port released, confirm nothing holds the host bind.
- Consider a Unix path for the leaked-runtime reaper, with an identity check that does not rest on a
runrow that may be gone. ARecycledPid/Unverifiableverdict that reports rather than kills would already beat silence. - Distinguish declared config from live binding in
inspect/ps. A consumer reading.active_config.network.ports[]gets the create-time mapping, which stays correct-looking while the bind belongs to another process. I have filed the wrapper-side half of that against our own tool; the underlying distinction is only available from msb. - Document the version-switch hazard (pairs with Homebrew tap ships only an always-latest formula, so users cannot install or hold a specific version #1672).
Happy to run any diagnostic you want on the affected host.
GSA-TTS engineer. Investigation AI-assisted (OpenCode); every source reference above was human-verified against main @ 553e9e8, and claims I could not verify are marked as such.
Activity