Skip to content

Commit 1062244

Browse files
authored
fix(paseo): verify runtime identity without false orphan verdicts (#567)
## Summary Fix the host-sleep false orphan verdict and unsafe process-kill guidance described in #492. - Compare live WebSocket daemon IDs on the published IPv4 port and inside the guest, rather than trusting a guest-writable file. - Treat host elapsed-time/guest-uptime disagreement as advisory when runtime IDs agree; missing or mismatched required IDs still fail. - Select the mapping for the configured guest port, validate bounded WebSocket responses, and reject partial or ambiguous probe output. - Remove generic kill-PID instructions and automatic full-argv collection; update troubleshooting and the decision record. ## Validation - 87/87 Paseo Node tests passed in a network-disabled, read-only, non-root container, including whole-script verdict scenarios and protocol-aware socket fixtures. - Ubuntu pre-commit hooks and `make validate` passed on exact tree `0435704c408b1107a643fb3f91a78226e0f07308`. - Normal host commit hooks passed after resolving local validator/linter dependencies; normal push hook scanned all nine commits without leaks. - Live disposable MSB sandbox using the pinned kit and Paseo 0.13.0: host published IPv4 and guest runtime IDs agreed; gate reported 7 passed, 0 failed, 0 warnings. This run exposed and corrected the actual nested session/status server_info schema; the regression now uses that schema and split TCP delivery. - Post-sleep age mismatch is covered by offline verdict fixtures; this live run did not force a new host sleep. - Independent agent reviewer verified all four changed files against the exact Git base/head and found no blocking QA/security findings. This is supporting evidence, not a GitHub CODEOWNER approval. ## Scope and limitations The gate certifies the reported published-IPv4/guest runtime ID agreement, not HTTP/UI readiness, all loopback aliases, or cryptographic protection against an actively impersonating local listener. No changes to production sandbox lifecycle or permissions. ## AI assistance OpenCode assisted implementation, tests, live verification, and review preparation. Commits are signed and include co-author attribution. Human author eligibility, release-rights attestation, and independent applicable CODEOWNER review remain for human confirmation. ## Related issue Relates to #492. Please review acceptance before closing the issue.
1 parent cf6d7a5 commit 1062244

4 files changed

Lines changed: 536 additions & 163 deletions

File tree

‎integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md‎

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,9 @@ acq ports <sandbox> # is container 6767 mapped to a host p
2020
acq exec <sandbox> -- sh -c 'curl -fsS http://127.0.0.1:6767/api/health && echo OK'
2121
```
2222

23-
- `/api/health` returns 200 and the port is mapped → open
24-
`http://127.0.0.1:<host-port-for-6767>`.
23+
- `/api/health` returns 200 and the port is mapped → run
24+
`scripts/paseo-verify-identity <sandbox>` to check that the published IPv4
25+
endpoint reaches the guest daemon before opening that IPv4 URL.
2526
- `/api/health` fails → the daemon isn't up yet. On **first boot** it installs the
2627
Paseo CLI first (a few seconds). Watch the logs:
2728

@@ -52,36 +53,31 @@ still carries the placeholder, so the provider receives a well-formed but invali
5253
credential. The auth error is therefore a *symptom of misrouting*, not a bad key —
5354
chasing the key wastes the most time on this failure.
5455

55-
**Diagnosis — compare the two identities.** A Paseo daemon has a stable id in
56-
`~/.paseo/server-id`. Ask the host-published port what it is, then ask the guest
57-
directly. They must match:
56+
**Diagnosis — compare two live daemon identities.** Probe `server_info` over
57+
`/ws` on the published IPv4 port and from inside the guest. Their runtime IDs
58+
must match:
5859

5960
```bash
6061
scripts/paseo-verify-identity <sandbox>
6162
```
6263

63-
That check fails loudly on a mismatch and prints the remediation. To do it by
64-
hand, the guest side is:
65-
66-
```bash
67-
acq exec <sandbox> -- sh -lc 'cat ~/.paseo/server-id; cat ~/.paseo/paseo.pid'
68-
```
69-
70-
and the host side is a `hello` frame over the published port's `/ws`, reading
71-
`serverId` from the `server_info` reply.
64+
The script performs both probes and reports a mismatch. The guest-writable
65+
`~/.paseo/server-id` file is only diagnostic; reading it does not prove which
66+
daemon is serving the guest port. The `localhost` and IPv4-mapped aliases are
67+
advisory observations, not the published IPv4 identity verdict.
7268

7369
**Cause — a host forwarder that outlived the sandbox it was created for.** The
7470
create-time `-p HOST:GUEST` publish runs a host-side `msb` process. That process
7571
can **survive `acq stop` *and* `acq rm`** and keep holding the host port. The
7672
replacement sandbox's forwarder then cannot bind, while `acq ports` still reports
7773
the mapping as configured — so every restart appears to succeed and changes
78-
nothing. Two independent tells:
74+
nothing. Two diagnostic clues, neither of which proves an orphan on its own:
7975

80-
- **Age.** The listener is *older than the sandbox it serves*. Compare its
81-
`ELAPSED` against guest uptime:
76+
- **Age.** Compare listener `ELAPSED` against guest uptime. Host sleep can make
77+
these clocks diverge for a healthy sandbox; check live IDs before acting:
8278

8379
```bash
84-
ps -ww -o pid,ppid,lstart,etime,command -p <listener-pid>
80+
ps -ww -o pid,ppid,lstart,etime,comm -p <listener-pid>
8581
acq exec <sandbox> -- sh -c 'cut -d. -f1 /proc/uptime'
8682
```
8783

@@ -94,15 +90,16 @@ nothing. Two independent tells:
9490

9591
```bash
9692
lsof -nP -iTCP:<host-port> -sTCP:LISTEN # note the PID
97-
ps -ww -o command= -p <listener-pid> # -ww: do not truncate the path
93+
ps -ww -o comm= -p <listener-pid> # executable only, not secret-bearing argv
9894
command -v msb # compare installation prefixes
9995
```
10096

10197
Do not rely on a truncated `ps` line — the version segment is often exactly
10298
what gets cut off.
10399

104100
> A `PPID` of `1` is **normal** here: launchd adopts the forwarder. It is not by
105-
> itself evidence of a leak. Age and build are the discriminating signals.
101+
> itself evidence of a leak. Age and build inform diagnosis; live runtime IDs
102+
> determine whether the published IPv4 endpoint reaches the guest daemon.
106103
107104
**How you get one.** Switching the host's `msb` version is the known trigger:
108105
replacing the active binary does not stop forwarders from the outgoing build, and
@@ -111,28 +108,31 @@ because upstream's `microsandbox` and the pinned `microsandbox-acq` both own
111108
upgraded, downgraded, or pinned `msb` while a sandbox had a published port, this
112109
is the failure to check for first.
113110

114-
**Fix.** Kill the stale forwarder by PID, confirm the port is genuinely free,
115-
then restart so the sandbox can bind its own:
111+
**Fix.** First confirm the listener does **not** belong to the sandbox being
112+
checked. Elapsed host process time can exceed guest uptime after a host sleep;
113+
matching WebSocket server IDs mean an age discrepancy is not an orphan verdict.
114+
Never kill an `msb machine --name <sandbox>` process for that sandbox: it is the
115+
VM itself. If identities differ and ownership confirms a stale forwarder from
116+
another sandbox/build, use that owner's supported lifecycle command first.
117+
Verify the port is free before restarting the affected sandbox:
116118

117119
```bash
118-
kill <listener-pid>
119120
lsof -nP -iTCP:<host-port> -sTCP:LISTEN # must print NOTHING before continuing
120-
# only if it survived SIGTERM:
121-
kill -9 <listener-pid>
122-
123121
acq stop <sandbox> && acq start <sandbox>
124122
scripts/paseo-verify-identity <sandbox>
125123
```
126124

127-
If the listener's `PPID` is **not** 1, do not `kill -9` blindly — it may have
128-
siblings under a supervisor. Investigate the parent first.
125+
If the listener survives the confirmed owner's normal shutdown, investigate
126+
the process tree before taking any process-level action; PPID 1 alone does not
127+
prove orphan status.
129128

130129
> **`paseo daemon restart` will not fix this.** Elsewhere in this guide that is
131130
> the right way to bounce the daemon, but here the stale component is the
132131
> **host-side forwarder**, not the guest daemon. Restarting the daemon alone was
133132
> observed to change nothing: the daemon PID changed while the host listener kept
134-
> the port and kept serving the old backend. The sequence above is required —
135-
> kill the forwarder, then restart the **sandbox** so it can bind a new one.
133+
> the port and kept serving the old backend. Release the port through the
134+
> confirmed owner's supported lifecycle first; any process-level action needs
135+
> separate ownership verification before restarting the affected sandbox.
136136
137137
Success is the two ids **agreeing with each other**, never matching a remembered
138138
value: the id changes whenever the VM is recreated.

‎integrations/isolation/acq-kits/paseo/docs/decisions/identity-gate-stale-host-forwarder.md‎

Lines changed: 23 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -82,17 +82,18 @@ or downgrades `msb` while a sandbox has a published port), not only by accident.
8282

8383
## Decision
8484

85-
**Add an identity gate: `scripts/paseo-verify-identity <sandbox>`.** It asserts
86-
that the host-published port and `acq exec` resolve to the **same**
87-
`~/.paseo/server-id`, and exits non-zero when they do not. It is read-only and
88-
makes no changes.
85+
**Add an identity gate: `scripts/paseo-verify-identity <sandbox>`.** It compares
86+
the live WebSocket `server_info` IDs from the published IPv4 endpoint and the
87+
guest daemon accessed through `acq exec`. The guest-writable `server-id` file is
88+
diagnostic only. It is read-only and makes no changes.
8989

9090
Design constraints, each learned from a concrete failure:
9191

9292
- **Compare the ids to each other, never to a remembered value.** The id changes
9393
whenever the VM is recreated, so any hardcoded expectation rots immediately.
94-
The gate also fails when the three host probes disagree *among themselves*,
95-
which means multiple backends behind one port.
94+
Only the published IPv4 mapping and guest runtime determine the identity
95+
verdict. `localhost` and IPv4-mapped probes are advisory: a name can reach a
96+
different address family and listener.
9697
- **Resolve the host port from `acq ports`; never assume host == guest.** The
9798
post-hoc publish path maps a different host port, and a probe hardcoded to
9899
`6767` would silently test the wrong endpoint. Derived hex for `/proc/net/tcp`
@@ -131,11 +132,9 @@ explains what to check rather than asserting a specific failure mode.
131132

132133
- A wrong-daemon route is caught in one command, before it presents as an
133134
authentication error two layers away from its cause.
134-
- The gate diagnoses but does **not** remediate: it prints the `kill`/restart
135-
sequence and requires a human to run it. Killing a process by PID is not a
136-
safe automatic action — if the listener is *not* launchd-adopted it may have
137-
siblings under a supervisor, so the instructions say to investigate the parent
138-
before `kill -9`.
135+
- The gate diagnoses but does **not** remediate. It does not recommend killing a
136+
listener by PID: the listener can be the current sandbox's `msb machine` VM.
137+
Confirm ownership and use the owner's supported lifecycle before remediation.
139138
- The gate cannot distinguish "UI not yet opened" from "host traffic landing
140139
elsewhere" on its own; the `"peer":"external"` vs `"peer":"loopback"` client
141140
counts in `daemon.log` are reported as a **warning** with both readings
@@ -163,15 +162,19 @@ defects, all fixed and re-verified:
163162

164163
Predicate tables (post-fix):
165164

166-
- Orphan age: observed healthy (age 1001s vs uptime 1000s) → PASS; the real
167-
orphan (7 days vs 1 day) → FAIL; slack boundary 1120 → PASS, 1121 → FAIL.
165+
- Historical orphan-age predicate: the original test classified 7 days vs 1
166+
day as FAIL and the slack boundary at 1120/1121. Host sleep later showed that
167+
guest uptime can pause while host process elapsed time advances. The current
168+
gate warns on age discrepancy when live host/guest IDs agree and fails if
169+
identity is not established.
168170
- Loopback ordering: observed `127.0.0.1 ::1` + IPv4 → silent; `::1 127.0.0.1` +
169171
IPv4 → note; `::1` + IPv6 → silent.
170172
- Build compare: the two observed real paths (`microsandbox-acq/0.6.18` vs
171173
`microsandbox/0.7.2`) → warns; same prefix → PASS; non-Cellar path → PASS (no
172174
false warning when no prefix is extractable).
173-
- Verdict matrix: ids agree → pass; mismatch → fail; two distinct host ids →
174-
fail; no host id → fail.
175+
- Historical verdict matrix compared file/host IDs and all host aliases. The
176+
current gate compares live published IPv4 and guest runtime IDs; an absent or
177+
mismatched required ID fails, while alias differences are advisory.
175178
- `bash -n` clean.
176179

177180
A fourth defect was found and fixed in the standalone precursor: `mktemp -t`
@@ -216,11 +219,13 @@ bind [::1]:<port> while the IPv4 forwarder holds *:<port> → both bind, no co
216219
That is the same wrong-daemon class this gate exists to catch, reachable by name
217220
resolution instead of by a stale forwarder, and reproducible on demand.
218221

219-
**End-to-end confirmation of the underlying fix** (separate from the gate):
222+
**Historical end-to-end confirmation of the underlying fix** (separate from the current gate):
220223
after killing the stale forwarder, `"peer":"external"` client counts went from
221224
**0 → 110**, and a live agent round-trip in the browser UI succeeded. Per the
222-
periodic end-to-end validation discipline, the gate's own PASS path is verified
223-
against a live sandbox, not a mock.
225+
periodic end-to-end validation discipline, the earlier gate's PASS path was
226+
verified against a live sandbox. The revised runtime-to-runtime gate requires a
227+
separate live check at its own revision; the historical observation does not
228+
certify the new protocol parser.
224229

225230
## Links
226231

0 commit comments

Comments
 (0)