msb: host ssh-agent forwarding breaks after a host reboot — stale create-time --vsock route survives resume/reattach (distinct from #392) #413

Summary

On the msb backend, after the host machine was rebooted (the sandbox was
not cleanly shut down first), resuming the existing sandbox and re-attaching left
the guest unable to reach the host ssh-agent, so the agent session could not
sign commits — despite SSH_AUTH_SOCK being correctly set on the host with the
signing key loaded.

This is a distinct failure from GSA-TTS/agentic-coding-quickstart#388 /
GSA-TTS/agentic-coding-quickstart#392. #392 fixed the SSH_AUTH_SOCK
env-var injection on running-reattach (restart the in-guest socat bridge + write
the /var/lib/acq/ssh-auth-sock marker). That fix works: the marker is present
and SSH_AUTH_SOCK is exported in the guest. What is broken here is one layer
deeper — the --vsock route's host endpoint is stale after a host reboot, so
the bridge connects to a dead host-side socket.

Environment / sequence on the host

Host: macOS. Host ssh-agent healthy with the signing key loaded:

$ ssh-add -l
256 SHA256:97q8H+... /Users/<user>/.ssh/id_ed25519 (ED25519)
256 SHA256:ruhDy...  <user>@gsa.gov (ED25519)

Sequence after the host reboot (unclean sandbox shutdown):

acq start opencode-agentic-coding
acq run opencode --name opencode-agentic-coding \
    ./agentic-coding-quickstart ./agentic-coding-patterns ./agentic-coding-playbook \
    --kit ./agentic-coding-patterns/integrations/isolation/acq-kits/paseo

Symptom (inside the guest)

$ export SSH_AUTH_SOCK=/home/agent/.acq/ssh-agent.sock   # marker is correct; var is set
$ ssh-add -l
error fetching identities: communication with agent failed

Committing therefore fails to sign ([git-ssh-sign] no SSH agent - cannot sign commits), the same operator-visible symptom as §34 in
docs/KNOWN_FAILURE_MODES.md, but from a different cause.

Root cause (diagnosed live in the guest)

The guest's socat bridge cannot reach the host over vsock:

$ socat -T2 - VSOCK-CONNECT:2:3552 </dev/null
W open("/dev/vsock", ...): Permission denied
E connect(5, AF=40 cid:2 port:3552, 16): Connection reset by peer

Connection reset by peer on the vsock connect means the host side of the
route has no live listener for that vsock port anymore.

The chain:

1. The msb --vsock HOST_PATH:3552/stream route is emitted only at create
   time in acq_backend_provision (acq.backends/msb.sh:2642-2644 →
   _acq_msb_vsock_flags_into). HOST_PATH is the host's SSH_AUTH_SOCK path
   captured at that moment (acq_host_socket_forwards in
   acq.backends/common.sh). It is persisted in the sandbox config and is
   never re-derived on msb start or on reattach.
2. Host reboot → the ssh-agent is restarted and gets a new SSH_AUTH_SOCK
   socket path; the host-side vsock listener bound to the old create-time
   host_socket path is gone. acq start → msb start merely resumes the VM
   with the persisted (now-stale) route.
3. acq run … --name opencode-agentic-coding re-attaches to the now-running
   sandbox → acq_backend_ensure_kits_applied calls
   _acq_msb_ensure_ssh_agent_forward (the fix(msb): re-drive ssh-agent forward on running reattach #392 heal). It sees the create-time
   --vsock route present (_acq_msb_has_ssh_agent_vsock_route → true) and
   dutifully restarts the in-guest socat bridge and writes the marker — but
   the route's host end is dead, so the bridge just fails at connect time.
4. Result: marker present, SSH_AUTH_SOCK exported, socat running — yet
   ssh-add -l → "communication with agent failed", and signing fails.

Why #392 does not cover this: #392's contract is "re-inject the env var and
re-drive the guest bridge on running-reattach." It explicitly requires and
trusts the persisted create-time route. It has no path to notice that the route's
host endpoint became invalid across a reboot, and no path to refresh it. The
route is create-time-only by construction, so today the only real remedy is a
recreate.

Impact

• Silent-ish: the operator sees signing fail and (as in this session) works
  around it by manually exporting SSH_AUTH_SOCK — which does not help here,
  because the failure is below the env var. There is no clear message pointing at
  "stale vsock route after reboot; recreate the sandbox."
• Any msb sandbox that outlives a host reboot loses agent forwarding until it is
  recreated, even after the fix(msb): re-drive ssh-agent forward on running reattach #392 fix.

Suggested directions (for discussion in a follow-up ADR/PR)

1. Detect and report the dead route: after (re)starting the bridge on
   reattach/start, probe the vsock endpoint (or verify ssh-add -l succeeds
   through the guest sock) and, on failure, emit an actionable message —
   "host ssh-agent forwarding is stale after a host reboot; recreate the sandbox
   (acq rm + acq run …) to refresh the route" — instead of leaving a silent
   dead bridge behind a present marker. (Fail-closed / no-silent-failure, per
   AGENTS.md.)
2. Refresh the route on start/reattach if msb gains a way to update a
   --vsock route on an existing sandbox (today it is create-time only). If/when
   supported, acq_backend_start / the heal could re-derive HOST_PATH from the
   current SSH_AUTH_SOCK and re-point the route, making agent forwarding
   survive a reboot without a recreate. Needs an msb capability check +
   upstream confirmation.
3. Document the reboot case explicitly in
   docs/KNOWN_FAILURE_MODES.md §34 (currently only the < 0.6.9 / missing-socat
   causes and the fix(msb): re-drive ssh-agent forward on running reattach #392 running-reattach env-var variant are covered; add a
   "stale vsock route after a host reboot" variant with the VSOCK-CONNECT
Connection reset by peer fingerprint and the recreate remedy).

Refs

• GSA-TTS/agentic-coding-quickstart#392 (running-reattach env-var re-inject — related but does not fix this)
• GSA-TTS/agentic-coding-quickstart#388 (live-verify of the fix(msb): re-drive ssh-agent forward on running reattach #392 re-drive — closed)
• ADR-0021 (docs/adr/0021-msb-host-ssh-agent-forwarding-via-vsock.md) — "Re-attach to a running sandbox"
• docs/KNOWN_FAILURE_MODES.md §34 ("Git signing fails on msb")

AI-assisted (OpenCode). Human-owned and reviewed per repo AI-contribution policy.