Skip to content

Commit 6cc4700

Browse files
committed
fix(acq-kits): harden openchamber wrapper failure path per #234 review
Address the review must-fixes on the PID-1 hold. All are failure-path hardening, not a redesign. 1. Self-heal the hold (Medium, must-fix). The old `started_here` branch did a bare `wait`; if the server it launched died (crash/OOM/lost race), `wait` returned, hold_pid1 returned, and PID 1 exited -> sandbox stopped — the exact bug this wrapper prevents. Replace the asymmetric two-branch hold with ONE self-healing loop: poll the server and re-launch it if it ever stops answering, so a dead server can never let the loop return. 2. Close the serve-launch TOCTOU (Medium). Bring-up detected only via `curl /health`, so two near-simultaneous no-arg starts both `nohup … serve` and raced the :4096 bind (loser dies). Add an atomic mkdir lock (serve_launch_once) with a PID file and stale-lock reclaim (dead holder -> lock reclaimed), so exactly one wrapper launches. 3. Move the serve log off a predictable /tmp path (Low). Now ~/.local/state/openchamber/opencode-serve.log in a 0700 dir; docs/verify updated to read it there. 4. Security note: state the loopback-only guarantee depends on the adapter publishing loopback-scoped, which is acq's publishedPorts translation (quickstart#221). Also update publishedPorts docs now that quickstart#221 is MERGED: acq carries backend_extras.sbx.publishedPorts, so ports 3000/4096 publish at create time with no manual step (older acq still gets the documented fallback). verify's step 2b now re-publishes idempotently (works on old and new acq). scripts/verify: add step 11 — a live self-heal regression guard that brings the server up, KILLs it, and asserts a server answers again (hold_pid1 re-launched; PID 1 held). Verified: shellcheck-clean parses (sh -n / bash -n); offline simulations of the lock (8 racers -> 1 launch), stale-lock reclaim, and the self-heal loop (server death -> reheal, loop never returns); markdownlint 0 errors. Co-authored-by: OpenCode Agent <[email protected]>
1 parent 1bcbce9 commit 6cc4700

6 files changed

Lines changed: 204 additions & 96 deletions

File tree

‎integrations/isolation/acq-kits/openchamber/README.md‎

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -74,19 +74,20 @@ explicitly (see [Usage](#usage)).
7474

7575
| Backend | Support | Notes |
7676
|---------|---------|-------|
77-
| **sbx** | Supported (ports need one manual step) | The startup script runs as a background hook. Port publishing is **not** carried by acq's neutral→sbx translation, so publish the two container ports once after create: `acq ports <sandbox> --publish 3000:3000` and `--publish 4096:4096`. `backend_extras.sbx.publishedPorts` documents the intended mapping and is honored by a direct sbx-v2 apply. |
77+
| **sbx** | Supported | The startup script runs as a background hook, and acq's neutral→sbx translation carries `backend_extras.sbx.publishedPorts` ([quickstart#221](https://github.com/GSA-TTS/agentic-coding-quickstart/pull/221), merged), so the two container ports are published to the host at create time — no manual step. |
7878
| **msb** | Not yet | Deferred: the neutral spec models neither port publishing nor a background-command flag, and no equivalent `backend_extras.msb` is wired. The install+supervise script and the `opencode` wrapper are backend-agnostic; only the port/background plumbing is missing. |
7979
| **ppp** (later) | Not yet | Same gap as msb. |
8080

8181
The one backend-specific dependency is **exposing the in-container ports to the
8282
host** and **running the startup command in the background**. The background hook
83-
is carried through acq's translation; **port publishing** was added to acq's
84-
translator upstream ([quickstart#221](https://github.com/GSA-TTS/agentic-coding-quickstart/pull/221)),
85-
but until that fix reaches the pinned patterns kit ref it is a manual
86-
`acq ports … --publish` step. Both live in `backend_extras.sbx` because
87-
`hybrid/v1` does not (yet) model published ports or a background flag. Extending
88-
`openchamber` to `msb`/`ppp` — and adopting the upstream `publishedPorts`
89-
translation (dropping the manual step) — is tracked as a follow-up on
83+
is carried through acq's translation; **port publishing** is now carried too, as
84+
of acq's translator fix ([quickstart#221](https://github.com/GSA-TTS/agentic-coding-quickstart/pull/221),
85+
merged) — so a current `acq` publishes ports 3000/4096 at create time with no
86+
manual step. (If you run an older `acq` that predates that fix, publish them once
87+
per sandbox with `acq ports <sandbox> --publish 3000:3000` / `--publish
88+
4096:4096`.) Both live in `backend_extras.sbx` because `hybrid/v1` does not (yet)
89+
model published ports or a background flag. Extending `openchamber` to
90+
`msb`/`ppp` is tracked as a follow-up on
9091
[#233](https://github.com/GSA-TTS/agentic-coding-patterns/issues/233);
9192
it needs either a neutral port-publish + background vocabulary in `acq` or an
9293
`msb`-native equivalent extra.
@@ -109,17 +110,18 @@ commit SHA of this repo (branches and tags are rejected for git kit refs).
109110
Because `acq run opencode` runs the bare command `opencode`, the kit's wrapper
110111
takes over: it starts the shared server and offers you a TUI in that terminal.
111112

112-
> **Publish the ports after create.** `acq`'s neutral→sbx translation does **not**
113-
> carry `backend_extras.sbx.publishedPorts`, so creating with this kit does not by
114-
> itself map the container ports to the host. Publish them once, per sandbox:
113+
> **Ports are published at create time.** A current `acq` carries the kit's
114+
> `backend_extras.sbx.publishedPorts` through translation
115+
> ([quickstart#221](https://github.com/GSA-TTS/agentic-coding-quickstart/pull/221),
116+
> merged), so both container ports are mapped to the host **loopback** on create.
117+
> If you're on an older `acq` that predates that fix, map them once per sandbox:
115118
>
116119
> ```bash
117120
> acq ports <sandbox> --publish 3000:3000 # OpenChamber web UI
118121
> acq ports <sandbox> --publish 4096:4096 # shared opencode server
119122
> ```
120123
>
121-
> (`publishedPorts` in `spec.yaml` documents the intended mapping and is honored
122-
> by a direct sbx-v2 apply, but not by the current acq translator — see
124+
> (`publishedPorts` in `spec.yaml` is the source of that mapping — see
123125
> [backend parity](#backend-parity).)
124126
>
125127
> **On HTTPS-inspected networks (e.g. Zscaler).** OpenChamber's first-boot
@@ -183,9 +185,18 @@ shared server is now published to the host (container port `4096` → a host
183185
loopback-only, so they are not LAN-reachable; but anyone with access to the
184186
host's loopback (any local user, or anything you forward those ports to) can
185187
drive OpenCode without a credential. Only run this on a host you trust as
186-
single-tenant, and don't forward the mapped ports to a wider interface. The
187-
server is started on demand by the wrapper, so it is only live once you've run
188-
`opencode` — not for the whole sandbox lifetime unless you start it.
188+
single-tenant, and don't forward the mapped ports to a wider interface.
189+
190+
**The loopback-only guarantee depends on the acq port-publish translation.** The
191+
"host **loopback** only" claim holds only because the backend publishes the
192+
`0.0.0.0:4096` container bind to a loopback-scoped host port. That mapping is
193+
carried by acq's `backend_extras.sbx.publishedPorts` translation, added in
194+
[quickstart#221](https://github.com/GSA-TTS/agentic-coding-quickstart/pull/221)
195+
(merged). On an **older `acq`** that predates that fix, the ports are **not**
196+
auto-published — you must publish them yourself (`acq ports <sandbox> --publish
197+
4096:4096`), and acq maps published ports to `127.0.0.1` per sandbox. Until the
198+
ports are published loopback-scoped, treat the unauthenticated `0.0.0.0` bind as
199+
reachable to whatever can route to the container, and don't expose it.
189200
190201
**Installer supply chain.** The first-boot install fetches OpenChamber's
191202
`install.sh` from a **pinned release tag** (`OPENCHAMBER_REF`) and refuses to run

‎integrations/isolation/acq-kits/openchamber/TROUBLESHOOTING.md‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,11 @@ still `exec`s the attach, update the kit and recreate the sandbox.
5858
**Symptoms:** `acq ports <sandbox>` shows no mapping for
5959
container port 3000 (or 4096), or the browser can't connect.
6060

61-
**Cause:** `acq`'s neutral→sbx kit translation does **not** carry the kit's
62-
`backend_extras.sbx.publishedPorts`, so creating the sandbox with this kit does
63-
not by itself publish the ports.
61+
**Cause:** on a **current `acq`** the ports are published automatically at create
62+
time (acq carries the kit's `backend_extras.sbx.publishedPorts` as of
63+
[quickstart#221](https://github.com/GSA-TTS/agentic-coding-quickstart/pull/221),
64+
merged). If you see no mapping, you are most likely on an **older `acq`** that
65+
predates that fix, so it did not publish the ports.
6466

6567
**Fix:** publish the two container ports (once per sandbox):
6668

@@ -70,8 +72,7 @@ acq ports <sandbox> --publish 4096:4096 # shared opencode server
7072
acq ports <sandbox> # confirm the mappings
7173
```
7274

73-
If you applied a *pre-translated* sbx-v2 kit directly via `sbx --kit`, the
74-
`publishedPorts` are honored at create time and this manual step isn't needed.
75+
Upgrading `acq` to a build that includes quickstart#221 removes the manual step.
7576

7677
## OpenChamber loads but shows no server
7778

@@ -110,21 +111,22 @@ one-time npm install), or the install failed.
110111
```bash
111112
acq exec <sandbox> -- sh -c 'cat /tmp/openchamber-install.log' # first-boot install
112113
acq exec <sandbox> -- sh -c 'cat /tmp/openchamber.log'
113-
acq exec <sandbox> -- sh -c 'cat /tmp/opencode-serve.log' # written by the wrapper
114+
acq exec <sandbox> -- sh -c 'cat ~/.local/state/openchamber/opencode-serve.log' # written by the wrapper
114115
acq exec <sandbox> -- sh -c 'openchamber status'
115116
```
116117

117-
If the shared server is failing (see `/tmp/opencode-serve.log`), confirm a
118+
If the shared server is failing (see
119+
`~/.local/state/openchamber/opencode-serve.log`), confirm a
118120
provider is configured — pair this kit with the `usai-provider` kit (the default
119121
GSA setup does). OpenChamber runs under a respawn loop, so a transient OpenChamber
120-
failure self-heals within a few seconds; the shared server is not supervised (the
121-
wrapper starts one instance), so if it exits, re-run `opencode`. To reproduce a
122+
failure self-heals within a few seconds; the shared server is held by the
123+
wrapper's `hold_pid1` loop, which re-launches it if it dies. To reproduce a
122124
single run by hand (mirrors what the wrapper and `files/home/openchamber-start.sh`
123125
do):
124126

125127
```bash
126128
acq exec <sandbox> -- sh -lc '
127-
nohup opencode serve --hostname 0.0.0.0 --port 4096 >/tmp/opencode-serve.log 2>&1 &
129+
nohup opencode serve --hostname 0.0.0.0 --port 4096 >~/.local/state/openchamber/opencode-serve.log 2>&1 &
128130
OPENCODE_SKIP_START=true OPENCODE_PORT=4096 OPENCHAMBER_ALLOW_UNAUTHENTICATED_LAN=true \
129131
openchamber --lan --port 3000 >/tmp/openchamber.log 2>&1 &
130132
'

‎integrations/isolation/acq-kits/openchamber/docs/decisions/wrapper-entrypoint-owns-server.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -113,14 +113,16 @@ Two related choices:
113113
(`command -v opencode`).
114114
- **Ports need a manual publish step under acq (transitional).** acq's
115115
neutral→sbx translator (`kit_translate_to_sbx`) did not carry
116-
`backend_extras.sbx.publishedPorts`, so applying via `acq` did not auto-map
117-
3000/4096 to the host. Fixed upstream in
116+
- **Ports published at create (as of quickstart#221).** acq's neutral→sbx
117+
translator (`kit_translate_to_sbx`) now carries
118+
`backend_extras.sbx.publishedPorts`, so applying via `acq` auto-maps 3000/4096
119+
to the host loopback at create time. Fixed in
118120
[quickstart#221](https://github.com/GSA-TTS/agentic-coding-quickstart/pull/221)
119-
(closes quickstart#219/#220); until that reaches the pinned patterns kit ref
120-
the workaround is a one-time `acq ports <sandbox> --publish 3000:3000` /
121-
`--publish 4096:4096`. The verify script does this automatically on the
122-
`RUN_ACQ=1` path, and the README / TROUBLESHOOTING document it. Adopting the
123-
fix here is tracked as #233.
121+
(merged; closes quickstart#219/#220). On an older `acq` predating that fix, the
122+
workaround is a one-time `acq ports <sandbox> --publish 3000:3000` /
123+
`--publish 4096:4096`. The verify script re-publishes idempotently on the
124+
`RUN_ACQ=1` path (working on both old and new acq), and the README /
125+
TROUBLESHOOTING document the fallback.
124126
- The `kind: sandbox` variant contemplated by the older ADR is no longer needed
125127
for shared sessions — the mixin achieves it via the wrapper.
126128

‎integrations/isolation/acq-kits/openchamber/files/home/.local/bin/opencode‎

Lines changed: 93 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,10 @@
2929
# entrypoint / PID 1 is alive. If this wrapper backgrounds the server and then
3030
# returns — OR execs a TUI that you later quit — PID 1 exits and the sandbox
3131
# transitions running -> stopped. So on the no-arg path the wrapper always ends
32-
# by BLOCKING on a long-lived foreground server (hold_pid1): the TUI, when
33-
# requested, runs as a CHILD, and quitting it falls through to that hold rather
34-
# than ending the entrypoint.
32+
# by BLOCKING FOREVER in hold_pid1: a self-healing loop that keeps a server
33+
# answering on :4096 and re-launches it if it ever dies (crash/OOM/lost race),
34+
# so a dead server can NEVER let the loop return and stop the sandbox. The TUI,
35+
# when requested, runs as a CHILD; quitting it falls through to that same hold.
3536
#
3637
# The managed server is UNSECURED (no OPENCODE_SERVER_PASSWORD): the sandbox — an
3738
# ephemeral container with proxied, allow-listed egress and no host FS — is the
@@ -42,7 +43,19 @@ set -eu
4243

4344
OC_PORT=4096
4445
OC_HOST=0.0.0.0
45-
SERVE_LOG=/tmp/opencode-serve.log
46+
47+
# Sandbox-local state dir for our log + serve lock. Prefer an XDG-ish per-user
48+
# location over a predictable /tmp path (the log can echo host-connect details).
49+
# It is created 0700 and lives for the sandbox's lifetime.
50+
OC_STATE_DIR="${XDG_STATE_HOME:-${HOME:-/home/agent}/.local/state}/openchamber"
51+
mkdir -p "$OC_STATE_DIR" 2>/dev/null || OC_STATE_DIR="$(mktemp -d 2>/dev/null || echo /tmp)"
52+
chmod 0700 "$OC_STATE_DIR" 2>/dev/null || true
53+
SERVE_LOG="$OC_STATE_DIR/opencode-serve.log"
54+
# Lock dir guarding the serve launch (mkdir is atomic, so it doubles as a
55+
# mutex): only one wrapper wins the race to `nohup … serve`, closing the
56+
# two-starts-both-bind-:4096 TOCTOU. Held only across launch, not for the
57+
# server's lifetime.
58+
SERVE_LOCK="$OC_STATE_DIR/serve.lock"
4659

4760
# --- Resolve the REAL opencode (never ourselves). ---------------------------
4861
# We are ~/.local/bin/opencode and are first on PATH, so `command -v opencode`
@@ -83,25 +96,66 @@ serve_up() {
8396
curl -fsS "http://127.0.0.1:$OC_PORT/global/health" >/dev/null 2>&1
8497
}
8598

86-
# Start the shared server in the background and wait for it to answer, so the
87-
# host-connect instructions and any TUI attach below are truthful. This is only
88-
# a TEMPORARY background start: on every no-TUI path we replace it by exec'ing a
89-
# foreground server so PID 1 stays alive (see the header "must not return" note).
90-
started_here=0
91-
if serve_up; then
92-
echo "opencode: shared server already running on 0.0.0.0:$OC_PORT"
93-
else
94-
echo "opencode: starting shared server on $OC_HOST:$OC_PORT ..."
95-
nohup "$real_opencode" serve --hostname "$OC_HOST" --port "$OC_PORT" \
96-
>"$SERVE_LOG" 2>&1 &
97-
started_here=1
99+
# Launch a backgrounded `opencode serve` exactly once across racing wrappers.
100+
# `mkdir` is atomic, so it is our mutex: only the wrapper that creates SERVE_LOCK
101+
# launches; the others skip the launch and just wait for the port to answer.
102+
# This closes the TOCTOU where two near-simultaneous no-arg starts both
103+
# `nohup … serve` and race the :4096 bind (the loser dies). We release the lock
104+
# once the server answers (or we give up), so a later respawn can re-acquire it.
105+
#
106+
# Stale-lock safety: if a lock holder was killed before it could `rmdir`, the
107+
# lock would otherwise wedge every future launch into the wait-only branch. We
108+
# record the holder PID and reclaim the lock if that PID is gone. `set -e` never
109+
# fires here — all the checks are in conditions or `|| true`.
110+
# Returns 0 if a server is answering by the end, 1 otherwise.
111+
_serve_wait() {
98112
_i=0
99113
while [ "$_i" -lt 30 ]; do
100-
serve_up && break
114+
serve_up && return 0
101115
_i=$((_i + 1))
102116
sleep 1
103117
done
104-
if serve_up; then
118+
serve_up
119+
}
120+
serve_launch_once() {
121+
_own=0
122+
if mkdir "$SERVE_LOCK" 2>/dev/null; then
123+
_own=1
124+
else
125+
# Lock exists — reclaim it if the recorded holder PID is gone (stale),
126+
# so a crashed launch can't wedge us forever.
127+
_holder="$(cat "$SERVE_LOCK/pid" 2>/dev/null || true)"
128+
if [ -n "$_holder" ] && ! kill -0 "$_holder" 2>/dev/null; then
129+
rm -rf "$SERVE_LOCK" 2>/dev/null || true
130+
if mkdir "$SERVE_LOCK" 2>/dev/null; then _own=1; fi
131+
fi
132+
fi
133+
134+
if [ "$_own" -eq 1 ]; then
135+
printf '%s' "$$" >"$SERVE_LOCK/pid" 2>/dev/null || true
136+
# Re-check under the lock in case a server came up between the caller's
137+
# check and our acquiring the lock.
138+
if ! serve_up; then
139+
echo "opencode: starting shared server on $OC_HOST:$OC_PORT ..."
140+
nohup "$real_opencode" serve --hostname "$OC_HOST" --port "$OC_PORT" \
141+
>>"$SERVE_LOG" 2>&1 &
142+
fi
143+
_serve_wait
144+
rm -rf "$SERVE_LOCK" 2>/dev/null || true
145+
else
146+
# Another (live) wrapper owns the launch; just wait for the port to answer.
147+
_serve_wait
148+
fi
149+
serve_up
150+
}
151+
152+
# Bring the server up now, so the host-connect instructions and any TUI attach
153+
# below are truthful. This first start is TEMPORARY: on every no-TUI path we end
154+
# by holding a foreground server so PID 1 stays alive (see the header note).
155+
if serve_up; then
156+
echo "opencode: shared server already running on 0.0.0.0:$OC_PORT"
157+
else
158+
if serve_launch_once; then
105159
echo "opencode: shared server is up on $OC_HOST:$OC_PORT"
106160
else
107161
echo "opencode: server did not answer yet; see $SERVE_LOG" >&2
@@ -147,11 +201,11 @@ if [ -t 0 ] && [ -t 1 ]; then
147201
esac
148202
fi
149203

150-
# --- Keep the sandbox alive by holding the FOREGROUND server as PID 1. --------
204+
# --- Keep the sandbox alive by holding a FOREGROUND server as PID 1. ----------
151205
# Reached on every no-arg path: declined prompt, non-interactive, OR after an
152206
# attached TUI is quit. We are the sandbox entrypoint; if we returned here PID 1
153-
# would exit and the sandbox would stop (the running->stopped bug). So block on
154-
# a long-lived foreground process instead of returning.
207+
# would exit and the sandbox would stop (the running->stopped bug). So loop
208+
# forever holding a live server in the foreground — and SELF-HEAL if it dies.
155209
cat <<EOF
156210
The shared server runs in the FOREGROUND as the sandbox's main process (so the
157211
sandbox stays running). Attach a TUI to the SAME session any time:
@@ -163,29 +217,25 @@ sandbox stays running). Attach a TUI to the SAME session any time:
163217
Server log: $SERVE_LOG
164218
EOF
165219

220+
# hold_pid1 — never returns; keeps a server answering on :$OC_PORT as long as
221+
# the sandbox lives. This is a single self-healing loop (no started_here fork):
222+
# whether we launched the server or another process owns it, if it ever stops
223+
# answering we (re)launch one and keep going. Crucially, we NEVER let the loop
224+
# fall through to the end of the script — a returning hold_pid1 would exit PID 1
225+
# and stop the sandbox, the exact failure this wrapper prevents.
166226
hold_pid1() {
167-
if [ "$started_here" -eq 1 ]; then
168-
# A background server we launched is already bound to :$OC_PORT. Don't start
169-
# a second one (the port is taken); keep PID 1 alive by waiting on that
170-
# background job. `wait` blocks in the foreground until it exits, so the
171-
# entrypoint stays open for the sandbox's lifetime.
172-
wait
173-
else
174-
# The server was already up before this wrapper ran (e.g. OpenChamber's
175-
# skip-start supervisor, or an earlier wrapper invocation, owns :$OC_PORT).
176-
# We must NOT start a second server on the same port (it would fail to bind
177-
# and, under `set -e`, kill this entrypoint — reintroducing the stop bug).
178-
# Instead, hold PID 1 open by parking on a harmless foreground loop that
179-
# keeps re-checking the server is alive. If the pre-existing server ever
180-
# disappears, we exit the loop and become the foreground server ourselves.
181-
echo "opencode: holding sandbox open; shared server already owns :$OC_PORT" >>"$SERVE_LOG"
182-
while serve_up; do
183-
sleep 30
184-
done
185-
echo "opencode: shared server on :$OC_PORT went away; taking over in foreground" >>"$SERVE_LOG"
186-
exec "$real_opencode" serve --hostname "$OC_HOST" --port "$OC_PORT" \
187-
>>"$SERVE_LOG" 2>&1
188-
fi
227+
while :; do
228+
if serve_up; then
229+
# Something owns :$OC_PORT. Park briefly and re-check; if it dies, the
230+
# next iteration re-launches. (We can't `wait` on a process we may not
231+
# own — a pre-existing server or one from a racing wrapper — so poll.)
232+
sleep 5
233+
continue
234+
fi
235+
# No server answering — (re)launch one under the lock, then loop to confirm.
236+
echo "opencode: no server on :$OC_PORT; (re)starting in foreground-hold at $(date -u +%FT%TZ)" >>"$SERVE_LOG"
237+
serve_launch_once || true
238+
done
189239
}
190240

191241
hold_pid1

0 commit comments

Comments
 (0)