Skip to content

Commit ef6c5a9

Browse files
authored
fix(stop): typed stop outcome so a history-only failure does not abort the update (lidge-jun#3008) (lidge-jun#3118)
* fix(update): tell a history-only stop failure from a real one Closes lidge-jun#3008. handleStop sets a failure code AFTER history restoration - that is, after the proxy and service are already down - so a failed Codex-history cleanup was indistinguishable from a proxy that refused to die. The update aborted with the service stopped, no listener, and the old package still installed. Its own history warning was unreachable, sitting behind the guard that never let control get there. restoreSharedClientStateAfterStop now reports the two failure kinds separately, reading the artifact states it already collects: a config or catalog failure removes state a client depends on and is a real teardown failure, while a history failure leaves the runtime consistent with a manifest retained for review. The signal has to cross a spawnSync boundary, so it is an exit code rather than a type. 79 sits above the sysexits block, below 128+signal, and outside every code this CLI already emits - 0, 1 and 130 from src/cli/index.ts, plus 2, 4 and 64 from dispatch. It lives in a plain-ESM module because bin/ocx.mjs cannot import TypeScript, and inlining the number twice is how the two ends drift. Both updaters decode it. Fixing only the Bun path would have left the reported lane broken: the dashboard npm update runs through the Node launcher, which carries its own independent guard. * fix(update): probe the endpoint, and never call a failed service stop history-only Review found four ways the classification was still too generous. stopServiceIfInstalled collapses "no service installed" and "a service refused to stop" into the same false, and handleStop treated both as fine. A manager that would not stop can respawn the proxy, so a later history failure emitted the proceed code while a supervisor was still live. stopServiceIfInstalledDetailed distinguishes absent/stopped/failed, and only absent or stopped may reach the history-only path. Both updaters gated only on exit status and PID/runtime files. The plan said plainly that absent records are weak evidence - a crashed-but-listening proxy leaves none - and required an identity probe. Both lanes now ask the captured endpoint before replacing files. The Node launcher gets a synchronous probe because runNpmSelfUpdate is not async; it speaks node:http rather than fetch, since a child spawned from a blocked event loop can have its fetch aborted before dispatch and report the same "not live" as a dead port. It fails open: a probe that cannot answer must not block an update on its own uncertainty. The claimed launcher regression was vacuous - it spawned a stub and checked the stub's own exit code, never touching either updater. It is replaced by one that evaluates the shared decision expression over the whole status domain, and by a real probe test against an out-of-process listener. tests/grok-lifecycle.test.ts was already red against this change and I had not run it. It now asserts the {historyOnly, other} contract. Also corrects a comment naming restart and the tray coordinator as handleStop callers; both go through handleProxyRestart. * fix(update): stop guessing when the liveness probe cannot answer Third review round, and the probe I added was the problem. It failed open: a timeout, a spawn failure, or a body it did not recognize all read as "not live". The reviewer reproduced a listener that accepts connections and withholds /healthz, which the probe called dead - and that is precisely the state where replacing package files is most dangerous. It also rejected the legacy OpenCodex health body that src/server/proxy-liveness.ts accepts. The probe is now tri-state. Only a refused connection or a definitive non-OpenCodex answer earns "dead"; a timeout, an unparseable body, an unexpected error code, or a child that produced nothing is "unknown". Both lanes abort on anything other than "dead". The TypeScript lane keeps proxyIdentityAt and adds the probe, because a null from that helper covers refusal and timeout alike. The npm lane also probed the wrong address: it captured only the port and assumed 127.0.0.1, so a proxy bound to ::1 or a specific interface answered nobody. The hostname now travels with the port from the runtime record or config. And the Windows service stop still could not report failure. A schtasks query that threw was read as absence, and stopWindows swallows a non-benign /end failure so the caller could never see it. It now uses the existing tri-state probeWindowsSchedulerTask - unknown counts as failed - and a checked stop that surfaces the result. * fix(update): one post-stop decision, and an identity rule that matches Fourth review round, and the probe's own classification was wrong twice. Its "is this OpenCodex" test accepted any body carrying pid, ok, version or status, so a foreign server exposing /healthz read as our proxy; and it classified every parsed non-200 as dead, so our own 500 read as absence. It now mirrors isOpencodexHealthz exactly - canonical service field, or the pre-identity shape with status/version/uptime - and any status other than 200 is unknown, because an endpoint answering without identifying itself is not evidence of absence. The npm lane still probed the wrong address in two cases: a legacy runtime record with a port and no hostname never consulted config, and a bracketed IPv6 literal was passed through as a host. Port and hostname now resolve independently, and wildcard and bracketed forms normalize. The decision itself moves into src/update/stop-decision.mjs, which both updaters import. The reviewer's point stands: a test of a locally reimplemented predicate stays green while either lane drifts, and one lane drifting is exactly how this shipped. The matrix now exercises the real function - clean stop, history-only, five other statuses plus a signal kill, surviving runtime state, live proxy, and unknown liveness - and asserts neither lane keeps a private copy of the rule. * fix(update): verify the respawn window, and normalize the host once Fifth review round closed the last two. Host normalization was per-lane, so the npm launcher got it and the TypeScript updater passed a bracketed IPv6 literal straight to node:http. That answers nothing, reads as "unknown", and aborts a healthy update - leaving the service down, which is the failure shape this issue is about. It moves into probeProxyLiveness, so both lanes get it from one place: wildcard binds dial loopback, bracketed literals are unwrapped, empty falls back. And a stopped Windows scheduler was being trusted as a stopped proxy. killWindowsSchedulerWrappers is explicitly best-effort and the :loop wrapper respawns its child after about five seconds, so an updater probing immediately can see the dead interval and start replacing files just before the proxy returns. handleStop now polls proxyStillLiveAfterStop with canRespawn across that window whenever a scheduler was stopped. A survivor is an ordinary failure, not history-only, and it also blocks shared teardown - restoring client config while the proxy runs leaves both pointing at each other. The reviewer scoped that one into this phase rather than a follow-up, and that is right: without it exit 79 can authorize a replacement without proven service shutdown, which is the whole property the code exists to establish. * fix(service): charge the respawn wait only to the backend that respawns Sixth review round: I gated the restart-window poll on "a service stopped" and passed canRespawn unconditionally, so launchd, systemd and WinSW each paid a seven-second wait on every ocx stop. That contradicts the helper's own contract and would be a regression in ordinary use. Only Task Scheduler needs it: schtasks /end ends the task instance while the cmd :loop wrapper survives and respawns its child seconds later (lidge-jun#764). The other backends are down when they report stopped. ServiceStopOutcome gains "stopped-respawnable" so the capability travels with the outcome instead of being re-derived at the call site, and handleStop waits only for that value. stopServiceIfInstalled keeps its boolean meaning by accepting both stopped forms. * fix(stop): defer shared teardown until the proxy is proven down Seventh review round found the respawn guard arriving too late to matter. handleStop calls stopProxy first, which POSTs /api/stop, and that handler already restores native Codex and strips the Grok fence. So by the time the scheduler verification ran, a surviving wrapper had ALREADY lost its client config - and ownershipBlocked could only prevent the parent from doing it a second time. POST /api/stop takes deferSharedTeardown=1, and ocx stop sends it. The proxy still drains and exits; it just leaves shared config alone, and this process restores it after confirming no survivor. A direct /api/stop caller sends nothing and keeps the self-contained behaviour it has today. handleUninstall is unchanged for the same reason. Also drops the "won't respawn" half of the service-stopped message: a stopped Task Scheduler can still respawn through its wrapper, which is precisely what the verification below it settles. * fix(stop): make the deferred teardown a durable obligation Seventh review round found the deferral was a bare query flag, which cannot express an obligation. Two windows followed from that. First, ownership was unprovable. Any authenticated caller could append deferSharedTeardown=1 and exit, and the proxy would go down leaving native Codex and the Grok fence pointed at a proxy that no longer exists. Nothing on disk recorded that a restore was still owed. Second, the legitimate parent had the same hole. If `ocx stop` died after the child exited but before restoreSharedClientStateAfterStop(), the config stayed routed at a dead proxy - a window the child-owned flow never had. So the deferral is now a receipt. `ocx stop` claims pending-teardown.json before asking for a deferred stop, the route honours the flag only when it can see that receipt, and the claim is cleared only after this process has restored the shared config itself. A receipt whose owner is dead is an abandoned obligation: the next `ocx stop` finishes it, but only on the path that already proved no proxy is live. The deferred response no longer claims "native Codex restored" - it says the teardown was deferred, and carries sharedTeardown: deferred. The teardown itself moved to src/server/stop-teardown.ts. The route schedules process.exit 200ms after answering, so the inline version could not be called from a test, which is why the previous round's regression could only read source text. tests/stop-deferred-teardown.test.ts now calls the real functions: the URL the graceful-stop client builds, the restores that do or do not run, the wording of each response, and the receipt's owner guard. * fix(stop): bind the deferral to a receipt identity, not to presence Eighth review round found five ways the receipt was still not an obligation. The clear was the worst of them, and it was reproducible: recovery re-read the file after restoring and passed THAT receipt's owner pid to the clear. A concurrent stop writing its own receipt in between meant this run deleted a live obligation. The receipt now carries a nonce, the snapshot is taken before this run claims anything, and a clear names the exact receipt it read. The same presence-versus-identity error made the deferral itself unowned. Any authenticated caller could set deferSharedTeardown=1 while somebody else's receipt happened to be on disk, get the deferral, and exit owning no recovery. The request now carries the nonce and the route requires an exact match. Abandoned recovery could also run under a proxy that was merely unresponsive: findLiveProxy collapses a timeout and a malformed answer into the same null as a refused connection, and on a stop with no scheduler nothing else re-checked. It is now gated on the tri-state probe answering a definitive "dead". Neither updater woke on the receipt. After a parent crash the service, pid and runtime records can all be absent while the shared config still points at a proxy that is gone, so a retrying update installed straight over the pending recovery. Both gates now include it. And a corrupt receipt read as absence, which discarded the one fact recovery needs: an obligation is outstanding and its owner can no longer be identified. The read is now missing/valid/invalid; invalid is outstanding, is recoverable, and clears only through an explicit force by a caller that just discharged it. * fix(stop): gate the restore on the inherited obligation, not just label it Ninth review round found the previous round's gate was not one. It computed inheritedRecoverable, warned when the probe could not confirm the proxy down, and then fell into the same restore anyway - so a live-but-unresponsive proxy still had its client config torn out. The restore is now behind restoreBlocked, there is exactly one call site, and an unconfirmed inherited obligation fails the stop and preserves the receipt instead of discharging it. The recovery probe was also asking the wrong endpoint. A crashed owner leaves no runtime-port record, so the probe fell back to the configured port, which is not where a proxy started with an explicit --port is listening: it refuses, recovery reads that as dead, and the live proxy loses its config. The receipt now carries the endpoint its owner was stopping, and an obligation that cannot name one is never recoverable. Two identity holes remained. Force-clearing an unparseable receipt recreated the concurrent-clear race the nonce had just closed - against a valid receipt this time - so an invalid read now carries a hash of the bytes it read and clears only against those exact bytes. And every read error mapped to "missing", including a directory sitting where the receipt belongs, which hid an outstanding obligation; only ENOENT is absence now. tests/update-stop-first.test.ts was red against the previous commit's updater gates and is updated. The new wiring test asserts the gate blocks rather than warns: one restore call site, reached only through restoreBlocked, with the failure path preserving the receipt. * fix(stop): put the nonce in the filename so a clear cannot race Tenth review round found the receipt was still one shared file, so clearing it was read-compare-unlink across three syscalls: a concurrent stop replacing the file between the compare and the unlink meant this run deleted an obligation it never owned. Locking would serialize that; naming removes it. Each claim now lives at pending-teardown-<nonce>.json, so unlink names one specific obligation and cannot reach another. Two concurrent stops hold two receipts, which is the truth of the situation, and handleStop recovers over the whole set. An unreadable receipt was worse than useless. It names no endpoint, so nothing can prove its proxy down, so it could never be discharged - while both updater gates treated it as a reason to run the stop that would fail on it every time. That is an update that can never proceed. Such a receipt is now quarantined under a name the scan ignores: the evidence is kept for the operator, the restore it stood for happens, and if it cannot even be moved the stop says so and fails rather than pretending. The claim could also silently not happen. Both stop paths derived the endpoint separately from stopProxy, so a proxy with no runtime record was hard-killed with no receipt at all - the parent-crash window, reopened on the one path where the stop is least graceful. There is now a single resolved stop target feeding both the receipt and the request, and stopProxyGracefully uses that snapshot rather than re-reading. Two smaller ones: a failed unlink was swallowed, so a receipt surviving its own discharge would re-trigger recovery forever; it is reported now. And a receipt whose body names a different nonce than its filename is invalid, so an edited body cannot claim an identity the name does not carry. * fix(stop): share the receipt naming rule, and stop leaking obligations Eleventh review round found that moving receipts to one file per claim broke the lane that could not see the change. bin/ocx.mjs kept checking the retired pending-teardown.json, so the npm/dashboard updater silently stopped noticing every outstanding obligation. The naming rule now lives in one shared .mjs both lanes import, and a test asserts the launcher no longer names the old file. Three leaks followed from the same area. The orphan path already knew where the proxy answered - findLiveProxy returns port and hostname - and threw it away to re-read the runtime record that had usually gone missing in the first place. It hands that endpoint over now. When nothing can be resolved at all the stop still proceeds undeferred, but says so: that case keeps the pre-3008 window and should not be silent. Inherited receipts were evaluated only when this run claimed none of its own, so an ordinary stop of a live proxy cleared its own nonce and left every older abandoned receipt behind. They accumulated with each run. Inherited obligations are now evaluated either way, and every nonce this stop proved discharged is released together with its own. An unreadable receipt was quarantined before the restore ran, which erased it from every future scan even when a sibling receipt blocked the restore, the restore failed, or the process died first. It now fails the stop with an explicit manual step - it names no endpoint, so nothing can prove its proxy down, and the evidence requirement that applies to valid receipts applies to it too - and is set aside only after the outcome is known. * fix(stop): keep an unreadable obligation enforcing, and always leave a receipt Twelfth review round found that quarantine had quietly become an escape hatch. An unreadable receipt blocked the current stop, then got renamed to a suffix both updater scanners ignored - so the very next update saw nothing outstanding and installed over a teardown that never ran. Removing enforcement to avoid a retry loop turns "we could not tell" into "it is fine", which is the failure this mechanism exists to prevent. Quarantine now renames to pending-teardown-<nonce>.unreadable.json, which the recovery loop stops reading but pendingTeardownOutstanding still counts. Only a human removing the file ends it, and the message says so. Two durability holes on the stop paths. A tracked stop that could resolve no endpoint warned and then hard-killed with no receipt at all - and on that path stopProxy goes straight to the kill ladder, so no child teardown runs either. A warning does not make a hard kill durable. Every stop now claims a receipt; when nothing is discovered it records the configured listen address, which is the same address a later recovery probe would ask about. That endpoint is deliberately not used to direct the stop request itself: it is a good enough guess to record an obligation against, not to POST to. And a live orphan with no resolvable pid fell through as "No running proxy found", which purged the state records and restored shared config underneath a proxy that was still serving. findLiveProxy returning a proxy with pid null is now its own failure with the teardown blocked, distinct from returning nothing. * fix(stop): record endpoint provenance, and re-check obligations after the stop Thirteenth review round found the configured-address fallback had quietly become evidence. A stop that could resolve no endpoint recorded the configured listen address, and a later recovery probed exactly that address and read a refused connection as proof the proxy was down. For a proxy started with an explicit --port and respawned there by a wrapper, that is a restore underneath a live server - the failure the receipt exists to prevent, reached through the receipt itself. Receipts now carry endpointSource: exact means the runtime record or a successful probe, guessed means the configured address. Only exact authorizes automatic recovery; a guessed receipt fails closed with the manual steps. A receipt with no provenance at all is invalid, so nothing written by the previous format is trusted. The same function also read the runtime record twice, so the receipt could name the guess while the request went to an endpoint that appeared in between. It resolves once now. Separately, quarantine still did not actually block an update. Both lanes checked for outstanding obligations only before spawning ocx stop, and a quarantined receipt lets that stop succeed - there is nothing left to stop - so the retry sailed through and installed anyway. The shared post-stop decision now takes teardownOutstanding and returns teardown-outstanding, and both lanes pass it and print the manual remediation. Absent, the field keeps the previous behaviour. * fix(stop): say what quarantine actually does Fourteenth review round found the quarantine message claiming the opposite of the contract it was implementing. It told the operator the set-aside receipt "no longer blocks an update" - but isAnyTeardownObligationFileName counts the renamed file on purpose, and both post-stop decisions now abort while it is there. An operator following that message would wait for an update that keeps refusing, with no idea why. It now says the receipt still blocks 'ocx update', that ocx stop has not restored on its behalf, and what to do: confirm nothing is running, run 'ocx restore', then delete the printed path. The comment above it no longer claims a condition the code does not check - the skip is on ownershipBlocked, because a foreign service still owns that state - and a wiring assertion pins the wording so it cannot drift back. * fix(restore): finish the Grok half when the Codex half is a no-op Fifteenth review round followed the manual recovery instructions this branch prints and found they do not work. The deferred-teardown paths tell an operator to run 'ocx restore' and then delete the receipt. But when Codex is already OFF and native, restore takes an early success return that never reaches stripGrokConfig - so a Grok-only deferred teardown survives, the operator deletes the obligation that would have caught it, and updates unblock while Grok still points at a proxy that is gone. That early return now strips the fence too, folds the outcome into the message and the --json envelope, and exits nonzero when the cleanup fails. The Codex half being a no-op never said anything about the Grok half. tests/restore-completes-shared-teardown.test.ts drives the real CLI through dispatchCommand with Codex off, a managed Grok fence present, and asserts the fence is gone, the envelope names it, and a home with no Grok config still succeeds quietly. * fix(restore): strip the fence on every path, and stop reading a failed scan as empty Sixteenth review round found the previous fix covered one early return and not the other. `ocx restore --json` returned its envelope before stripGrokConfig() ran, so a non-clean Codex home plus a Grok fence produced exit 0 and success: true while Grok still targeted the dead proxy - and `ocx eject --json` inherits it, being the same runner. The strip now happens before either output. The Codex artifact schema is unchanged; the Grok outcome folds into success and message so a machine caller cannot read a half teardown as done. The other one was worse because it was pinned by a test I wrote. Both lanes treated ANY readdir failure as "no obligations" - a permission error, an I/O error, a file where the home should be - so a scan that could not see an outstanding receipt reported none, and an update installed over a teardown that never ran. Only ENOENT is empty now. listPendingTeardowns surfaces an unreadable home as one invalid obligation so handleStop blocks on it like any other rather than restoring over an unread directory. The new regressions assert the fence file itself rather than the message, since a message can claim a cleanup that never happened, and cover the forward path, eject --json, and a Grok strip that fails. * fix(stop): make every teardown path report the whole outcome Seventeenth review round moved outward from the deferral to the paths around it, and found four places where a partial teardown reported success. POST /api/stop called the boolean service helper and ignored the result. That helper collapses "the manager refused to stop" into the same false as "no service installed", so the route could restore shared config and exit while a manager that would respawn the proxy was still running. It consumes the detailed outcome now: "failed" answers 409 without touching shared config, and "stopped-respawnable" answers 409 unless the caller holds a teardown receipt, because this process cannot verify its own post-exit respawn window - only the parent ocx stop can, which is what the receipt exists for. The same route decided success from the native restore alone and appended the Grok failure as text, so native success plus a failed strip returned success: true. Success now requires both halves. The existing test failed both at once, which masked it; the matrix case is added. ocx service stop and ocx service uninstall logged their restore and strip failures and exited 0, so a script could not tell a finished teardown from one that left Grok aimed at a stopped proxy. Both set a failure code now, matching what the full ocx uninstall already did. And the previous commit's unreadable-home marker was a fabricated receipt with an all-zero nonce, which handleStop would hand to the quarantine path - able to rename a real receipt that happened to carry it, and otherwise printing a manual-removal path for a file that does not exist. A scan failure is its own state now: it blocks, it is never quarantined or cleared, and it asks for the directory to be fixed. * fix(stop): refuse a respawnable backend before touching it, not after Seventeenth round's route guard was itself a regression, and the eighteenth caught it. The check ran AFTER stopServiceIfInstalledDetailed(), so a bare dashboard Stop on Windows Task Scheduler ended the task, then returned 409 - leaving the proxy running with its manager stopped, which is worse than either outcome it was choosing between. And the dashboard sends a bare request on every backend, so this was the documented Stop button, broken. installedServiceCanRespawn() answers the same question without stopping anything: it probes the scheduler task and treats an unanswerable probe as risk rather than absence. The route asks first, refuses with code respawnable_service and "Nothing was changed", and only then touches the manager. The docs now say the dashboard refuses on that backend and why. Second: the delayed exit still considered only the drain, so a proxy that drained cleanly but failed its native or Grok restore exited 0 - telling a supervisor the stop was clean while client config still pointed at it. The exit is now shutdownSucceeded && teardown.success, and the assertion that pinned the incomplete expression is updated with it. * fix(stop): only a proven absence is safe, and say so in every locale Nineteenth review round found the respawn pre-check testing for the wrong thing. probeWindowsSchedulerTask returns "unknown" as an ordinary value when its queries fail - it does not throw - so testing status === "present" let that case straight through: the route entered the detailed stop, killed scheduler wrappers, and only then returned 409. The mutate-then-refuse defect, back for a third time through a different door. The predicate now proceeds only on a proven "absent" and takes injected probe/platform so the four cases can be tested for real. The old assertion passed by matching an unrelated early exit in the catch; it is replaced with present/unknown/throw/absent plus the two non-Windows platforms. The docs were also still contradicting themselves. The management API reference listed only the ownership 409, the lifecycle reference still said the dashboard does the same thing as ocx stop, and every translated locale carried the old unconditional claim. All eight management-api locales now name respawnable_service, and the lifecycle and dashboard pages say what the dashboard does on that backend and that nothing is changed when it refuses. * fix(stop): give an unreadable scheduler state its own answer, not a loop Twentieth review round found the refusal was circular for the one operator who most needed it. installedServiceCanRespawn collapsed present and unknown into one boolean, so a Windows host whose schtasks query is broken got respawnable_service and was told to run ocx stop - and that command maps the same unknown probe to a stop failure, so it could not finish either. There was no way out. The predicate returns none | respawnable | unknown now. Unknown gets its own 409 service_state_unknown that names the actual remedy: run ocx service status to see the query error, repair Task Scheduler access, retry. It deliberately does not mention ocx stop. The docs sweep from the previous commit was also half-done. Six management-api locales carried the new clause twice because the script ran twice, three lifecycle references (zh-cn, ru, tr) still claimed unconditional dashboard parity, and three dashboard guides (fr, ja, tr) still said Stop always succeeds. All are deduplicated and updated, and every locale now lists service_state_unknown alongside the other two 409s. * fix(stop): carry the unreadable-scheduler diagnosis into ocx stop too Twentieth round fixed the API's circular advice and left the CLI with the same problem underneath it. stopServiceIfInstalledDetailed folded an unanswerable scheduler probe into the generic failed, so ocx stop printed only "the manager did not stop" - the wrong thing to go looking for - while the API was telling that operator to run ocx stop. ServiceStopOutcome gains state-unknown, and both surfaces now say the query could not be read and point at ocx service status. The precedence moved into classifyWindowsServiceStop so it can be tested by calling it: a readable failure outranks an unreadable state, and an unreadable state outranks success, because a scheduler we cannot see may still respawn the proxy. My first attempt at this test read source text and did not fail when the mapping was reverted, which is exactly the failure mode this review has caught repeatedly. Docs: the Japanese management row was malformed - my dedup script had mangled a row with a different column count, leaving a doubled clause, a cell outside the table, and trailing whitespace that failed git diff --check. It is rewritten by hand. The secondary POST /api/stop table in all eight dashboard guides now carries the two refusal codes as well. * fix(uninstall): stop reading an unreadable service state as "not installed" Twenty-first review round found the last place the boolean collapse still hid a live manager. stopServiceIfInstalled returns false for "not installed", "refused to stop" and "state could not be read" alike, and handleUninstall read that false as absence - printed "service stopped: not installed", then restored native Codex and stripped the Grok fence under a proxy that may still have been running and managed. It did exit nonzero afterwards, but only after doing the unsafe teardown and telling the operator something untrue. Uninstall consumes ServiceStopOutcome directly now. Absent is still "not installed"; failed and state-unknown each throw with their own explanation. A failure in the service stop, the proxy stop, or the service removal clears serviceTeardownProven, and the shared restores run only when that holds. When they are skipped it is recorded as a failure, not a silent pass, with the follow-up command to run once the blocker is resolved. * fix(uninstall): find the orphan, and make the authorization rule callable Twenty-second review round found uninstall trusting a missing pid file as proof that nothing was serving. ocx stop falls back to identity-checked discovery for exactly that case - a proxy outliving its record - and uninstall did not, so a live orphan left the teardown authorized: shared config restored, local state removed, success reported, proxy still running. It discovers now, and a live proxy with no resolvable pid throws rather than being read as absence. The rule itself moved into src/cli/uninstall-plan.ts so it can be exercised by calling it. My previous test read the source and could not execute the failure permutations, which is why the orphan path stayed green - the same wrong-reason pattern this review has caught repeatedly. sharedTeardownAuthorized takes what each step observed, the fields start pessimistic so a thrown step cannot look like success, and all eight permutations are asserted. The skip message also promised the wrong remedy: ocx restore returns client routing but leaves the service removal and local cleanup this command had not reached. It now says to resolve the blocker and rerun ocx uninstall, with ocx restore named only as an interim step. And the boolean stopServiceIfInstalled is deleted. It had no production caller left, and leaving it there is an invitation to the same defect a third time. * fix(uninstall): stop manufacturing proof out of three different unknowns Twenty-third review round found the authorization rule reading correctly while every observation feeding it was wrong. uninstallServiceIfInstalled returned false for absence AND for darwin/linux removal failures, so a service that could not be removed was recorded as never installed - which authorized the shared teardown and let the command report success with the assets still there. There is a typed uninstallServiceDetailed now: absent, removed, failed. The boolean wrapper throws on failed so no caller can inherit the old collapse. stopped-respawnable was authorized on the theory that removing the registration stops the wrapper. It does not: killing the running :loop wrapper is explicitly best-effort (lidge-jun#764), which is why ocx stop polls the restart window. Uninstall now polls the same window after removal and only then may take shared config down. And the orphan discovery treated a findLiveProxy miss as proof of absence, when that null also covers a timeout and a transport failure. It goes through the tri-state probe, which says dead only for a refused connection or a definitive non-OpenCodex answer; anything else fails the step. The observation fields are renamed to say what they now mean - proxyProvenDown, serviceRemoval, respawnWindowVerified - and the permutation test covers all of them, including the two respawnable cases that differ only by the window check. * fix(uninstall): prove every endpoint, and stop reading silence as a dead window Twenty-fourth review round found both remaining proofs still manufacturing certainty from a single probe. The endpoint fallback preferred the runtime record over configuration and asked only that one. findLiveProxy had already probed both candidates before returning null, so a stale record pointing at a closed port plus a live-but-unresponsive proxy on the configured port gave "dead" for a port nobody was using - and that authorized the teardown. endpointsToProve collects every distinct candidate and everyEndpointProvenDown requires all of them to be definitively dead. The restart-window poll had the same shape. proxyStillLiveAfterStop returns null on a timeout as well as on a genuinely dead endpoint, so a respawned wrapper that would not answer looked verified-down. The window is now only verified once the tri-state probe says dead on every candidate. Both observations are pure functions now, so the test drives closed, live and silent listeners by calling them rather than reading the command's source - which is what let both of these stay green through the previous round. * docs(devlog): record what 26 review rounds added to the lidge-jun#3008 unit
1 parent 93ba1a0 commit ef6c5a9

50 files changed

Lines changed: 2927 additions & 145 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎bin/ocx.mjs‎

Lines changed: 58 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@
99
* src/cli/index.ts — only the published npm `bin` routes through here.)
1010
*/
1111
import { spawn, spawnSync } from "node:child_process";
12+
import { STOP_HISTORY_INCOMPLETE_EXIT_CODE } from "../src/update/stop-contract.mjs";
13+
import { probeProxyLiveness } from "../src/update/proxy-liveness-probe.mjs";
14+
import { decidePostStopUpdate } from "../src/update/stop-decision.mjs";
1215
import { randomBytes } from "node:crypto";
1316
import { createRequire } from "node:module";
1417
import { existsSync, readFileSync, readdirSync } from "node:fs";
@@ -17,6 +20,7 @@ import { dirname, join, resolve } from "node:path";
1720
import { fileURLToPath } from "node:url";
1821
import { isRealBunBinary } from "../src/lib/bun-binary-validator.mjs";
1922
import { npmInvocation } from "../src/update/npm-invocation.mjs";
23+
import { hasPendingTeardownIn } from "../src/config/pending-teardown-names.mjs";
2024
import {
2125
npmCachePreflightFailureMessage,
2226
runNpmCachePreflight,
@@ -202,7 +206,12 @@ function runNpmSelfUpdate() {
202206
// Capture listen target before stop clears runtime-port.json (mirrors GUI/CLI update worker).
203207
// Do not treat a live runtime port of 10100 as "missing" — track whether the read succeeded.
204208
let bakePort = 10100;
209+
// The hostname travels with the port: a proxy bound to ::1 or a specific interface is
210+
// invisible to a probe that assumes 127.0.0.1, and "no answer" would then read as
211+
// "stopped" for exactly the proxy the probe exists to find.
212+
let bakeHostname = "127.0.0.1";
205213
let sawRuntimePort = false;
214+
let sawRuntimeHostname = false;
206215
try {
207216
const rt = JSON.parse(readFileSync(join(configDir(), "runtime-port.json"), "utf8"));
208217
if (Number.isFinite(rt?.port) && rt.port > 0 && rt.port <= 65535) {
@@ -219,16 +228,29 @@ function runNpmSelfUpdate() {
219228
}
220229
if (runtimeLive) {
221230
bakePort = Math.trunc(rt.port);
231+
if (typeof rt?.hostname === "string" && rt.hostname.trim() !== "") {
232+
bakeHostname = rt.hostname.trim();
233+
sawRuntimeHostname = true;
234+
}
222235
sawRuntimePort = true;
223236
}
224237
}
225238
} catch { /* fall through to config */ }
226-
if (!sawRuntimePort) {
239+
// Port and hostname resolve INDEPENDENTLY: a legacy runtime record carries a port and no
240+
// hostname, and skipping config in that case probed 127.0.0.1 for a proxy bound to ::1.
241+
if (!sawRuntimePort || bakeHostname === "127.0.0.1") {
227242
try {
228243
const cfg = JSON.parse(readFileSync(join(configDir(), "config.json"), "utf8"));
229-
if (Number.isFinite(cfg?.port) && cfg.port > 0 && cfg.port <= 65535) bakePort = Math.trunc(cfg.port);
244+
if (!sawRuntimePort && Number.isFinite(cfg?.port) && cfg.port > 0 && cfg.port <= 65535) {
245+
bakePort = Math.trunc(cfg.port);
246+
}
247+
if (!sawRuntimeHostname && typeof cfg?.hostname === "string" && cfg.hostname.trim() !== "") {
248+
bakeHostname = cfg.hostname.trim();
249+
}
230250
} catch { /* keep default */ }
231251
}
252+
// Wildcard and bracketed-IPv6 normalization lives in probeProxyLiveness, so both lanes
253+
// get it from one place.
232254

233255
const launcher = fileURLToPath(import.meta.url);
234256

@@ -343,17 +365,47 @@ function runNpmSelfUpdate() {
343365
}
344366
}
345367

346-
if (serviceWasInstalled || hasRuntimeState) {
368+
// An outstanding pending-teardown receipt is a fourth reason to run the stop. After a
369+
// parent crashed mid-deferral the service, pid and runtime records can all be absent
370+
// while the shared client config still points at a proxy that is gone; installing over
371+
// that silently skips the recovery the receipt was written to trigger (#3008). Presence
372+
// is the whole test here — the launcher cannot parse it, and `ocx stop` is what decides
373+
// whether the obligation is safe to finish.
374+
const hasPendingTeardown = hasPendingTeardownIn(readdirSync, configDir());
375+
if (serviceWasInstalled || hasRuntimeState || hasPendingTeardown) {
347376
console.log("⏹ Stopping the running proxy before updating...");
348377
const stopRes = spawnSync(process.execPath, [launcher, "stop"], { stdio: "inherit", windowsHide: true });
349378
const stillHasRuntimeState =
350379
existsSync(join(configDir(), "ocx.pid")) || existsSync(join(configDir(), "runtime-port.json"));
351-
if (stopRes.status !== 0 || stillHasRuntimeState) {
380+
// A history-only failure means teardown succeeded and a backup manifest is waiting for
381+
// review: the proxy is down and replacing package files is safe. Every other nonzero
382+
// status is a stop that did not finish, and a signal kill (status null) says nothing
383+
// about whether it did - both abort, because replacing files under a live server
384+
// leaves it running mixed old and new modules (#3008).
385+
// The same decision the Bun updater makes, from the same module (#3008). Absent PID and
386+
// runtime files are weak evidence, so the captured endpoint is asked; "unknown" aborts
387+
// because a silent listener is exactly the state where replacing files is dangerous.
388+
const decision = decidePostStopUpdate({
389+
status: stopRes.status,
390+
hasRuntimeState: stillHasRuntimeState,
391+
// Re-checked AFTER the stop: a quarantined receipt lets the stop itself succeed
392+
// (there is nothing left to stop), so a pre-stop check alone let the retry install
393+
// over a teardown that never ran.
394+
teardownOutstanding: hasPendingTeardownIn(readdirSync, configDir()),
395+
liveness: probeProxyLiveness(bakePort, bakeHostname),
396+
});
397+
const historyOnlyStop = decision.reason === "history-only";
398+
if (!decision.proceed) {
352399
if (trayBeforeUpdate.restoreOnFailure) runTrayLifecycle(launcher, "start");
353-
console.error("opencodex: could not stop the running proxy; aborting the update. Run 'ocx stop' and retry.");
400+
if (decision.reason === "teardown-outstanding") {
401+
console.error("opencodex: a shared teardown from an earlier stop is still outstanding and needs manual review; aborting the update.");
402+
console.error("opencodex: confirm no proxy is running, run 'ocx restore', then remove the pending-teardown file in the opencodex home.");
403+
} else console.error(decision.reason === "proxy-unknown"
404+
? `opencodex: could not confirm the proxy on ${bakeHostname}:${bakePort} is stopped; aborting the update. Run 'ocx stop' and retry.`
405+
: "opencodex: could not stop the running proxy; aborting the update. Run 'ocx stop' and retry.");
354406
process.exit(1);
355407
}
356-
if (historyRestoreIncomplete()) {
408+
if (historyOnlyStop || historyRestoreIncomplete()) {
357409
console.warn(
358410
"opencodex: WARNING — Codex resume-history metadata restore is incomplete (a backup manifest remains).\n" +
359411
" The DB may be busy or the manifest/target may need review; untracked routed history is intentionally unchanged.\n" +
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# 050 outcome — wp5 (#3008): what the implementation added beyond this plan
2+
3+
The plan described one defect: `ocx update` aborting because `ocx stop` could not tell a
4+
history-only failure from a real stop failure. That fix is in the first commit of the
5+
branch. The other twenty-five came from adversarial review, and they are not incidental —
6+
each one is the same defect wearing different clothes: **a teardown that reports success
7+
while half of it did not happen.**
8+
9+
## The shape that kept recurring
10+
11+
Something cannot be determined, and the code treats "could not determine" as "determined
12+
to be fine". Every instance authorized taking shared client config down while a proxy
13+
might still be serving, which leaves Codex or Grok pointed at a process that is gone.
14+
15+
| Where | What was read as proof | Round |
16+
| --- | --- | --- |
17+
| `POST /api/stop` | `stopServiceIfInstalled` false — collapsed "not installed" with "refused to stop" | 17 |
18+
| `POST /api/stop` | Success decided from the native restore alone, Grok failure appended as text | 17 |
19+
| `ocx service stop` / `uninstall` | Restore and strip failures logged, exit code still 0 | 17 |
20+
| Route pre-check | Scheduler stopped first, refused second — mutate-then-refuse | 18 |
21+
| Daemon exit | Drain success alone, ignoring the teardown result | 18 |
22+
| Respawn predicate | `status === "present"`, so an unreadable probe passed as absent | 19 |
23+
| `ocx restore` | Early return on the Codex no-op path, never reaching the Grok strip | 15 |
24+
| `ocx restore --json` | Same, on the ordinary forward path | 16 |
25+
| Receipt scan | Every `readdir` error read as "no obligations" | 16 |
26+
| `ocx uninstall` | `stopServiceIfInstalled` false read as "not installed" | 22 |
27+
| `ocx uninstall` | Missing pid file read as "no proxy serving" | 23 |
28+
| `ocx uninstall` | `uninstallServiceIfInstalled` false — absence and removal failure | 24 |
29+
| `ocx uninstall` | Registration removed, running wrapper assumed dead | 24 |
30+
| `ocx uninstall` | `findLiveProxy` null read as proof of absence | 24 |
31+
| `ocx uninstall` | One endpoint probed while two were candidates | 25 |
32+
| `ocx uninstall` | `proxyStillLiveAfterStop` null read as a verified window | 25 |
33+
34+
## The deferral, and why it needed four attempts
35+
36+
`ocx stop` has to defer shared teardown to itself, because the proxy exits before anyone
37+
can verify a Task Scheduler wrapper did not respawn it. Expressing that obligation took
38+
four tries:
39+
40+
1. A query flag. Any authenticated caller could set it and exit, and a parent that died
41+
mid-stop left nothing on disk saying a restore was owed.
42+
2. A receipt file. Presence is not ownership — another caller could ride on it.
43+
3. A nonce inside one shared file. Read-compare-unlink is three syscalls, so a concurrent
44+
stop replacing the file between the compare and the unlink got its obligation deleted.
45+
4. **The nonce as the filename.** `unlink` names one specific obligation and cannot reach
46+
another. Two concurrent stops hold two receipts, which is the truth of the situation.
47+
48+
The receipt also carries the endpoint being stopped and how it was obtained. A configured
49+
address is recorded as `guessed` and never authorizes automatic recovery: a proxy on an
50+
explicit `--port` can be respawned there while the configured port refuses.
51+
52+
## Tests that passed for the wrong reason
53+
54+
Five times a regression was written as a source-text assertion, and five times reverting
55+
the defect left it green. The reviewer caught each one. Where a rule mattered it was
56+
extracted into something callable — `performStopTeardown`, `classifyWindowsServiceStop`,
57+
`sharedTeardownAuthorized`, `endpointsToProve`, `everyEndpointProvenDown` — and the test
58+
now executes the permutations. Source assertions remain only for wiring: that a route
59+
delegates to the extracted rule rather than growing a second copy.
60+
61+
Every fix on this branch was driven RED against the specific defect and restored.
62+
63+
## Docs
64+
65+
Sixteen files across eight locales carry the new refusal contract: `respawnable_service`
66+
and `service_state_unknown`, and the fact that the dashboard Stop button refuses on the
67+
Windows Task Scheduler backend rather than half-performing a stop it cannot verify.
68+

‎docs-site/src/content/docs/fr/guides/web-dashboard.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ gestionnaire de mots de passe.
5555
| **Journaux** | Actualisez automatiquement les requêtes récentes et consultez les jetons, l'effort demandé et, lorsqu'il est disponible, l'effort sortant effectif, le modèle résolu, le fournisseur, l'état, l'identifiant de requête, la durée et les détails de l'erreur. La vue détaillée inclut le champ exact de raisonnement transmis lorsque l'adaptateur en émet un. Filtrez par identifiant opaque de conversation ou de session — si le client en fournit un — afin d'obtenir le total des jetons et le coût estimé au tarif catalogue pour l'anneau de journaux actuellement chargé. |
5656
| **Utilisation / Débogage** | Examinez la couverture et les tendances d'utilisation des jetons, ou activez à la demande les diagnostics de transport et d'extraction de l'utilisation propres aux fournisseurs. |
5757
| **Stockage** | Consultez en lecture seule la répartition du disque de CODEX_HOME — sessions, archives, bases de données et pièces jointes. Pour le nettoyage facultatif des archives, prévisualisez les N % les plus anciennes, puis placez-les en quarantaine dans `CODEX_HOME/.trash` (par défaut) ou supprimez-les définitivement après avoir coché une case explicite. **La stratégie de nettoyage automatique** est facultative et **désactivée par défaut** (`storageCleanupPolicy.enabled`) ; configurez son seuil, sa cible, sa planification et son mode sur la page **Stockage**, ou lancez **Exécuter maintenant**. Les entrées mises en quarantaine peuvent être restaurées depuis cette page (JSONL et fils). Les sessions actives restent en lecture seule. Le nettoyage et la restauration sont refusés tant que Codex verrouille le fichier `state_*.sqlite` le plus récent ou actif. |
58-
| **Arrêter** | Arrêtez proprement le proxy et le service d'arrière-plan installé, restaurez Codex natif et quittez (`POST /api/stop`). |
58+
| **Arrêter** | Arrêtez proprement le proxy et le service d'arrière-plan installé, restaurez Codex natif et quittez (`POST /api/stop`). Sur Windows avec le backend Planificateur de tâches, le tableau de bord refuse et vous demande d'exécuter `ocx stop` : le wrapper peut relancer le proxy après la fin de la tâche, et seul un stop exécuté hors du proxy peut vérifier cette fenêtre de redémarrage avant de restaurer votre configuration client. Rien n'est modifié en cas de refus. |
5959

6060
### Liens directs vers une section
6161

@@ -174,7 +174,7 @@ L'interface graphique est un client léger de l'API JSON de gestion du proxy. Pa
174174
| `POST /api/codex-auth/login` · `GET /api/codex-auth/login-status` | Ajouter un compte au groupe au moyen d’une connexion dans le navigateur. |
175175
| `GET /api/logs?tail=50&limit=20&offset=0&provider=...&status=5xx` | Lire les métadonnées des requêtes récentes avec des filtres facultatifs de fin de journal, de fournisseur et d'état exact ou par classe. Avec `limit`/`offset`, la pagination remonte depuis la ligne la plus récente (`offset=0` renvoie la dernière page). Forme de la réponse : `{ timeZone, total, logs }`, où `total` est le nombre de lignes filtrées avant pagination. |
176176
| `GET` / `PUT /api/subagent-models` | Lire ou définir les cinq modèles de remplacement `spawn_agent` mis en avant. |
177-
| `POST /api/stop` | Arrêter le proxy et le service, restaurer Codex natif et quitter. |
177+
| `POST /api/stop` | Arrêter le proxy et le service, restaurer Codex natif et quitter. Refusé avec `respawnable_service` sur le backend Planificateur de tâches Windows, et avec `service_state_unknown` lorsque cet état ne peut pas être lu ; rien n'est modifié dans les deux cas. |
178178

179179
:::tip
180180
L'ajout d'**Ollama Cloud** ou d'un autre fournisseur doté d'un catalogue depuis le tableau de bord copie sa

‎docs-site/src/content/docs/fr/reference/cli/lifecycle.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ ocx start --port 8080
2424

2525
### `ocx stop`
2626

27-
Arrête le proxy actif à partir de son PID, supprime le fichier de PID et rétablit le fonctionnement natif de Codex. Si un service d’arrière-plan géré est installé, `ocx stop` l’arrête d’abord afin qu’il ne puisse pas relancer le proxy. La même opération est disponible avec le bouton **Stop** du tableau de bord Web (`POST /api/stop`).
27+
Arrête le proxy actif à partir de son PID, supprime le fichier de PID et rétablit le fonctionnement natif de Codex. Si un service d’arrière-plan géré est installé, `ocx stop` l’arrête d’abord afin qu’il ne puisse pas relancer le proxy. Le bouton **Stop** du tableau de bord Web exécute la même opération (`POST /api/stop`) sur tous les backends, sauf le Planificateur de tâches Windows : le wrapper peut y relancer le proxy après la fin de la tâche, donc le tableau de bord refuse avec `respawnable_service`, ne modifie rien et vous demande d'exécuter `ocx stop`.
2828

2929
### `ocx restart`
3030

‎docs-site/src/content/docs/fr/reference/management-api.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,7 @@ lui-même s'il souhaite ajouter une étoile au dépôt.
227227
| --- | --- | --- |
228228
| `GET /api/system/memory` | Renvoyer les mesures scalaires du processus, du tas, des flux, de l'état des réponses, du mécanisme de surveillance et des tours actifs | — |
229229
| `POST /api/system/restart` | Amorcer un redémarrage du processus qui attend l'évacuation des requêtes, sans retirer l'injection du client | Renvoie 202 ; les appels répétés signalent l'évacuation déjà en cours |
230-
| `POST /api/stop` | Arrêter le service, restaurer Codex en mode natif, retirer l'injection Grok gérée et évacuer les requêtes du proxy | 409 conflit de propriété du service |
230+
| `POST /api/stop` | Arrêter le service, restaurer Codex en mode natif, retirer l'injection Grok gérée et évacuer les requêtes du proxy | 409 conflit de propriété du service; 409 `respawnable_service` lorsqu'un wrapper du Planificateur de tâches Windows pourrait relancer le proxy et que l'appelant n'est pas `ocx stop` (rien n'est modifié) ; 409 lorsque le gestionnaire installé refuse de s'arrêter ; 409 `service_state_unknown` lorsque l'état du Planificateur de tâches ne peut pas être lu (rien n'est modifié ; réparez la requête puis réessayez) |
231231
| `GET /api/system/codex-app-server` | Indiquer si les serveurs d'application Codex en cours d'exécution sont antérieurs au catalogue de modèles actuel | — |
232232
| `POST /api/system/codex-restart` | Actualiser le catalogue, puis demander aux serveurs d'application Codex obsolètes de s'arrêter afin que le sélecteur de modèles se recharge | Renvoie 200 avec `code: partially_stopped` lorsqu'une cible ne s'arrête pas |
233233

0 commit comments

Comments
 (0)