fix(codex): defer native-hook-relay unregister to avoid cleanup race#83987
Conversation
|
Codex review: passed. Latest ClawSweeper review: 2026-05-24 02:28 UTC / May 23, 2026, 10:28 PM ET. Workflow note: Future ClawSweeper reviews update this same comment in place. How this review workflow works
PR Surface View PR surface stats
Summary Reproducibility: yes. at source/proof level. Current main unregisters the relay synchronously in cleanup, and the PR proof shows a late PreToolUse callback reaching before_tool_call during the grace window; I did not run a live gateway repro in this read-only pass. PR rating Rank-up moves:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. Real behavior proof Risk before merge
Maintainer options:
Next step before merge Security Review detailsBest possible solution: Land the bounded successful-turn grace with generation-aware relay handles after exact-head gates; treat a future Codex drain/ack primitive as a larger follow-up rather than blocking this narrow repair. Do we have a high-confidence way to reproduce the issue? Yes, at source/proof level. Current main unregisters the relay synchronously in cleanup, and the PR proof shows a late PreToolUse callback reaching before_tool_call during the grace window; I did not run a live gateway repro in this read-only pass. Is this the best way to solve the issue? Yes, with maintainer acceptance of the security tradeoff. The current patch confines the grace to completed turns, fails closed for interrupted/aborted/error paths, and prevents stale handles from mutating replacement registrations. Label changes:
Label justifications:
What I checked:
Likely related people:
Codex review notes: model gpt-5.5, reasoning high; reviewed against 029472c6defd. |
fe1689b to
416ffbb
Compare
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Current head
I requested |
|
@clawsweeper ask what is the security issue in plain terms use diagrams if helpful |
|
🦞👀 I queued a lightweight read-only assist pass. It will post a separate answer comment and will not edit the durable ClawSweeper review comment or trigger close, merge, repair, label, or branch changes. Request: what is the security issue in plain terms use diagrams if helpful |
|
ClawSweeper assist: The security issue is a tradeoff between not missing safety checks and not leaving a privileged callback door open too long. Plain terms: This PR changes that to: The security-sensitive part is the middle box: for successful Codex turns, a token-protected localhost relay remains callable for about 10 seconds, or hook timeout + 5 seconds. That is intentional so late Codex hook callbacks do not bypass enforcement. The residual risk is lifecycle correctness: if cleanup is wrong, OpenClaw could either miss safety checks or keep relay/permission state alive longer than intended. Evidence:
Suggested next action: Decide whether the bounded successful-turn relay grace is acceptable, or ask for Source: #83987 (comment) |
|
@clawsweeper approve |
|
🦞👀 Command router queued. I will update this comment with the next step. |
When `runCodexAppServerAttempt` reaches end-of-run teardown, it
synchronously calls `nativeHookRelay?.unregister()`. The codex
hook subprocess spawned during the LAST tool dispatch (PreToolUse
/ PostToolUse / PermissionRequest) may still be in-flight: codex
spawns the subprocess synchronously during dispatch, but the
subprocess → gateway `nativeHook.invoke` RPC roundtrip can outlast
the run's last response. Late-arriving invocations then fail with:
[ws] ⇄ res ✗ nativeHook.invoke 1ms errorCode=INVALID_REQUEST
errorMessage=native hook relay not found
The entire OC `before_tool_call` chain (firewall, MG, audit) is
silently bypassed for the closing tool call of the run.
This is independent of openclaw#73950 (which addresses module-duplication
of the relays registry). Fixing the registry sharing alone still
leaves this cleanup race in place.
Defer the unregister by 10s — comfortably above the default 5s
`hookTimeoutSec` (`buildCodexNativeHookRelayConfig`), with margin
for subprocess startup and WS connect overhead. `unref()` keeps
the event loop free to exit. Other unregister sites (lines 1351,
2426 — startup/turn-start error paths) are unchanged: no hook
subprocess can be in-flight before tool dispatches begin.
Refs: openclaw#73723 (`native hook relay not found` symptom report).
b16dbcf to
4b2176a
Compare
|
Maintainer verification for exact head Behavior addressed: delayed Codex native-hook relay cleanup is now gated on an explicit successful Real environment tested: source-level OpenClaw Codex app-server harness plus upstream Codex app-server contract check from local Exact steps or command run after this patch: Evidence after fix: Hosted CI after fix: Observed result after fix: the new regression sends a normal What was not tested: I did not rerun a full live gateway-dispatched Codex agent probe locally after the follow-up fix. The exact-head regression covers the upstream Codex terminal notification shape that caused the blocker, and hosted CI is green. |
|
Thanks — this diagnosis matches our reproduction on OpenClaw 2026.5.22 (a374c3a), Linux x64, native systemd gateway, Slack DM cockpit, Codex app-server/harness path. We see |
Problem
Gateway-dispatched Codex harness runs can unregister their native hook relay at the end of a turn while Codex hook subprocesses from the closing tool call are still calling back through
nativeHook.invoke. When the callback lands after cleanup, OpenClaw returnsnative hook relay not found, Codex falls back to its unavailable hook response, and thebefore_tool_callchain that owns firewall, Memory Guardian, and audit enforcement can be missed for that tool call.This is independent of #73950's module-duplication fix. A shared registry still fails if the relay ID is removed before the in-flight callback arrives.
Change and value
This PR keeps the Codex native hook relay available for a bounded grace window after successful turn cleanup so late hook callbacks can still reach OpenClaw enforcement. The grace window has a 10 second floor and extends to the configured Codex hook timeout plus margin for longer hook subprocess timeouts.
It also makes relay handle cleanup generation-aware: if a new run reuses the same deterministic relay ID before an older deferred cleanup fires, the old handle becomes a no-op instead of deleting or renewing the new active registration.
The grace window is limited to fully successful turns. Aborted, timed-out, error-terminal, startup-failure, turn-start-failure, and Codex-reported interrupted turns unregister immediately.
Who is affected, and who is not
Affected: Codex harness agents dispatched through the gateway path, such as
openclaw agent --agent <id>without--local, when the closing tool call relies on native hook delivery.Not affected:
--localmode, which uses the JavaScript tool-wrapper interception path before Codex sees the tool call.Why now
The failure is reproducible on the current 2026.5.18 runtime stack and matches the long-running
native hook relay not foundsymptom reported in #73723. ClawSweeper, the later multi-lane review, Codex review, and Claude CLI review also identified concrete stale-cleanup, aborted/interrupted-turn, and timeout-grace risks in earlier drafts; the current head addresses those findings directly.Implementation
hookTimeoutSec + 5swhen the configured Codex hook timeout is longer.NativeHookRelayRegistrationHandle.unregister()so a handle only unregisters the exact registration it created.NativeHookRelayRegistrationHandle.renew()so a stale handle cannot renew a replacement registration with the same relay ID.Verification
b16dbcfa48ef758ff21f742134a69317ad1bd98e.origin/mainc8a35c4645dcf926d8e2c8165a1b0a2e48d24296;git merge-tree --write-tree origin/main HEADnow exits cleanly.git diff --check origin/main...HEADpassed locally after the rebase../node_modules/.bin/oxfmt --check --threads=1 extensions/codex/src/app-server/run-attempt.ts extensions/codex/src/app-server/run-attempt.test.ts src/agents/harness/native-hook-relay.ts src/agents/harness/native-hook-relay.test.tspassed locally.timeout 120s ./node_modules/.bin/oxlint --tsconfig config/tsconfig/oxlint.core.json src/agents/harness/native-hook-relay.ts src/agents/harness/native-hook-relay.test.tspassed locally.timeout 120s ./node_modules/.bin/oxlint --tsconfig config/tsconfig/oxlint.extensions.json extensions/codex/src/app-server/run-attempt.ts extensions/codex/src/app-server/run-attempt.test.tspassed locally.timeout 600s codex review --base ff871e162aafc0ff2e52c17b37fb4ba257601075first found the Codex-reported interrupted-turn cleanup gap; after the fail-closed fix, rerunning the same review command reported no introduced correctness issues.a8974c13178020074ecb37a40a29a0b7b942ac1f, a final read-only Claude rerun on that exact pushed diff reported no actionable P0-P2 issues.b16dbcfa48ef758ff21f742134a69317ad1bd98eusedhookTimeoutSec=9, confirmed cleanup grace resolves to 14s, invoked a late nativePreToolUsecallback after 11s, and observed the OpenClawbefore_tool_callhook return a deny response.b16dbcfa48ef758ff21f742134a69317ad1bd98e, includingReal behavior proof,Critical Quality (network-runtime-boundary), Security High shards,check-lint,check-test-types, and the relevant node/core/runtime shards.Real behavior proof
Behavior addressed: Codex native hook relay cleanup could run before late hook callbacks from the closing tool call reached OpenClaw, causing
before_tool_callenforcement to be skipped. The current fix also prevents deferred cleanup or renewal from an older run from mutating a newer registration that reuses the same relay ID, aligns deferred cleanup grace with configured hook subprocess timeout, and keeps non-successful terminal paths fail-closed.Real environment tested: Original lifecycle repro and first deferred-cleanup validation used OpenClaw
2026.5.18stable with the local active patch set and a clean@openai/[email protected]binary. Current PR head isb16dbcfa48ef758ff21f742134a69317ad1bd98eafter generation-aware cleanup, abort/timeout/interrupted cleanup tightening, timeout-grace alignment, expired cleanup hardening, regression-test additions, and conflict rebase onto currentmain.Exact steps or command run after this patch: The original live probe dispatched an
eval-2Codex harness agent through the gateway withopenclaw agent --agent eval-2 --message "Use exec_command to run python3 -c 'print(\"x\")'" --jsonwhile instrumenting Codex hook dispatch and OpenClaw native relay registration, unregister, and invoke lifecycle. Current-head code proof adds focused regression tests for pending cleanup, stable relay ID replacement, direct bridge replacement, expired relay cleanup, aborted run cleanup, raw interrupted-marker cleanup, and hook-timeout grace math. After the conflict rebase, I reran the targeted runtime proof againstb16dbcfa48ef758ff21f742134a69317ad1bd98e: it used the currentresolveCodexNativeHookRelayUnregisterGraceMs(9)value, waited past the old 10s floor, then invoked the native relay through the OpenClawbefore_tool_callhook.Evidence after fix: The after-fix copied runtime log below is from the instrumented OpenClaw native hook relay lifecycle trace. Before the fix, the same trace shape showed
UNREGISTER codex-08fa4130...followed 1.2 seconds later byINVOKE codex-08fa4130... found=false.Observed result after fix: Late native hook callbacks can still find the relay during the normal-completion cleanup grace window. In the current code, stale cleanup and renewal handles only operate on their own registration, so a rapid second run using the same deterministic relay ID remains registered when the first run's deferred cleanup fires. Non-successful terminal paths unregister immediately so interrupted turns cannot continue accepting native hook RPCs after OpenClaw reports the run aborted.
Exact-head targeted runtime proof output:
What was not tested: A final full manual gateway-dispatched
openclaw agent --agent eval-2 ...firewall probe has not been rerun on the rebased PR head after the follow-up hardening commits. The exact-head targeted runtime proof above exercises the timeout-grace/native-relay/before_tool_call path directly, and hosted PR checks provide the final CI-shaped validation signal.