Skip to content

fix(codex/app-server): release session lane when projector throws on turn/completed#69072

Merged
steipete merged 3 commits into
openclaw:mainfrom
ayeshakhalid192007-dev:fix/67996-codex-lane-release
Apr 20, 2026
Merged

fix(codex/app-server): release session lane when projector throws on turn/completed#69072
steipete merged 3 commits into
openclaw:mainfrom
ayeshakhalid192007-dev:fix/67996-codex-lane-release

Conversation

@ayeshakhalid192007-dev

Copy link
Copy Markdown
Contributor

Summary

  • Problem: After the Codex plugin finishes a turn, follow-up messages on the same session queue forever. The gateway never releases the session:agent:main:<session> lane lock.
  • Why it matters: Users can send exactly one Codex reply per session, then the channel appears frozen. /status shows the run as still in progress.
  • What changed: In extensions/codex/src/app-server/run-attempt.ts, the notification pump now wraps projector.handleNotification in try/finally and computes the terminal-turn flag before invoking the projector, so a throw inside a downstream consumer still fires resolveCompletion(). The projector error is logged at debug and the run resolves cleanly.
  • What did NOT change (scope boundary): No protocol, lane manager, or projector logic was altered. No behavior change on the happy path — only the error-isolation seam around one async call. The sibling models.test.ts got the same await vi.waitFor / vi.mock pattern that shared-client.test.ts already uses, so the full codex extension suite is reliably green while the new regression test runs.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause (if applicable)

  • Root cause: `handleNotification` awaited `projector.handleNotification` and only fired `resolveCompletion?.()` after the await. If any downstream consumer (e.g. `onAgentEvent` projecting a `plan` item from `turn/completed`) threw, the rejection bubbled out of `enqueueNotification` and was swallowed by the notification queue's `.catch` handler — the terminal-turn resolve never ran, `completion` stayed pending, and `runCodexAppServerAttempt` hung until `params.timeoutMs`. By the time it timed out, the lane manager had already marked the command as in-flight, so new messages queued behind a run that would never cleanly resolve.
  • Missing detection / guardrail: No unit coverage exercised a throwing projector callback on `turn/completed`. The `does not drop turn completion` test covered reordered notifications but not consumer failures.
  • Contributing context (if known): The Codex projector was recently extended to emit richer `plan` items, which is how downstream consumers started hitting the throw path more often in practice.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: `extensions/codex/src/app-server/run-attempt.test.ts` — `releases completion when a projector callback throws during turn/completed`.
  • Scenario the test should lock in: A harness where `params.onAgentEvent` throws while processing a `plan` item inside `turn/completed` must still let `runCodexAppServerAttempt` resolve with `{ aborted: false, timedOut: false }`. If the fix is reverted, the run hangs on `completion` and the test times out.
  • Why this is the smallest reliable guardrail: It asserts the exact invariant — terminal-turn lane release — without coupling to projector internals or the gateway lane manager.
  • Existing test that already covers this (if any): None before this PR.
  • If no new test is added, why not: N/A — new test added.

User-visible / Behavior Changes

None user-visible on the happy path. Codex sessions where a downstream consumer throws during `turn/completed` now release their lane and surface the run result instead of wedging.

Diagram (if applicable)

```text
Before:
turn/completed -> projector.handleNotification -> throws
-> resolveCompletion() never called
-> completion pending
-> runCodexAppServerAttempt waits params.timeoutMs
-> session lane stays locked
-> follow-up messages queue forever

After:
turn/completed -> compute isTurnCompletion
-> try { projector.handleNotification } catch { debug-log }
-> finally { if (isTurnCompletion) resolveCompletion() }
-> completion resolves immediately
-> session lane released
-> follow-up messages flow
```

Security Impact (required)

  • New permissions/capabilities? `No`
  • Secrets/tokens handling changed? `No`
  • New/changed network calls? `No`
  • Command/tool execution surface changed? `No`
  • Data access scope changed? `No`
  • If any `Yes`, explain risk + mitigation: N/A

Repro + Verification

Environment

  • OS: Linux (Ubuntu 25.x, kernel 6.17) for test runs; issue originally reported on macOS 13.7.8
  • Runtime/container: Node 22, pnpm 10.33
  • Model/provider: codex (app-server)
  • Integration/channel (if any): any channel routing through the Codex plugin
  • Relevant config (redacted): default codex app-server configuration

Steps

  1. Start a Codex agent session and send a message that triggers a `plan` item in the final `turn/completed` notification.
  2. Arrange for a downstream consumer of `onAgentEvent` to throw on that item (e.g. a UI that asserts an invariant broken by the new plan shape).
  3. Send a second message on the same session.

Expected

  • The second message is processed by Codex.

Actual

  • Before the fix: the second message never starts. `/status` shows the first run as still in progress until `params.timeoutMs` elapses. The session lane stays locked for the full timeout window.

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Regression test in `extensions/codex/src/app-server/run-attempt.test.ts` fails on `upstream/main` without the fix and passes with it.

Human Verification (required)

  • Verified scenarios:
    • Local `pnpm test extensions/codex` green: 17/17 files, 81/81 tests.
    • Regression test in `run-attempt.test.ts` reproduces the hang on `upstream/main` without the fix and passes with it.
    • `pnpm tsgo:extensions` and `pnpm tsgo:extensions:test` clean.
    • `pnpm oxfmt --check` + `pnpm oxlint` clean on all touched files.
    • `pnpm build` clean with no `INEFFECTIVE_DYNAMIC_IMPORT` warnings.
  • Edge cases checked:
    • Projector throws on a non-terminal notification — run continues; lane release does not fire early because `isTurnCompletion` is scoped to `method === "turn/completed"` with matching `turnId`.
    • Projector throws on `turn/completed` for a different (stale/interleaved) `turnId` — release still gated on `isTurnNotification(notification.params, turnId)`.
  • What you did not verify: No live production Codex traffic on a running gateway — verification is unit-level plus the full extension suite.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? `Yes`
  • Config/env changes? `No`
  • Migration needed? `No`
  • If yes, exact upgrade steps: N/A

Risks and Mitigations

  • Risk: Swallowing projector errors at debug level could hide real downstream bugs.
    • Mitigation: The throw is logged via `embeddedAgentLog.debug` with method + error, so traces are still discoverable. The swallowed error never masks the turn outcome because only the lane-release side effect runs in `finally`. The right fix for any surfaced projector bug is still to fix the consumer, not to re-trap the lane.

@greptile-apps

greptile-apps Bot commented Apr 19, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a lane-lock regression where a throwing projector callback during turn/completed prevented resolveCompletion() from ever being called, leaving the session lane perpetually locked. The fix is a clean try/finally seam in handleNotification with the terminal-turn flag computed before the projector call, ensuring lane release regardless of downstream consumer errors.

Confidence Score: 5/5

Safe to merge — the fix is minimal, well-scoped, and backed by a targeted regression test that reproduces the hang.

The logic change is correct: computing isTurnCompletion before projector.handleNotification and using try/finally guarantees the completion path runs regardless of downstream throws. The test properly exercises the failure scenario. No happy-path behavior is altered. All remaining observations are P2 stylistic points that do not affect correctness.

No files require special attention.

Reviews (1): Last reviewed commit: "fix(codex/app-server): release session l..." | Re-trigger Greptile

@steipete
steipete force-pushed the fix/67996-codex-lane-release branch from 89ae4a6 to 5c6271f Compare April 20, 2026 23:46
@steipete
steipete merged commit 25428c4 into openclaw:main Apr 20, 2026
92 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Landed via temp rebase onto main.

  • Gate: pnpm check; pnpm test extensions/codex/src/app-server/event-projector.test.ts extensions/codex/src/app-server/run-attempt.test.ts extensions/codex/src/app-server/models.test.ts; GitHub PR checks green. Earlier OPENCLAW_LIVE_TEST=1 pnpm test:live completed with provider/auth/model-drift failures unrelated to this PR.
  • Source head: 5c6271f
  • Landed tip: 25428c4

Thanks @ayeshakhalid192007-dev!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c6271f66e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +446 to +449
try {
this.params.onAgentEvent?.(event);
} catch {
// Downstream event consumers must not corrupt the canonical Codex turn projection.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve diagnostics when suppressing agent-event callback failures

Catching and dropping all onAgentEvent exceptions here makes downstream consumer failures completely silent: emitAgentEvent no longer throws, so run-attempt.ts's projector-level debug logging path is never reached for this class of error. In production this means broken event consumers (the exact regression class this patch targets) can keep losing plan/item/compaction events with no warning signal, which materially hurts incident triage and makes behavior regressions hard to trace.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex harness holds session lane lock after run completes — blocks all subsequent messages

2 participants