Skip to content

Fix broker shutdown hang on unresponsive socket - #300

Open
dhruvac29 wants to merge 2 commits into
openai:mainfrom
dhruvac29:fix-broker-shutdown-timeout
Open

dhruvac29 wants to merge 2 commits into
openai:mainfrom
dhruvac29:fix-broker-shutdown-timeout

Conversation

@dhruvac29

Copy link
Copy Markdown

Summary

Fixes a SessionEnd hang where sendBrokerShutdown could wait indefinitely if the broker accepted the shutdown socket connection but never replied, errored, or closed.

This adds a bounded timeout to the shutdown RPC and uses a single guarded cleanup path for data, error, close, and timeout.

Changes

  • Add timeoutMs = 5000 to sendBrokerShutdown
  • Resolve shutdown on timeout instead of waiting forever
  • Destroy the socket during cleanup
  • Add a regression test for a broker that accepts broker/shutdown but never responds

Testing

node --test tests/broker-lifecycle.test.mjs
npm test

Full suite passed locally: 87 passed, 0 failed.

Fixes #288 .

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95c05c111b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread tests/broker-lifecycle.test.mjs Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR prevents SessionEnd from hanging indefinitely by adding a bounded timeout to the broker shutdown RPC and ensuring the shutdown socket is cleaned up via a single guarded completion path.

Changes:

  • Add an optional timeoutMs (default 5000) to sendBrokerShutdown and resolve on timeout.
  • Centralize shutdown completion logic for data, error, close, and timeout; destroy the socket during cleanup.
  • Add a regression test for the “accepts connection but never responds” broker behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
plugins/codex/scripts/lib/broker-lifecycle.mjs Adds a bounded timeout + guarded cleanup to prevent indefinite shutdown waits.
tests/broker-lifecycle.test.mjs Introduces a regression test for an unresponsive broker shutdown socket.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/broker-lifecycle.test.mjs Resolved
Comment thread tests/broker-lifecycle.test.mjs Resolved
fireblue added a commit to fireblue/codex-plugin-cc that referenced this pull request May 17, 2026
…ai#303, openai#315)

- Drop non-JSONL garbage on app-server stdout (openai#311)
- Fix broker shutdown hang on unresponsive socket (openai#300)
- Treat jobs as inactive when their PID is dead (openai#297)
- Fix stale shared broker auth after account switch (openai#303)
- Widen /codex:review wait threshold to ~5 files / 200 lines (openai#315)
- Update test fixtures to include pid for PID-alive checks

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@agent-ptolemy

Copy link
Copy Markdown

Confirmed on plugin v1.0.4 (Node 26, macOS). Reproduces as SessionEnd hook ... failed: Hook cancelled on nearly every session exit.

Applied this fix's approach locally — bounded timeout in sendBrokerShutdown that resolves (not rejects) so handleSessionEnd still reaches teardownBrokerSession/clearBrokerSession. Verified against three paths: dead endpoint (~6ms), a broker that connects but never replies (~1s, was previously hanging to the 5s hook kill), and the normal reply path (unaffected).

One minor note: a 1s default bounds the wait well under the 5s hook budget; 5s leaves no cushion before the harness cancels the hook. Either works for the hang itself, but a sub-budget default also fixes the "Hook cancelled" message, not just the indefinite hang.

@mateusmatana

Copy link
Copy Markdown

Windows 11 / plugin 1.0.6 reproduction adds a host-bound constraint to this PR.

The current Codex host reports:

clamping SessionEnd hook timeout to 3s in .../openai-codex/codex/1.0.6/hooks/hooks.json

Using a disposable named pipe that accepts broker/shutdown, reads it, and never sends data or closes:

{"event":"still_running","ms":3218}
{"event":"exit","ms":3227,"code":null,"signal":"SIGTERM"}

Controls on the same machine: no broker, five runs, exit 0 in 1244.3/280.5/346.5/362.4/353.1 ms; responsive pipe, exit 0 in 360 ms. No real broker or job was touched.

So the proposed default timeoutMs = 5000 remains outside the host's effective 3 s budget and will still be killed before later SessionEnd cleanup. Please use an internal deadline around 1500 ms (or otherwise strictly below 3 s), keep socket cleanup idempotent, and add an integration assertion that the unresponsive-pipe path completes the whole SessionEnd cleanup before 3 s. The current hooks.json on main also still requests 5 s; setting SessionEnd to the supported 3 s would remove the clamp warning.

This is independent of Codex thread-writer-locks; the fixture never reads or writes them.

— codex-session: 01a08730 (win)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sendBrokerShutdown has no timeout — SessionEnd hook can hang indefinitely

4 participants