Skip to content

codex:codex-rescue subagent hangs on poll loop; task broker runs under wrong cwd #7

New issue
New issue
Closed
#8
Closed
codex:codex-rescue subagent hangs on poll loop; task broker runs under wrong cwd#7
#8

Description

@axisrow
axisrow
opened on Jul 19, 2026

Summary

A codex:codex-rescue invocation (via Agent({ subagent_type: "codex:codex-rescue" }) from Claude Code) hung for 15+ minutes and never returned Codex's output to the session. Root cause is two-fold: (1) the rescue subagent's wait-for-completion pattern is incompatible with Claude Code's runtime sleep restrictions, and (2) the Codex task broker spawns under a wrong cwd (/private/tmp/pr523-review/..., a different repo), so even a completed run would not surface in the caller's context.

Environment

  • Claude Code: 2.1.214
  • Skill runtime: codex:rescue → subagent codex:codex-rescue → codex-companion.mjs task ...
  • Codex fork: etopro-plugins/codex-fork 1.0.6-fork.3 (sha 2ee9914), installed scope: "user" from git-subdir source https://github.com/axisrow/codex-plugin-cc.git → plugins/codex
  • Companion: ~/.claude/plugins/cache/etopro-plugins/codex-fork/1.0.6-fork.3/scripts/codex-companion.mjs
  • Companion subagent forwarding model: glm-5-turbo, effort high, task-call timeout 300000
  • Caller session cwd: /Users/axisrow/Projects/claude-code-cycle-review-skill (branch refactor/skill-linear-numbering)
  • task-resume-candidate --json --cwd <caller> returned available: false → routed to a fresh task

Reproduction

  1. From a Claude Code session in repo A, invoke the codex:rescue skill (or Agent({ subagent_type: "codex:codex-rescue" })) with a non-trivial design question.
  2. The rescue subagent drafts a Codex prompt and invokes codex-companion.mjs task --cwd <repo-A> ... in the foreground.
  3. The Bash call exceeds Claude Code's foreground-sleep / timeout rules; the runtime auto-backgrounds it.
  4. The subagent, attempting to recover, falls back to sleep 60 && tail (blocked) then until grep Thread completed; do sleep 5; done (unbounded) — which itself violates the foreground-sleep rule and hangs the session.
  5. Meanwhile 8+ app-server-broker processes accumulate with --cwd pointing at temp codex-plugin-test-* dirs under an unrelated repo path (/private/tmp/pr523-review/...), not at --cwd <repo-A>.

Observed process state

8+ app-server-broker.mjs serve --endpoint unix:.../cxc-XXXX/broker.sock --cwd /private/var/folders/.../T/codex-plugin-test-XXXX --pid-file .../cxc-XXXX/broker.pid

plus dozens of orphaned cxc-* sockets in $TMPDIR.

Impact

  • codex:rescue is unusable from a Claude Code session: the rescue subagent hangs instead of returning Codex's output.
  • Hygiene: orphaned app-server-broker processes and cxc-* sockets accumulate across sessions, leaking resources (this compounds with the broader orphan-process problem Claude Code/Codex sessions already create on this host).

Likely causes (for the fork maintainer to confirm)

  1. Wait pattern blocked by runtime. The rescue subagent's "wait for Codex thread to finish" step uses a foreground sleep / until grep loop, both incompatible with Claude Code's Bash sandbox (foreground sleep is blocked; unbounded until never exits while Codex is thinking). The companion should expose a blocking-wait primitive the subagent can call once (analogous to wait-for-reviews.sh's backgrounded-sleep pattern, or task --wait that blocks internally rather than returning a thread id the caller must poll).
  2. Broker cwd mismatch. Brokers spawn with --cwd pointing at temp codex-plugin-test-* dirs under an unrelated repo path (/private/tmp/pr523-review/...), not the --cwd the caller passed to task. The broker lookup/selection seems to reuse a stale or wrong broker instead of spawning one bound to the caller's repo. Either brokers must be keyed/selected by cwd, or a fresh broker must be spawned (and reaped) per task --cwd.
  3. No reaping. Brokers and sockets are not cleaned up on task completion / session exit.

Suggested investigation

  • In scripts/codex-companion.mjs, trace how task --cwd <X> selects (or spawns) an app-server-broker and verify the broker's --cwd matches <X>; today it does not.
  • In the codex:codex-rescue subagent's wait step, replace the sleep/until grep polling with a single blocking companion call (e.g. task --wait blocking, or a dedicated wait-for-thread <id> helper) that does not violate Claude Code's foreground-sleep rule.
  • Add broker+socket reaping on task completion and session exit.

Artifacts available on request

  • Subagent transcript: /private/tmp/claude-501/-Users-axisrow-Projects-claude-code-cycle-review-skill/68e398df-bf2d-4981-b7c8-30035d298416/tasks/a9f13e147505596f0.output (JSONL; contains the blocked-sleep tool error, the until grep fallback, the task invocation with --cwd, and the [codex] Thinking. stream).
  • ps -eo pid,etime,command | grep app-server-broker snapshot (above).

Reported from a downstream consumer (claude-code-cycle-review-skill uses this fork for its local Codex review). Happy to run further diagnostics in this environment on request.

Activity

  1. axisrow commented on Jul 19, 2026

    @axisrow
    axisrow
    on Jul 19, 2026
    OwnerAuthor

    Reopening — "all good" doesn't match what the artifacts show. Adding the rollout evidence and a correction.

    Correction on broker cwd

    My original report flagged app-server-broker processes with --cwd /private/tmp/pr523-review/... as "the task broker running under the wrong repo". That was misread. Those pr523-review / codex-plugin-test-* brokers are a separate test harness (codex-plugin-cc's own test suite spawning throwaway brokers), not the rescue call. The rescue call's Codex session actually started in the correct cwd. Withdrawing that part of the claim.

    What the rollout actually shows (the real bug)

    The rescue task produced a Codex session that never emitted a final assistant answer:

    • File: ~/.codex/sessions/2026/07/19/rollout-2026-07-19T11-16-32-019f7860-1d9f-7021-9a69-02b646b689fe.jsonl
    • session_id: 019f7860-1d9f-7021-9a69-02b646b689fe
    • cwd: /Users/axisrow/Projects/claude-code-cycle-review-skill (correct — caller repo)
    • lines: 44
    • assistant text output: none (only reasoning/thinking + tool phases; no \"type\":\"text\" assistant block)
    • model: None in the rollout (companion passed --write but no explicit --model; see below)

    So Codex ran for 44 steps, produced zero textual answer, and the rescue subagent — which was polling for a Thread completed/Turn completed marker — never received one in time (it had already been killed by the Claude runtime's foreground-sleep/until-loop restriction, as reported). Net: the caller never gets Codex's answer, which is the actual functional break, independent of which cwd the broker used.

    Two concrete asks, revised

    1. Confirm the wait contract. When codex-companion.mjs task --background returns a thread id, what is the blessed way for the caller to block until that thread emits its final answer — a single companion call that doesn't trip Claude Code's Bash foreground-sleep rule? Today the rescue subagent invents its own sleep/until grep loop and dies on it.
    2. Explain the empty rollout. 44 steps, no assistant text. Did the thread get cancelled when the subagent was killed (the broker is a separate process tree), or did it stall internally? If killed-on-subagent-death, the fix is the same wait-contract in (1): keep the caller alive until the thread is genuinely done.

    Happy to attach the 44-line rollout verbatim if useful.

  2. axisrow
    changed the title [-]codex:codex-rescue subagent hangs on poll loop; task broker runs under wrong cwd[/-] [+]rescue subagent polls after host auto-backgrounds a long foreground companion call[/+] on Jul 19, 2026
  3. axisrow
    changed the title [-]rescue subagent polls after host auto-backgrounds a long foreground companion call[/-] [+]codex:codex-rescue subagent hangs on poll loop; task broker runs under wrong cwd[/+] on Jul 19, 2026

  4. axisrow commented on Jul 19, 2026

    @axisrow
    axisrow
    on Jul 19, 2026
    OwnerAuthor

    Разбор полётов — диагноз уточнён по коду, не по симптому

    Issue написан с симптома зависшего прогона. Прогнали трассировку кода + записанный транскрипт субагента + независимый Codex-rescue прогон против этого репо (session 019f7899-a6f7-7e43-a426-65908274bf91). Три изначальных «Likely causes» проверены по коду — два опровергнуты, третий частичен.

    Что было на самом деле (записано в транскрипте)

    rescue-субагент по умолчанию запустил foreground task. Codex-ход был медленным (не дедлок), превысил Bash-tool timeout Claude Code → хост авто-бэкграунднул единственный task Bash-вызов. После этого субагент нарушил своё же правило «ровно один Bash-вызов, без polling» и импровизировал cat / sleep 60 / until grep Thread completed; do sleep 5; done — а это само по себе нарушает foreground-sleep restriction, что и дало воспринимаемый «hang». Codex-ход к тому моменту уже завершился (record 20 транскрипта содержал ответ); юзер прервал на record 22. Висел не companion, а polling субагента.

    Верификация трёх диагнозов по коду

    1. «Wait pattern = sleep/until-grep poll loop» — ОПРОВЕРГНУТО. plugins/codex/agents/codex-rescue.md:22-30 требует ровно один Bash-вызов и явно запрещает polling/monitoring (Do not inspect, read files, grep, monitor progress, poll status, fetch results...). Никакого poll-loop'а в агенте нет. Polling сгенерирован субагентом после того, как хост авто-бэкграунднул его foreground-вызов. Реальный wait — это foreground companion path, блокирующий на state.completion (codex.mjs:603).

    2. «Broker spawns under wrong cwd» — ОПРОВЕРГНУТО. resolveCommandCwd (codex-companion.mjs:157) сохраняет явный абсолютный --cwd неизменным через parser → task → app-server → ensureBrokerSession → argv брокера и OS child cwd (broker-lifecycle.mjs:115-125). Процессы с путём /private/tmp/pr523-review/... из ps-снапшота — это отдельный test-checkout, не установленный companion (production спавнит scriptPath из установленного модуля, broker-lifecycle.mjs:240-242). Транскрипт record 4 передал корректный cwd. Если cwd неверный — это caller запустил companion не из той директории без --cwd, а не баг broker selection.

    3. «No reaping» — ЧАСТИЧНО. Runtime SessionEnd зарегистрирован (hooks.json:15) и реально чистит (session-lifecycle-hook.mjs:83-114: sendBrokerShutdown + teardownBrokerSession(killProcess: terminateProcessTree) + clearBrokerSession + unlink socket). Брокеры шарятся и греются по дизайну (15-min idle timeout, app-server-broker.mjs:10-11). Чистка зависит от того, что Claude доставит SessionEnd — резкая смерть процесса может её обойти. Накопленные orphan'ы в снапшоте — это test-harness leak (адресован test-only в PR test: reap orphaned brokers and clean temp dirs on npm test exit (closes openai#163) #5), не runtime-поведение.

    Итог

    Это gap в поведенческом контракте rescue-команды/агента, а не баг брокера или runtime. Контракт не делал Bash-tool-timeout / авто-бэкграунд терминальным — субагент мог «восстановиться» вторым (polling) Bash-вызовом, который и завис.

    Фикс — PR #8

    • rescue теперь по умолчанию --background (rescue работа open-ended; foreground-ран, который хост авто-бэкграундит на timeout, нельзя превратить обратно в foreground wait — поэтому стартуем в background). --wait всё ещё явно выбирает foreground.
    • terminal-on-timeout: если единственный task Bash-вызов вернулся по host Bash-tool timeout, субагент не делает второго вызова (никаких status/result/cat/sleep/until-grep). Codex-ход продолжает работать и поднимается caller'ом через /codex:status / /codex:result — никогда субагентом.
    • commands.test.mjs обновлён под новый контракт (8/8 зелёные).

    Брокер cwd-routing, runtime SessionEnd и test-only teardown из PR #5 намеренно не тронуты — они корректны как есть.

    Закрываю этим PR'ом. Тело issue оставляю как написано — для provenance (видно, с чего начали и куда пришли).

  5. axisrow commented on Jul 19, 2026

    @axisrow
    axisrow
    on Jul 19, 2026
    OwnerAuthor

    Thanks for the code-traced diagnosis — and you're right, the hang was on my side, not the companion/broker/runtime. Acknowledging the errors in my original report:

    1. The poll-loop was mine, not the companion's. codex-rescue.md:22-30 forbids polling and mandates a single Bash call; the sleep/until grep loop was generated by the rescue subagent (i.e. my side) after the host auto-backgrounded its foreground task on the Bash-tool timeout. The companion never polled.
    2. The Codex turn had finished. I claimed "44 steps, no assistant text ⇒ the companion never returned an answer". The answer was in transcript record 20; the user interrupted at record 22. The polling — not the companion — was the wedge.
    3. (broker cwd, no-reaping — already withdrawn / partial per your trace.)

    The real gap — the rescue contract didn't make the host Bash-tool-timeout terminal, so the subagent "recovered" with a second polling call that hung — is exactly what PR #8 fixes (--background by default + terminal-on-timeout, no second call). Looks correct.

    Side-effect for me: my cycle-review skill (step 4) launches adversarial-review --background and then waits in-process — but per your contract the caller is the one who later fetches via /codex:status//codex:result. My step-4 instructions don't do that fetch, which is why today's run on PR #11 also surfaced no Codex result. I'll fix that on my side (the skill), separate from this issue.

    Closing this as resolved by PR #8. Thanks again for the careful trace.

  6. axisrow
    closed this as completedon Jul 19, 2026
  7. axisrow
    added a commit that references this issue on Jul 19, 2026

    fix(rescue): default to --background, make Bash-tool timeout terminal (…

    52bd80f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions