codex-companion: per-job state JSON stuck at status=running after task_complete; streaming .output drain disconnects mid-turn

Summary

A codex-companion task invocation can have its streaming .output file silently stop updating mid-turn while Codex itself keeps running. When the task later reaches task_complete successfully, the per-job state JSON at ~/.claude/plugins/data/codex-inline/state/<workspace>/jobs/task-<id>.json is never updated from "status": "running" to a terminal state. The PID in the JSON remains pointed at a now-dead process. The task's findings are fully recoverable from the canonical rollout JSONL at ~/.codex/sessions/YYYY/MM/DD/rollout-<timestamp>-<turn_id>.jsonl (task_complete event's last_agent_message field), but the bridge-layer signal that the task finished is lost.

Observed incident

2026-04-24 on macOS 15.0.2 / Darwin 25.3.0; plugin v1.0.2 (gitCommitSha 8e403f9d4b496b0b0aff50fa3673f889f6a22cb1); codex-cli v0.118.0.
Codex-companion job id: task-mocfoox2-12xdjb. Codex thread id: 019dbdd4-b9bf-7af3-9275-610b76fd1ec6.
Streaming .output at /private/tmp/claude-<uid>/.../tasks/bkt2c8mcy.output stopped growing at 04:54:03Z after 72 lines; final entry was a normal Command completed: ... (exit 0). No error marker, no EOF indicator, no termination log line.
Per-job log at ~/.claude/plugins/data/codex-inline/state/<workspace>/jobs/task-mocfoox2-12xdjb.log also stopped at 04:54:03Z.
Codex continued running for ~11 more minutes. Canonical rollout JSONL at ~/.codex/sessions/2026/04/24/rollout-2026-04-24T07-52-08-019dbdd4-b9bf-7af3-9275-610b76fd1ec6.jsonl has the task_complete event at 05:05:07.319Z with the full last_agent_message payload (9,888 chars, a substantive review output).

At time of incident-observation (~20 min after drain stopped), the companion state JSON still reads:

{
  "status": "running",
  "phase": "running",
  "pid": 41501,
  "threadId": "019dbdd4-b9bf-7af3-9275-610b76fd1ec6",
  "turnId": "019dbdd4-ba6c-7282-9a51-4e1e55fddecc",
  ...
}

kill -0 41501 → 1 (no such process). PID is stale.

Broker log at /var/folders/.../cxc-X56GUo/broker.log is 0 bytes — no diagnostic trail at the bridge layer either.

Consequences

Caller cannot distinguish stalled vs completed tasks via companion-visible state. Recovery requires reaching for the canonical rollout JSONL, which isn't documented as a fallback.
codex-companion status --all --json reports dead tasks as running indefinitely, polluting any "what's currently running" view. In our incident we spawned a second diagnosis task expecting to find a stalled job; the status API reported no running jobs (so the entry had already been cleared from status --all --json's view somewhere in companion logic) yet the per-job JSON file still showed running. So the inconsistency is worse: status --all --json and the on-disk JSON disagree with each other on the same job.
Cancel helpers misbehave on ghost tasks (observed adjacent symptom: codex-companion cancel <task> failed with an environment error on the stuck job — possibly a separate bug, but the state divergence is the root enabler).

Suggested fixes (any of these closes the gap)

(a) Broker-side terminal marker on pipe close. When the broker detects the Codex-CLI-side pipe has closed, write a terminal event (e.g., type=task_ended with reason=pipe_closed) to both the streaming .output AND the per-job state JSON before shutting down the job slot. This would also surface unexpected crashes, which today produce zero on-disk evidence (broker.log is 0 bytes in this incident).
(b) Companion-side liveness reconciler. Before codex-companion status returns, walk jobs/*.json, kill -0 pid for any status=running entries, and if the PID is gone, try to find the matching rollout JSONL (by threadId / turnId) and either (i) update to a terminal state from the rollout's task_complete, or (ii) mark status=unknown (pid_gone_no_rollout_match) so callers can at least tell it apart from an actively-running task.
(c) Documented recovery path. Independent of any fix, document the rollout JSONL as the canonical transcript in the companion's recovery docs — callers hitting this today have no reliable signal that a "still running" job is actually done-and-stuck-in-state, and no documented path to find the transcript.

Reproduction

I don't have a clean deterministic reproducer. The incident happened on a ~5-minute review task that did a lot of file reads and eventually produced a long last_agent_message. The drain stopped mid-task (after ~1.5 min of active tool-use). Candidate triggers: broker-side backpressure, Codex CLI burst that temporarily saturated the pipe, or an unrelated broker hiccup. Will update if reproducible.

Workaround

Recover from canonical rollout JSONL using threadId / turnId from the stuck companion state JSON:

import json

rollout_path = "~/.codex/sessions/YYYY/MM/DD/rollout-<timestamp>-<turn_id>.jsonl"
with open(rollout_path) as f:
    for line in f:
        evt = json.loads(line)
        if evt.get("type") == "event_msg" and evt["payload"].get("type") == "task_complete":
            print(evt["payload"]["last_agent_message"])

The rollout filename embeds <timestamp>-<turn_id>; matches the turnId field in the stuck companion state JSON. The threadId field in the JSON also matches the session id at the top of the rollout file (type=session_meta event's payload.id).

Round 4a: two-number framing (clean vs operational wallclock)

@travisbmiller

Cross-platform repro confirmation + Claude Code agent-wrapper amplifies the symptom

Confirming this issue on a different environment, plus an additional layer that affects how Claude Code users perceive it.

Environment

OS: Linux WSL2 (kernel 6.6.114.1-microsoft-standard-WSL2)
Node: v24.14.0
Codex CLI: 0.130.0
Plugin: v1.0.4
Workspace: ~290KB total of plan files (9 markdown documents being read by the review)

Reproduction

Same pattern as @travisbmiller's macOS report, reproduced 5 times in a single Claude Code session running adversarial review on a planning package:

Round	Input scope	Time-to-death	Last visible action
1 (initial)	~290KB across 9 files	~3min	Mid-stream `Assistant message captured`
2	~165KB across 7 files	~2min	`Command completed: head -25 ...`
3	~25KB (6 frontmatters + 1 full file)	~2min	`Assistant message: about to read full file`

In all cases:

codex-companion status --json reports status=running indefinitely
kill -0 <pid> returns failure (PID dead)
~/.claude/plugins/data/codex-openai-codex/state/<workspace>/jobs/task-*.log mtime freezes
The streaming .output file in /tmp/claude-<uid>/.../tasks/ also stops growing
But the rollout JSONL at ~/.codex/sessions/YYYY/MM/DD/rollout-*-<thread_id>.jsonl contains the full task_complete event with last_agent_message — the workaround you described works ✅

Smaller input does NOT prevent the hang in this environment — even ~25KB of file reads triggered death. So this isn't purely a backpressure issue; chat-session-state contamination (per @nixichbendeyil's #277) appears to compound it.

New angle: Claude Code agent-wrapper exits prematurely

When invoked through Claude Code via the codex:codex-rescue subagent, there's an additional symptom: the wrapping agent returns to the parent immediately after dispatching codex-companion, without waiting for completion. The agent's final message is something like:

"Waiting for Codex to finish the analysis. Will notify when complete."

But no notification ever arrives because the companion never updates state. From the parent agent's perspective, the task simply vanishes — there's no callback, no error, no completion event. The parent has no way to know that:

The companion is stuck in running state
The actual result is sitting in the rollout JSONL

This compounds the issue described here: even users who would know to fall back to the rollout JSONL don't realize they need to, because Claude Code's agent dispatching layer assumes companion state is authoritative.

Recovery worked, but is undiscoverable for end users

I successfully recovered 4 stalled review rounds using the technique described in this issue:

import json
from pathlib import Path

session_dir = Path.home() / ".codex/sessions/YYYY/MM/DD"
for f in session_dir.glob("rollout-*.jsonl"):
    with open(f) as fp:
        for line in fp:
            d = json.loads(line)
            if d.get("type") == "event_msg" and d.get("payload", {}).get("type") == "task_complete":
                msg = d["payload"].get("last_agent_message", "")
                if msg:
                    print(f"=== {f.name} ===\n{msg}\n")

This recovered the substantive review from a session whose codex-companion status showed cancelled (I cancelled the zombie before realizing the result was already there). The rollout JSONL captures the final answer even when the user has cancelled the companion task.

Endorsement of proposed fixes

Of the three fixes you suggested, (b) Companion-side liveness reconciler would have the highest impact for Claude Code users specifically, because Claude Code's agent dispatching layer treats codex-companion status as the authoritative completion signal. If the reconciler updated the JSON to terminal state when PID is dead and a matching rollout JSONL exists, the existing dispatching flow would just work without changes upstream in Claude Code.

A bonus enhancement to (c) Documented recovery path: if the docs included a one-liner script (like the snippet above) to recover the latest stalled task, end users could self-recover without needing to know Codex internals. Happy to contribute a codex-companion recover --task-id <id> subcommand if there's interest.

Severity from end-user perspective

For Claude Code workflows that depend on Codex review feedback (e.g., adversarial planning loops, multi-round code review), this is effectively a silent reliability cliff: the workflow stops producing visible output but the user has no signal that anything is wrong. In my session, I went 5 rounds before realizing the failures were systematic (not bad prompts), and only by reading #277 did I find this issue (#264) and the JSONL recovery path.

Merge pull request #5 from tgkim-openerd/feat/v2.1.0-status-tail-watch

fix(state): reap stale running jobs via OS-level PID liveness check (o…

docs(v2.0.0): sync README + MIGRATION_v2.0.md + TROUBLESHOOTING.md wi…

feat(status): /codex:status --tail + --watch with telemetry stitch (P…

Confirming this on macOS with plugin 1.0.5 and codex-cli 0.144.1, and adding one consequence I have not seen mentioned: a stale running job silently breaks --resume-last for that workspace, permanently.

resolveLatestTrackedTaskThread refuses to resume while any tracked task still looks active:

const activeTask = visibleJobs.find((job) =>
  job.jobClass === "task" && (job.status === "queued" || job.status === "running"));
if (activeTask) {
  throw new Error(`Task ${activeTask.id} is still running. Use /codex:status before continuing it.`);
}

So when a job is orphaned by the pipe-closure path described above, it is not just a cosmetic status inconsistency. Every subsequent task --resume-last in that workspace throws, pointing at a job whose worker died days earlier. Nothing surfaces the real cause, and the error text sends you to /codex:status, which reports the same phantom job as running.

What I found in local state, all with confirmed-dead pids:

job	status	started	pid alive
`task-msh16cy6-15ypsx`	running	2026-08-06	no
`task-ms9bsbzs-wsedvg`	running	2026-07-31	no
`task-ms9b2i53-u4m0o8`	running	2026-07-31	no

Two of those had been wedged for five days. Resume had been broken in that workspace the whole time.

This makes the liveness reconciliation in suggestion (b) more valuable than it may appear from the status-reporting framing alone. A cheap version that does not require broker changes: treat a tracked job whose pid no longer exists as terminal at read time, in both status and resolveLatestTrackedTaskThread. That alone unblocks resume without needing the broker to emit the missing terminal event.

Two smaller notes from the same investigation:

Job state is keyed by git workspace root, so status and result invoked from a different cwd see a different job list. Worth documenting alongside the recovery path in suggestion (c).
Jobs are also filtered by CODEX_COMPANION_SESSION_ID, so task-resume-candidate returns available: false for any Claude session that did not itself create the job. Combined with the above, resume across sessions never succeeds. That may be intentional, but it is not obvious from the CLI surface.

Method note: this came from parsing ~50 Claude Code sessions containing 322 Codex invocations, so it is one user's environment rather than an independent multi-platform repro.

Navigation Menu

Repository navigation

codex-companion: per-job state JSON stuck at status=running after task_complete; streaming .output drain disconnects mid-turn #264

Summary

Observed incident

Consequences

Suggested fixes (any of these closes the gap)

Reproduction

Workaround

Cross-platform repro confirmation + Claude Code agent-wrapper amplifies the symptom

Environment

Reproduction

New angle: Claude Code agent-wrapper exits prematurely

Recovery worked, but is undiscoverable for end users

Endorsement of proposed fixes

Severity from end-user perspective

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

codex-companion: per-job state JSON stuck at status=running after task_complete; streaming .output drain disconnects mid-turn #264

Description

Summary

Observed incident

Consequences

Suggested fixes (any of these closes the gap)

Reproduction

Workaround

Activity

luidpivetta commented on May 14, 2026

Cross-platform repro confirmation + Claude Code agent-wrapper amplifies the symptom

Environment

Reproduction

New angle: Claude Code agent-wrapper exits prematurely

Recovery worked, but is undiscoverable for end users

Endorsement of proposed fixes

Severity from end-user perspective

tagbrown commented on Aug 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions