No OS-level process liveness check — stale 'running' state after worker death (root cause for #164, #202)

Summary

When a Codex worker process dies unexpectedly (SIGKILL, OOM, broker socket disconnect, app-server crash), the companion job state file retains status: "running" indefinitely. No code path in the companion checks whether the stored PID is actually alive. This is the shared root cause behind #164 (review stuck in "running") and #202 (zombie job blocks subsequent tasks), plus a third unreported symptom: result <job-id> returning "No job found" while status <job-id> reports the full (stale) job object.

Three Symptoms, One Root Cause

Symptom	Affected command	Existing issue
`--wait` poll loop spins until timeout against a dead process	`status --wait`	#164
"Task X is still running" blocks all subsequent task calls	`task` (via `resolveLatestTrackedTaskThread`)	#202
`result <job-id>` says "No job found" while `status <job-id>` returns the job	`result` vs `status`	(new)

Root Cause Analysis

1. Detached worker spawned with zero crash monitoring

spawnDetachedTaskWorker() in codex-companion.mjs (lines 613–624):

function spawnDetachedTaskWorker(cwd, jobId) {
  const child = spawn(process.execPath, [...], {
    detached: true,      // own process group
    stdio: "ignore",     // parent never sees output
    windowsHide: true
  });
  child.unref();         // parent drops all references
  return child;
}

No child.on('exit', ...) handler
No child.on('error', ...) handler
Parent has no visibility into worker lifecycle after spawn

2. `runTrackedJob()` catch block only handles JS errors

tracked-jobs.mjs (lines 142–204): The try/catch around await runner() handles JavaScript exceptions but cannot fire when the detached worker process is killed at the OS level (SIGKILL, OOM killer, etc.). The promise simply never settles, and the state file remains at status: "running".

3. No PID liveness probe in any status/query path

The following functions read the state file and return status as-is, without checking if job.pid is alive:

Function	File	Lines
`enrichJob()`	`job-control.mjs`	161–181
`buildSingleJobSnapshot()`	`job-control.mjs`	242–254
`waitForSingleJobSnapshot()`	`codex-companion.mjs`	293–309
`resolveLatestTrackedTaskThread()`	`codex-companion.mjs`	312–320
`resolveResultJob()`	`job-control.mjs`	256–279

Zero occurrences of kill(pid, 0) or ps -p exist outside terminateProcessTree(), which is only called by handleCancel().

4. `result` vs `status` inconsistency

resolveResultJob() (line 258) filters for completed | failed | cancelled. A zombie "running" job is caught at line 269 and throws "Job X is still running." But without a reference argument, it filters to current-session jobs only (line 258: filterJobsForCurrentSession). If the stale job belongs to a previous session, result says "No finished job found" while status (which reads all jobs) returns it as "running". This creates a confusing split where both commands appear to contradict each other.

Real-World Evidence

Incident 1: adversarial-review on macOS (2026-04-11)

Job:       review-mnugvp3h-uy1ly9
Kind:      adversarial-review
Created:   2026-04-11T15:05:41.837Z
Updated:   2026-04-11T15:07:23.215Z  ← last state write
Cancelled: 2026-04-11T15:27:20.275Z  ← manual cancel 20 min later

Backend PID died silently at ~15:07.
status reported "running" for 20 minutes until manual /codex:cancel.
result <job-id> returned "No job found" during the stale window.
ps -p <pid> confirmed the process was dead (no row returned).

Incident 2: task broker disconnect (2026-04-07)

[2026-04-07T05:40:45.631Z] Codex turn interrupt failed: connect ECONNREFUSED .../broker.sock
[2026-04-07T05:40:45.632Z] Cancelled by user.

Broker socket became unreachable — same pattern: process dead, state file stale, only resolved by manual cancel.

Suggested Fix

Add a isProcessAlive(pid) utility and call it in the status/query path:

// lib/process.mjs
export function isProcessAlive(pid) {
  if (!Number.isFinite(pid)) return false;
  try {
    process.kill(pid, 0);
    return true;
  } catch (e) {
    return e.code === 'EPERM'; // exists but no permission
  }
}

Then in enrichJob() or buildSingleJobSnapshot():

import { isProcessAlive } from "./process.mjs";

export function enrichJob(job, options = {}) {
  // Auto-reap zombie jobs
  if (
    (job.status === "running" || job.status === "queued") &&
    job.pid != null &&
    !isProcessAlive(job.pid)
  ) {
    const reaped = {
      ...job,
      status: "failed",
      phase: "failed",
      pid: null,
      errorMessage: "Worker process exited without updating job status (zombie reap).",
      completedAt: new Date().toISOString()
    };
    // Persist the reap so it doesn't recur
    upsertJob(job.workspaceRoot ?? resolveWorkspaceRoot(process.cwd()), reaped);
    writeJobFile(job.workspaceRoot ?? resolveWorkspaceRoot(process.cwd()), job.id, reaped);
    return enrichJob(reaped, options); // re-enrich with corrected status
  }
  // ... existing enrichment logic ...
}

This fixes all three symptoms at the source:

status --wait will see the reap and stop polling
resolveLatestTrackedTaskThread() will no longer block on a dead task
result will find the job in failed state instead of "still running"

Environment

codex-plugin-cc: v1.0.2 (local cache), v1.0.3 available
Codex CLI: v0.118.0
Claude Code: latest
Platform: macOS Darwin 24.6.0 (Apple Silicon)
Node.js: v22.21.1

Related Issues

Review jobs stuck in 'running' after process death — no dead-PID detection #164 — Review jobs stuck in 'running' after process death (symptom 1)
Zombie running job blocks all subsequent task calls when process crashes #202 — Zombie running job blocks all subsequent task calls (symptom 2)
cancel without a job id can cancel another Claude session's hidden running task #82 — Cancel scoping across sessions (related state management issue)
[Bug] codex app-server crashes on macOS inside Claude Code sandbox (SCDynamicStore NULL panic) #141 — App-server crash on macOS (one of the triggers for unexpected process death)

fix: add --full-access flag and prevent zombie/hanging jobs

Hitting this reliably on Linux with codex-plugin-cc v1.0.3 + codex-cli v0.121.0, when the plugin is invoked as the codex:codex-rescue subagent from Claude Code.

Additional trigger I've observed (broker-side, not worker): app-server-broker.mjs itself dies during long reasoning turns or after a few failed shell-heredoc commands from the worker. The broker socket under /tmp/cxc-*/broker.sock remains on disk but nothing answers. From the companion's perspective, it's the same zombie state.

Concrete repro this morning

Dispatched codex:codex-rescue with a multi-step edit task (~250 words of brief, ~5 file writes).
Worker read ~10 files during exploratory phase, posted 2 plan messages.
Tried 3 Python heredoc commands (python - <<'PY') that failed with exit 127/1 due to shell quoting mangling.
After the 3rd failed heredoc, broker PID disappears from ps. state.json still reads status: "running".
All subsequent dispatches bounce off the lock: "Task task-mo42z5fy-kcmbsh is still running. Use /codex:status before continuing it."

Same job's state.json had pid: 3246522 — kill -0 3246522 returns ESRCH.

Second repro (different trigger, same symptom)

Yesterday: stale brokers from April 13 were still alive (PID 2208403/2208411/2208418) pointing at worktrees that no longer existed on disk. New dispatches hung on broker init because the old /tmp/cxc-WTXKGN/broker.sock etc. still accepted connections but the handshake never completed. Killing those three PIDs + removing the stale /tmp/cxc-*/ dirs unblocked new dispatches.

Workaround I'm using

Same as #202's workaround, documented here for completeness:

Edit ~/.claude/plugins/data/codex-openai-codex/state/<workspace-slug>/state.json.
For any entry with status: "running" whose pid is not in ps, flip status to "failed" and pid to null.
Also flip phase to "failed", add errorMessage and completedAt.
Same edits in jobs/<task-id>.json.
Next dispatch succeeds.

I've had to do this 2-3 times in this session alone.

Patch applied locally

I applied the suggested fix in this issue's body (added isProcessAlive to lib/process.mjs + the auto-reap block in enrichJob in lib/job-control.mjs). Syntax checks pass (node --check). Will report back if it holds on this machine over the next few days.

Happy to open a PR if that would help move this along.

fix: reconcile stale 'running' jobs when tracked PID is dead

fix: add captureTurn transport watchdog and runTrackedJob hard timeout

fix: reconcile stale 'running' jobs when tracked PID is dead

fix: add captureTurn transport watchdog and runTrackedJob hard timeout

fix(broker): P0 stability — idle timeout + PID liveness + shutdown ti…

Another instance of this symptom from Windows 11. I am not yet sure of the root cause; reporting what I have in case it helps maintainers or other folks correlate.

Environment

codex-plugin-cc 1.0.4 (git sha 11a720b)
Codex CLI 0.128.0
Windows 11 Pro 26200, Node v23.10.0
Reviews invoked via the codex:codex-rescue subagent from a Claude Code skill (the standard /codex:rescue Agent-tool path)

Symptom

Same shape this issue already documents. Three production failures over 17 hours, all on the same machine, all against the workspace C:/Users/dhite/Projects/python/WheelHouse:

task-mpdop7yn-7zxanm (pid 20032)
task-mpemqlen-c0b5sx (pid 7156)
task-mpep94bb-ro8lqn (pid 25712)

Every failed job's per-job JSON file shows status: "running", phase: "starting", with a pid that is dead by the time anyone notices. The job log file ends after exactly the two lines the parent dispatcher emits before the worker starts running:

[ts] Starting Codex Task.
[ts] Queued for background execution.

The third line, Starting Codex task thread. (emitted from runAppServerTurn inside withAppServer), is never written. None of the three failed jobs appear in the workspace state.json at all -- so the worker died after writeJobFile (line 151 of tracked-jobs.mjs) but before the immediately following upsertJob could write state.json. That places the death window inside executeTaskRun -> runAppServerTurn -> CodexAppServerClient.connect, specifically inside ensureBrokerSession or its first RPC.

Useful side observation

The same job id can be recovered. After one of the failures (task-mpepysuo-nuanul from a hand-launched repro), I re-ran node codex-companion.mjs task-worker --cwd ... --job-id task-mpepysuo-nuanul synchronously in the foreground. It picked up the stale phase: "starting" per-job JSON, reached Starting Codex task thread. within seconds, completed the turn, and wrote the terminal record cleanly. Same workspace, same prompt, same env, same job id -- the only difference is detached background vs synchronous foreground.

This is a useful data point: the death is not deterministic against the job's contents or workspace state. The same inputs can succeed when run in foreground.

What I tried to reproduce, and what happened

I wrote a small Node harness that dispatches codex-companion.mjs task --background ... via the same Bash tool the production failures used, polls the per-job JSON for pid liveness, and watches the log for the Starting Codex task thread. line. Three batches, ten iterations each:

Batch	Conditions	Result
1	trivial prompt, default sandbox	10 OK, 0 failures
2	trivial prompt, `--write` (workspace-write sandbox)	10 OK, 0 failures
3	trivial prompt, `--write`, cached broker forcibly torn down before EVERY iteration (so every dispatch takes the slow fresh-broker code path)	10 OK, 0 failures

Total 30 dispatches, zero startup deaths. Each successful dispatch went phase: starting -> finalizing -> done in 14-26 seconds.

So the failure is rare or condition-specific, not a constant-rate hazard. I cannot point at a root cause from the data I have. Plausible angles I have not been able to rule out: long multi-paragraph review prompts referencing staged files, concurrent activity from other companion processes against the same workspace, state.json size or content, or transient OS conditions (the day of the production failures, the Application event log shows a "UPS On Battery" warning -- power was noisy that day).

What this issue's proposed fix would do for us

The isProcessAlive(pid) probe and auto-reap in enrichJob from this issue's body would make any future startup death visible as status: "failed" instead of leaving a permanent status: "running" orphan. Right now our agent-review-loop wrapper has to fall back to a separate STALE_LOG heuristic to notice the worker has died, and the workspace accumulates orphan running entries that pollute /codex:status output. The probe also closes the result <job-id> vs status <job-id> inconsistency the issue body describes.

For our case the probe alone is sufficient, because the failure is rare enough that we do not need to also stop the worker from dying -- making the failure visible is enough for the wrapper to retry against a fresh job id.

I am applying the probe locally as a temporary patch to my plugin cache. Happy to convert it to a PR against this repo if maintainers prefer that flow.

Cross-references

Earlier-filed openai/codex#21813 -- "Detached task-worker exits without writing failed status on broker socket disconnect" -- same observable shape, different proximate trigger, same proposed fix. Probably worth treating as duplicates of this issue.
openai/codex-plugin-cc#228 -- foreground signal handlers. Different trigger (SIGTERM to the foreground companion). The signal handler proposed there would also catch our case if the worker happens to be signalled by the OS rather than killed outright; complementary to the probe in this issue.
openai/codex-plugin-cc#279 (Bug 2) -- same spawnDetachedTaskWorker critique from a different angle.

Follow-up to my earlier comment (#222 (comment)). The Windows-11 production failures I described above have a subtlety the issue body's proposed fix does not cover: the isProcessAlive probe in enrichJob alone would leave them permanently invisible.

The state-missing variant

runTrackedJob in tracked-jobs.mjs writes the per-job JSON via writeJobFile before it calls upsertJob to register the job in state.json. Our three production failures all died inside that window:

Per-job JSON <state-dir>/jobs/<task-id>.json is present with status: "running", phase: "starting", pid: <dead>.
state.json has no entry for that job id.

enrichJob (and every other listing path) operates on the result of listJobs(workspaceRoot), which reads state.json. A reap added at the top of enrichJob will not run for these orphans because they are never enumerated. From /codex:status they are invisible; from /codex:result <job-id> they raise "No job found"; only direct inspection of the jobs/ directory turns them up.

What we added locally on top of this issue's fix

In addition to the isProcessAlive probe and an in-enrichJob reap, we added a second sweep that walks <state-dir>/jobs/*.json directly:

function discoverAndReapStateMissingOrphans(workspaceRoot) {
  // For each <jobs-dir>/*.json:
  //   - skip if jobId is already in state.json
  //   - skip if status not in {"running", "queued"}
  //   - skip if pid missing or alive
  //   - otherwise: write status:"failed", phase:"orphaned", pid:null
  //                to both the per-job JSON AND state.json (via upsertJob)
}

The sweep runs before listing in three places so any state.json-shaped query rescues orphans first:

buildStatusSnapshot (for /codex:status listings)
buildSingleJobSnapshot (for /codex:status <job-id>)
resolveResultJob (for /codex:result <job-id>)

After the sweep, the three production-failure jobs that had been permanent invisible orphans now show up as status: "failed" with phase: "orphaned" and the error message "Worker process exited without registering in state.json (orphan reap; pid <n> no longer alive)."

Safety notes

Both reaps re-read the per-job JSON inside the writer to avoid clobbering a worker that wrote a terminal status between the kill(pid, 0) probe and the reap write.
The sweep is best-effort and silent on I/O errors; an unreadable per-job JSON file is skipped rather than fatal.
phase: "orphaned" (vs. a regular phase: "failed") acts as a marker that the sweep, not the worker itself, recorded the terminal state.

How this relates to the main proposal

The issue body's enrichJob change is necessary and catches the longer-tail case where the worker died after upsertJob succeeded. Our extra sweep catches the shorter-tail case where the worker died after writeJobFile but before upsertJob. Both reaps coexist cleanly; the sweep handles state-missing orphans first, the per-job reap handles state-known zombies on the listing path.

Happy to fold the sweep into a PR alongside the main fix if maintainers want it, or leave it as a local divergence until the broader issue lands.

fix(state): reap stale running jobs via OS-level PID liveness check (o…

docs(v2.0.0): sync README + MIGRATION_v2.0.md + TROUBLESHOOTING.md wi…

Supporting field data for this root cause, from production use of the plugin's codex-rescue path:

4 jobs in state.json marked running; all 4 PIDs dead — the oldest for 12 days. The completion-update path only fires on a clean try/catch exit, so external kill / OOM / SIGKILL leaves status: "running" forever.
A second failure shape worth covering in the same liveness sweep: a returned "completed" whose body says "still running in the background" — that's the wrapper's own stall message, not output from the underlying task.

Detection recipe we settled on before trusting any job status, which may be useful as the fix's acceptance criteria:

ps -p <pid> — dead PID + status: running ⇒ zombie.
Read the job's JSONL transcript and require a terminal assistant message near the end — no such line ⇒ no real answer, regardless of state.json.
Treat "completed"-with-stall-body as wrapper failure, not a result.

Confirming the liveness gap with a concrete data point (2026-07-13, plugin 1.0.0, Codex CLI 0.144.3, macOS/Darwin 25.5, arm64). After a detached task-worker was reaped mid-turn (see #432), status --all reported it as live indefinitely:

status: "running", phase: "verifying", elapsed: "1h 6m", pid: 47998

…while kill -0 47998 → ESRCH (process dead). A second wedged job from the same session showed the identical pattern. Nothing ever transitioned them off running; cancel was required.

process.kill(pid, 0) in the status/enrich paths (your proposed probe) would have flipped both to failed immediately. +1 — this is the highest-leverage of the layered fixes, since it neutralizes the symptom regardless of why the worker died.

Confirming this is still present in 1.0.6 (latest), and on the background task path — not only the review/rescue path most reports above cover.

Environment

codex-plugin-cc 1.0.6, macOS (Darwin 27.0.0), invoked from Claude Code via codex-companion.mjs task --background --write

Evidence (2026-08-02)

Job task-msc0avk8-tu4mns: worker died mid-turn while status <id> --json kept returning

status: "running", phase: "verifying", pid: 63224

ps -p 63224 → no row (process dead)
the job's logFile mtime was frozen from the moment of death onward
a plain poll loop on status alone spun indefinitely — exactly the Review jobs stuck in 'running' after process death — no dead-PID detection #164 shape

Recovery data point (relevant to the proposed auto-reap)

cancel <task-id> followed by task --resume-last --write on the same thread recovered cleanly: the worker writes the working tree directly, so every file change up to the crash was intact on disk, and the resumed turn continued from git status instead of restarting. So auto-reaping zombies to failed (as proposed in the issue body) loses no work; including the resumable threadId in the reap error message would make recovery a one-step task --resume-last.

Workaround until the liveness probe lands

We watch three signals instead of status alone: (1) status != running, (2) kill -0 $pid, (3) logFile mtime staleness (>300 s). Signal 3 is worth considering as a complementary heartbeat in the status output itself — it also catches the alive-but-wedged worker that a PID probe alone reports as healthy.

Field report, plugin 1.0.5, macOS (darwin 25.6.0), shared-session broker runtime.

Confirming the "Task X is still running" symptom with one forensic detail I haven't seen documented: the block can silently "self-heal" after ~12.5 minutes, which makes the bug present as random latency while the stuck record stays invisible.

Sequence from our transcripts (2026-08-17): a foreground task call was killed by the host's 2-minute command timeout (exit 143). Its job record stayed running. A retry loop then called task every ~15 s; each attempt failed with "Task … is still running" and each failure wrote a new failed job record. After exactly 50 failed attempts (12.5 min), pruneJobs (MAX_JOBS = 50, lib/state.mjs) evicted the stale running record from the sorted list. Only then did the next task call go through. So the observed user experience is "Codex does nothing for 15 minutes, then works", with no hint that a stale record was the cause.

Two more stale running records with dead PIDs from separate incidents (Aug 13, Aug 15) are still sitting in our state dir, so this happens in routine use. A process.kill(pid, 0) liveness check in resolveLatestTrackedTaskThread and buildStatusSnapshot, as proposed here, would cover all of it.

No OS-level process liveness check — stale 'running' state after worker death (root cause for #164, #202) #222

Description

Summary

Three Symptoms, One Root Cause

Root Cause Analysis

1. Detached worker spawned with zero crash monitoring

2. runTrackedJob() catch block only handles JS errors

3. No PID liveness probe in any status/query path

4. result vs status inconsistency

Real-World Evidence

Incident 1: adversarial-review on macOS (2026-04-11)

Incident 2: task broker disconnect (2026-04-07)

Suggested Fix

Environment

Related Issues

Activity

fedoseevstanislav commented on Apr 18, 2026

Concrete repro this morning

Second repro (different trigger, same symptom)

Workaround I'm using

Patch applied locally

ghbaud commented on May 21, 2026

Environment

Symptom

Useful side observation

What I tried to reproduce, and what happened

What this issue's proposed fix would do for us

Cross-references

ghbaud commented on May 21, 2026

The state-missing variant

What we added locally on top of this issue's fix

Safety notes

How this relates to the main proposal

sorcerai commented on Jul 10, 2026

boo13 commented on Jul 13, 2026

cauyxy commented on Aug 2, 2026

Environment

Evidence (2026-08-02)

Recovery data point (relevant to the proposed auto-reap)

Workaround until the liveness probe lands

jkleske commented on Aug 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. `runTrackedJob()` catch block only handles JS errors

4. `result` vs `status` inconsistency