Preserve Codex output after missing turn completion by 100yenadmin · Pull Request #99217 · openclaw/openclaw

100yenadmin · 2026-07-02T19:50:05Z

Summary

Canonicalizes the missing Codex turn/completed recovery started in #99167 and extended here. Thanks to @100yenadmin for the original report, reproduction evidence, and implementation.

Recover a completed assistant reply only when it is the newest valid terminal candidate and no later item, tool, delivery, spawn, cron, media, heartbeat, or source-reply work exists.
Preserve typed-before-raw Codex event ordering without discarding a genuinely newer id-less post-tool reply.
Keep recovery fail-closed for partial output, active items, replay-invalid attempts, non-completion watchdogs, delayed echoes, and terminal side effects.
Freeze accepted completion and abort ownership through final hooks and reply delivery so later /stop, restart, or subagent controls cannot misclassify a finalizing run.
Cover the Codex projector and watches plus embedded runner, fallback, reply registry, abort commands, dispatch, gateway shutdown, and subagent control surfaces.

This mitigates lost final replies when Codex emits a genuine terminal assistant item but omits turn/completed. It does not claim to fix the upstream Codex event omission.

What Problem This Solves

Completion and cancellation cross several asynchronous boundaries. A narrow timeout fallback can accidentally revive stale pre-tool text, discard newer id-less output, duplicate backend cancellation, or turn a successfully recovered reply into an aborted result while final hooks or delivery are still running.

The canonical fix makes candidate freshness, terminal side effects, abort ownership, and finalization ownership explicit. Recovery succeeds only for the newest completed assistant result after all relevant work has settled.

P0 Regression Audit

The takeover review found and fixed three proof or runtime gaps beyond the contributor head:

The fallback E2E harness passed the abort signal only to the inner attempt, allowing a false fallback summary. The harness now matches production signal ownership.
A recovered client-close success still consulted stale pre-hook state, so /stop during agent_end could convert accepted success into abort. The accepted outcome is now frozen through finalization.
Real Codex raw assistant messages omit ResponseItem::Message.id. Pending typed/raw echo pairing could therefore discard a newer id-less post-tool final and return stale pre-tool text. Pairing is invalidated only when later work truly supersedes the candidate, with focused positive and negative regressions.

Repeated autoreview is clean after these fixes. A full gateway-project A/B run showed the same three suite-order/environment failures on this branch and exact base; the affected tests passed in isolation on both, so they are not attributed to this PR.

Evidence

Published PR head: 116bb037d396d8bf4623c0c411513c67b742acef.
Tested rebased lineage head: daf6d5ce3ab0e6c2ccb8a0bbd90e5129b25b09f9.
Canonical base: 36dd9ee3c3c188cc0619f0a3277c64c4b96a9615.
Published and tested Git tree: 681183d911cee5b7348da7cecb215c9f9c258bbf.
Exact 56-file content hash: 7f29539a99b3a54e92601eb37a788e20679a81899cc1380ddce5b8e6a8294a10.
Blacksmith Testbox through Crabbox regression matrix:
- Provider/id: blacksmith-testbox / tbx_01kwkngrb4w4pp0rq60b4wxh21.
- Actions run: 28652114082.
- Result: 28 test files and 1,510 tests passed across unit-fast, gateway, auto-reply, agents, E2E, and Codex extension projects.
Blacksmith Testbox through Crabbox changed checks/build:
- Provider/id: blacksmith-testbox / tbx_01kwkngvxbgj14fkv9fr6smfw6.
- Actions run: 28652117396.
- Command: pnpm check:changed && pnpm build.
- Result: passed.
Live AWS Crabbox:
- Provider/run/lease: aws / run_55458f5381b9 / cbx_40a108919c40.
- Slug/type: coral-lobster / c7a.32xlarge.
- Codex CLI: 0.142.4.
- Exact returned token: CRABBOX-CANONICAL-509F22.
- Fault markers: CRABBOX_FAULT_TOOL_OBSERVED, CRABBOX_FAULT_INJECT pre_tool_terminal_assistant, CRABBOX_FAULT_OBSERVED post_tool_typed_final, CRABBOX_FAULT_DROP method=turn/completed, and CRABBOX_PROOF missing_turn_completed=pass.
- Lease cleanup: stopped successfully.

Behavior addressed: Preserve the newest completed Codex assistant reply when turn/completed is missing, without stale-candidate revival, newer-output suppression, duplicate cancellation, or abort/finalization ownership regressions.

Real environment tested: AWS Crabbox Linux c7a.32xlarge with real Codex app-server authentication and Codex CLI 0.142.4; Blacksmith Testbox through Crabbox for the full changed-surface matrix and repository gates.

Exact steps or command run after this patch: Sync and hash the exact rebased 56-file overlay, launch the live OpenClaw Codex app-server path, complete a baseline turn, call the real sessions_list tool with limit=1 and includeLastMessage=false, inject stale pre-tool assistant text, preserve the genuine post-tool assistant completion, drop only turn/completed, and wait for OpenClaw recovery. The published PR head was then verified to have the identical Git tree.

Evidence after fix: The real tool call completed, the genuine post-tool reply returned CRABBOX-CANONICAL-509F22, the proxy logged the stale-candidate injection and deliberate turn/completed drop, and the run emitted CRABBOX_PROOF missing_turn_completed=pass.

Observed result after fix: OpenClaw returned the newest post-tool assistant text instead of stale I will check. or turn_completion_idle_timeout; the run ended successfully and the one-shot lease stopped.

What was not tested: An organic upstream Codex omission without fault injection, live macOS/Windows execution, or a raw-only post-tool completion. In the observed real Codex tool turn, the post-tool final used the typed assistant lane; the id-less raw pairing path is covered by focused protocol regressions.

Recovery policy change: 1 timeout outcome changed. A Codex missing-terminal timeout can now deliver completed assistant text instead of the generic warning when the fail-closed predicate proves no unsafe later activity.
Abortability contract: 1 internal backend abortability hook added. Reply and gateway abort paths now rely on finalization-aware abortability before accepting session-state behavior.

Stored data model
Persistent data-model change detected: serialized state: src/agents/subagent-control.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] Merging changes a P1 message-delivery state machine: an incorrect recovery predicate could deliver stale or unsafe assistant text, or fail to release a safe final answer.
[P1] Merging changes finalization abort ownership for /stop, restart, gateway abort, and subagent controls; maintainers should accept that finalizing successful replies reject late aborts instead of persisting them as aborted partials.
[P1] The live proof uses deliberate fault injection for the missing terminal event; the broader upstream Codex event-emission root cause and organic production data points remain tracked by the linked issues.

Maintainer options:

Accept bounded recovery semantics (recommended)
Maintainers can accept the fail-closed predicate and finalization-aware abort contract as the intended mitigation for captured safe assistant text when Codex omits turn/completed.
Hold for broader live matrix
If maintainers want more confidence before changing this lifecycle behavior, pause merge until the same recovery is proven on additional platforms or an organic upstream omission reproduces under live Codex.
Narrow the recovery contract further
A repair path would be to reduce the recovery window or add additional invariant checks if maintainers decide the current predicate is too permissive for message delivery.

Next step before merge

No automated repair is indicated; this open PR needs maintainer acceptance and merge checks, not a separate fix PR.

Security
Cleared: The diff changes runtime/session logic, docs, and tests without package, lockfile, workflow, dependency-source, secret, or downloaded-code execution changes.

Review details

Best possible solution:

Land after maintainer accepts the bounded recovery and finalization semantics; keep the upstream Codex terminal-event root cause tracked separately.

Do we have a high-confidence way to reproduce the issue?

Yes for review purposes: current main lacks the recovery path, upstream Codex defines turn/completed as the terminal event, and PR proof injects the missing event and observes recovered output, though I did not rerun live proof in this read-only review.

Is this the best way to solve the issue?

Yes: because upstream still owns reliable turn/completed, this is the best bounded OpenClaw mitigation, recovering only latest quiescent terminal assistant text while preserving fail-closed behavior for later work or side effects.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 8abc633cf8e2.

Label changes

Label justifications:

P1: The PR targets a current Codex-backed agent regression that can suppress final answers and leave session state ambiguous for real users.
merge-risk: 🚨 message-delivery: The diff changes when OpenClaw delivers captured assistant text instead of a missing-terminal warning after Codex omits turn/completed.
merge-risk: 🚨 session-state: The diff changes abort and finalization ownership for active/finalizing reply and agent runs across gateway and embedded-runner paths.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): PR body and maintainer comments provide after-fix AWS Crabbox live Codex app-server proof plus Testbox checks/build, including a real tool call, deliberate turn/completed drop, and recovered token observation.
proof: sufficient: Contributor real behavior proof is sufficient. PR body and maintainer comments provide after-fix AWS Crabbox live Codex app-server proof plus Testbox checks/build, including a real tool call, deliberate turn/completed drop, and recovered token observation.

Evidence reviewed

PR surface:

Source +944, Tests +3089, Docs +7. Total +4040 across 56 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	28	1174	230	+944
Tests	27	3202	113	+3089
Docs	1	10	3	+7
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	56	4386	346	+4040

Acceptance criteria:

[P1] Review existing live proof and relevant CI/Testbox results on the PR before merge.
[P1] If maintainers want extra assurance, rerun the Codex app-server fault-injection recovery proof against the current head SHA.

What I checked:

Repository policy applied: Full root AGENTS.md and scoped AGENTS.md guidance were applied, including the Codex dependency-source gate and review of gateway, agent, Codex app-server, docs, and sibling surfaces. (AGENTS.md:1, 8abc633cf8e2)
Upstream Codex terminal contract: The sibling Codex source documents that each app-server turn starts with turn/started and ends with turn/completed, and its FAQ says a turn is complete only when turn/completed arrives. (../codex/codex-rs/app-server/README.md:1353, da4c8ca57d40)
Main lacks bounded recovery: Current main only checks that a resolved final assistant item is completed; it does not have the PR's latest-completed-terminal-assistant recovery method or missing-terminal timeout release gate. (extensions/codex/src/app-server/event-projector.ts:222, 8abc633cf8e2)
Recovery gate is fail-closed: The PR recovers only when the assistant completion is complete, no active app-server request/item or dynamic tool/finalization blocker remains, no terminal notification arrived, and cancellation did not occur. (extensions/codex/src/app-server/run-attempt.ts:3106, be0b446fcc2b)
Event ordering is explicitly covered: The PR tracks the latest completed terminal assistant state, handles raw id-less assistant messages, and rejects candidates superseded by later or earlier-active completions. (extensions/codex/src/app-server/event-projector.ts:229, be0b446fcc2b)
Regression coverage exercises safe and unsafe recovery: Tests cover successful recovery from a missing-terminal timeout, id-less raw echo behavior, and fail-closed cases for raw tool calls, active items, out-of-order completions, later assistant items, and newer empty assistant output. (extensions/codex/src/app-server/run-attempt.turn-watches.test.ts:562, be0b446fcc2b)

Likely related people:

fuller-stack-dev: Recent work and PR commits touch Codex missing-terminal recovery, replay behavior, and the same app-server terminal-output paths. (role: recent area contributor and branch takeover author; confidence: high; commits: 77a682c5def2, 9af84baf0d36, b7f7ae9afa97; files: extensions/codex/src/app-server/run-attempt.ts, extensions/codex/src/app-server/event-projector.ts, src/auto-reply/reply/reply-run-registry.ts)
steipete: Recent current-main history includes terminal outcome ordering, prose terminal classifiers, reply timer, gateway, and embedded-runner lifecycle work adjacent to this state machine. (role: recent adjacent contributor; confidence: medium; commits: a09f6b1b2787, 0314819f918a, 490a155; files: extensions/codex/src/app-server/event-projector.ts, src/auto-reply/reply/reply-run-registry.ts, src/gateway/chat-abort.ts)
omarshahine: Recent current-main history includes Codex message-tool-only source reply completion work on the same app-server projection and attempt paths. (role: recent Codex source-reply contributor; confidence: medium; commits: 9b9a124cc520; files: extensions/codex/src/app-server/event-projector.ts, extensions/codex/src/app-server/run-attempt.ts)
Eva: Recent current-main history includes active non-streaming run and Telegram active-run steering work that intersects reply/run lifecycle behavior. (role: recent adjacent contributor; confidence: medium; commits: 8d9a7ab; files: src/auto-reply/reply/reply-run-registry.ts, src/agents/embedded-agent-runner/runs.ts, extensions/codex/src/app-server/run-attempt.ts)
jalehman: Current-main history shows transcript mirror refactor work on the Codex event projector path touched by the PR. (role: adjacent refactor contributor; confidence: low; commits: 258b83c4386c; files: extensions/codex/src/app-server/event-projector.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

100yenadmin · 2026-07-02T20:01:48Z

Added the after-fix behavior proof requested in the PR body. It now includes the PR-head command output summary for:

safe completion-watch turn_completion_idle_timeout assistant text delivery
non-completion/progress assistant output staying on the timeout error path
mutating/side-effect timeout cases staying fail-closed
Codex app-server turn-watch boundary scenarios

@clawsweeper re-review

clawsweeper · 2026-07-02T20:01:51Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

100yenadmin · 2026-07-02T20:04:57Z

PR body is now updated for current head c53227734a6337507d4f538e049a7523e04e2ebf with the after-fix proof excerpt and notes that the tree is unchanged from the prior proof run.

@clawsweeper re-review

100yenadmin · 2026-07-02T20:18:30Z

Fixed in ab0d13485e39b58d59b3e10d2df676c160349494. The recovery predicate now treats attempt.itemLifecycle.activeCount > 0 as unsafe for assistant-text recovery, so completion-watch assistant output is surfaced only when no Codex item remains active. Test: src/agents/embedded-agent-runner/run.codex-app-server-recovery.test.ts:does not deliver assistant output while Codex item lifecycle is active (RED before guard, GREEN after guard). Focused proof also reran run.codex-app-server-recovery.test.ts 13/13, extensions/codex/src/app-server/run-attempt.turn-watches.test.ts 66/66, and git diff --check clean.

@clawsweeper re-review

clawsweeper · 2026-07-02T20:18:33Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

100yenadmin · 2026-07-02T20:40:32Z

Proof/status update for current head e82b799baf5a9d6360b269218c75c3f68dc7767a:

The patch now includes code, focused recovery tests, and a docs contract for the safe assistant-text recovery boundary.
Local focused proof remains green:
- Embedded runner recovery: 1 passed, 13 passed.
- Codex app-server turn-watch boundary: 1 passed, 66 passed.
- Whitespace/docs check: git diff --check HEAD~1..HEAD passed.
Mantis Telegram Desktop proof run https://github.com/openclaw/openclaw/actions/runs/28619264000 failed because that visual lane could not synthesize the required Codex app-server missing turn/completed condition. The Mantis agent wrote a capture-failure manifest instead of fabricating evidence, so I am treating that as a proof-harness mismatch rather than a recovery-predicate failure.

Proof boundary: this PR proves OpenClaw recovery behavior for the missing-turn/completed completion-timeout class. It does not claim the upstream Codex app-server root emission bug is fixed.

github-actions · 2026-07-03T03:16:26Z

Dependency graph guard cleared

This PR no longer has blocked dependency graph changes. A future dependency graph change requires a fresh /allow-dependencies-change comment after the guard blocks that new head SHA.

Current SHA: be0b446fcc2b0cbeb2f28f17872cdddf63da24fa

fuller-stack-dev · 2026-07-03T03:18:41Z

Maintainer takeover complete. I kept the contributor history, updated the branch through GitHub's non-force update path, and applied the canonical tree as 9af84baf0d36714e0d48df60c93aa2d86896d8a7.

Canonical changes:

Recovery now requires a terminal assistant item with no later work; typed contributor-rewritten text wins, with raw assistant echo only as a blank-text fallback.
Any later raw item, active item, partial output, replay-invalid state, tool/delivery/spawn/cron/media/heartbeat/source-reply evidence, or non-completion watchdog keeps the failure path.
Reply delivery freezes the abort outcome through finalization. /stop, natural-language aborts, restarts, and subagent stop/steer no longer persist a cutoff or report a finalizing unabortable run as killed/replaced.
Regression coverage now spans the Codex projector/run watches, embedded runner, reply registry, abort commands, dispatch, restart ownership, and subagent controls.

Updated proof:

Post-rebase focused tests: 722 passed (573 auto-reply, 55 agents, 94 Codex extension).
Autoreview: no actionable findings.
Blacksmith Testbox tbx_01kwjyserh9s77c8965pwchakt (quick-prawn), Actions run 28635530660: pnpm check:changed && pnpm build passed.
Live AWS Crabbox run_cbd01421905b, lease cbx_91955196cd34 (amber-barnacle), Codex CLI 0.142.4: the harness passed through the genuine rawResponseItem/completed, dropped only turn/completed, returned CRABBOX-CANONICAL-72D5CA, and emitted CRABBOX_PROOF missing_turn_completed=pass.
The live/Testbox proof ran on d512735f295f605fe6f790fe85b3ac0bd61cde6b; all PR-touched files are byte-identical on the final head. Later rebases only added unrelated generated locale commits from main.

Conflict status: the contributor head had not moved during takeover. main advanced repeatedly through generated locale commits, so I refreshed the PR branch and reapplied the canonical tree. The final remote tree matches the local prepared tree, contains current main, and GitHub reports the PR mergeable.

fuller-stack-dev · 2026-07-03T09:54:28Z

Maintainer takeover complete. I rebased the contributor work onto current main with zero conflicts and completed a P0 regression audit of the completion/cancellation state machine.

Additional audit fixes:

Aligned the fallback E2E abort harness with production ownership so it cannot produce a false fallback summary.
Froze recovered client-close success through final hooks and delivery, preventing /stop during agent_end from converting accepted success into abort.
Preserved genuinely newer id-less post-tool raw replies while still rejecting delayed echoes and raw output emitted during active later work.

Final review and proof:

Repeated autoreview: clean, no accepted/actionable findings.
Published PR head/base: 116bb037d396d8bf4623c0c411513c67b742acef / 36dd9ee3c3c188cc0619f0a3277c64c4b96a9615.
Tested rebased lineage head: daf6d5ce3ab0e6c2ccb8a0bbd90e5129b25b09f9.
Published and tested Git tree: 681183d911cee5b7348da7cecb215c9f9c258bbf.
Exact 56-file content hash: 7f29539a99b3a54e92601eb37a788e20679a81899cc1380ddce5b8e6a8294a10.
Regression matrix: 28 files, 1,510 tests passed on Testbox tbx_01kwkngrb4w4pp0rq60b4wxh21, Actions 28652114082.
Changed checks/build: passed on Testbox tbx_01kwkngvxbgj14fkv9fr6smfw6, Actions 28652117396.
Live AWS Crabbox: run run_55458f5381b9, lease cbx_40a108919c40, slug coral-lobster, Codex CLI 0.142.4.
Live result: real sessions_list tool call, stale pre-tool assistant injection, genuine post-tool final CRABBOX-CANONICAL-509F22, dropped turn/completed, and CRABBOX_PROOF missing_turn_completed=pass.
AWS lease cleanup completed successfully.

The contributor PR head remained unchanged during review. The guarded takeover published the verified tree as a signed commit, then GitHub merged current main into the PR branch without rewriting contributor history. The final PR tree is byte-identical to the fully tested tree. The full gateway-project A/B comparison reproduced the same three suite-order/environment failures on branch and base; those tests passed in isolation on both.

@clawsweeper re-review

clawsweeper · 2026-07-03T09:54:31Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

shannon0430 · 2026-07-03T11:33:34Z

Adding a production data point that matches this issue's scope ("completed output lost + run left ambiguously active"), with two details that may help narrow it down: the loss reproduces without any native thread rotation, and the stuck-session recovery is what ultimately discards the completed reply.

Environment: OpenClaw 2026.6.11 (e085fa1), agentRuntime=codex (openai/gpt-5.5, Codex OAuth), Slack channel-thread sessions, messages.queue.byChannel.slack: "followup". All identifiers below are sanitized.

Timeline (2026-07-03, single incident, all evidence from transcripts/trajectory/diagnostic logs)

t	event
T+0	user message M1 arrives → embedded run R1 starts (`prompt.submitted`)
T+62s	second user message M2 arrives mid-run → enqueued (followup mode), `queueDepth=1`
T+108s	R1 completes normally: trajectory has `model.completed` + `session.ended`, no `turn.aborted`; the final assistant reply is fully written to the session transcript (`stopReason: "stop"`)
—	No Slack delivery ever happens for R1's reply: no `delivered reply` log line for it, and the message is absent from the thread (verified via API readback). The session stays `activeWorkKind=embedded_run` with no further progress
T+11m	stuck-session recovery fires (`queueDepth>0` ∧ no-progress ≥ 360s) and force-clears the run — the completed-but-undelivered reply is now permanently gone
T+12m	the followup queue drains M2 → M2's reply generates and delivers fine. Net user experience: M1 was silently ignored

Sanitized diagnostic excerpts (JSON-lines gateway diagnostic log):

10:15:34Z keep_lane activeWorkKind=embedded_run reason=active_reply_work
10:21:34Z reclaiming stale active reply work age=360s queueDepth=1
10:21:49Z abort_embedded_run aborted=true drained=false released=0 forceCleared=true

Observations

Rotation-independent: zero codex native thread rotations in the incident window (an earlier occurrence on 2026-07-02 did coincide with rotations, so rotation appears incidental, not causal).
Delivery pipeline itself was healthy: a different session delivered to the same channel within the same minute.
Trigger correlation: over a 2.6-day window we counted 15 quick-succession bursts (2+ inbound ≤120s apart in one channel); 3 of them (20%) lost a reply — always the first message's reply. Every observed loss (including single-message cases) had one thing in common: another message arrived while the run was active (queueDepth>0 at completion time).
Machine-detectable signature: delivered turns write the final assistant text twice into the transcript (the original + a delivery-mirror / channel-final copy ~1s later). Orphaned turns have only the single original. We now use this in an external watchdog to detect the loss within minutes.

Suggestion (aligns with this issue's "freeze accepted completion / abort ownership" direction)

Before stuck-session recovery force-clears an embedded run, it could check the transcript for a completed final assistant message that has no delivery mirror — and either hand it back to the reply-delivery path or at least surface it explicitly, instead of discarding it as part of the abort. In our incident the answer existed for 11 minutes before the recovery destroyed it.

Happy to share more sanitized trajectory/diagnostic detail if useful.

fuller-stack-dev · 2026-07-03T12:07:59Z

Merged via squash.

Prepared head SHA: be0b446fcc2b0cbeb2f28f17872cdddf63da24fa
Landed commit: 414ecd2b96899d955f10e93ca166915bdd9d24ad

Preserve Codex output after missing turn completion

d99de54

openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jul 2, 2026

openclaw-barnacle Bot added the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 2, 2026

openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 2, 2026

fuller-stack-dev self-assigned this Jul 2, 2026

100yenadmin force-pushed the fix/codex-turn-completed-timeout-proof branch from b5d5a2a to c532277 Compare July 2, 2026 20:02

100yenadmin force-pushed the fix/codex-turn-completed-timeout-proof branch from c532277 to ab0d134 Compare July 2, 2026 20:17

clawsweeper Bot added the mantis: telegram-visible-proof Mantis should capture Telegram visible proof. label Jul 2, 2026

clawsweeper Bot had a problem deploying to qa-live-shared July 2, 2026 20:26 Failure

100yenadmin force-pushed the fix/codex-turn-completed-timeout-proof branch from ab0d134 to e82b799 Compare July 2, 2026 20:39

openclaw-barnacle Bot added the docs Improvements or additions to documentation label Jul 2, 2026

clawsweeper Bot removed mantis: telegram-visible-proof Mantis should capture Telegram visible proof. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 2, 2026

openclaw-barnacle Bot added commands Command implementations channel: feishu Channel integration: feishu extensions: device-pair channel: irc channel: qqbot channel: qa-channel Channel integration: qa-channel extensions: qa-lab labels Jul 3, 2026

fuller-stack-dev added 3 commits July 2, 2026 20:17

fix: narrow Codex completion-timeout output recovery

f860905

Merge branch 'main' into fix/codex-turn-completed-timeout-proof

b235426

test(codex): narrow binding path guard

9af84ba

fuller-stack-dev added 2 commits July 3, 2026 02:53

fix(codex): preserve id-less post-tool replies

b7f7ae9

Merge branch 'main' into fix/codex-turn-completed-timeout-proof

116bb03

fuller-stack-dev added 2 commits July 3, 2026 05:36

Merge remote-tracking branch 'origin/main' into pr-99217-prep

17248fe

Merge remote-tracking branch 'origin/main' into pr-99217-prep

be0b446

clawsweeper Bot mentioned this pull request Jul 3, 2026

fix(discord): guard message-tool-only final delivery #84485

Closed

github-actions Bot mentioned this pull request Jul 3, 2026

📡 Upstream Digest — 2026-07-03 13:45 UTC curtismercier/openclaw-mods#1197

Open

Uh oh!

Uh oh!

Conversation

100yenadmin commented Jul 2, 2026 • edited by fuller-stack-dev Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What Problem This Solves

P0 Regression Audit

Evidence

Related

Uh oh!

chatgpt-codex-connector Bot commented Jul 2, 2026

Uh oh!

clawsweeper Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

100yenadmin commented Jul 2, 2026

Uh oh!

clawsweeper Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

100yenadmin commented Jul 2, 2026

Uh oh!

100yenadmin commented Jul 2, 2026

Uh oh!

clawsweeper Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

100yenadmin commented Jul 2, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency graph guard cleared

Uh oh!

fuller-stack-dev commented Jul 3, 2026

Uh oh!

fuller-stack-dev commented Jul 3, 2026

Uh oh!

clawsweeper Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shannon0430 commented Jul 3, 2026

Timeline (2026-07-03, single incident, all evidence from transcripts/trajectory/diagnostic logs)

Observations

Suggestion (aligns with this issue's "freeze accepted completion / abort ownership" direction)

Uh oh!

fuller-stack-dev commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

100yenadmin commented Jul 2, 2026 •

edited by fuller-stack-dev

Loading

clawsweeper Bot commented Jul 2, 2026 •

edited

Loading

clawsweeper Bot commented Jul 2, 2026 •

edited

Loading

clawsweeper Bot commented Jul 2, 2026 •

edited

Loading

github-actions Bot commented Jul 3, 2026 •

edited

Loading

clawsweeper Bot commented Jul 3, 2026 •

edited

Loading