Skip to content

Gateway restart can leave Codex custom tool call without output #84727

Description

@pfrederiksen

Summary

A gateway restart can abort an in-flight Codex app-server custom tool call after Codex has persisted the custom_tool_call item but before OpenClaw records the matching custom_tool_call_output. The next Codex app-server resume keeps loading that invalid native transcript and repeatedly logs:

Custom tool call output is missing for call id: call_<redacted>

Proof from a live OpenClaw 2026.5.19 update

During the local update/restart, the gateway drain timed out while an embedded run was active:

2026-05-20T21:31:38.479Z WARN still draining 2 active task(s) and 1 active embedded run(s) before restart
2026-05-20T21:31:38.502Z WARN wait for active embedded runs timed out...
2026-05-20T21:31:38.506Z WARN active embedded run drain grace reached; aborting active run(s) before restart
2026-05-20T21:31:38.522Z WARN drain timeout reached; proceeding with restart

After restart, the app-server repeatedly reported the missing output. Redacted local log proof:

custom missing count: 29
first: 2026-05-20T21:33:12.565Z codex app-server stderr: ... ERROR codex_core::util: Custom tool call output is missing for call id: call_<redacted>
last:  2026-05-20T23:12:07.703Z codex app-server stderr: ... ERROR codex_core::util: Custom tool call output is missing for call id: call_<redacted>

The persisted Codex native transcript confirmed the orphan:

$OPENCLAW_HOME/agents/<agent-id>/agent/codex-home/sessions/2026/05/20/rollout-<timestamp>-<thread-id>.jsonl
line 28: response_item payload.type=custom_tool_call call_id=call_<redacted> name=exec
(no matching custom_tool_call_output existed before manual local repair)

The gateway itself recovered and stayed healthy, so this is specifically transcript hygiene after restart abort rather than gateway liveness:

GET /readyz -> 200 {"ready":true,...}
GET /healthz -> 200
dashboard localhost -> 200

Expected behavior

OpenClaw should not leave Codex-owned native transcripts in a state that Codex rejects on every future resume. On restart abort/recovery, it should either:

  • avoid interrupting between custom tool call persistence and output persistence, or
  • repair the bound Codex rollout transcript by inserting a synthetic interrupted/cancelled custom_tool_call_output for orphaned custom_tool_call records before resume.

Impact

This produces repeated scary ERROR stderr entries in the gateway log even though the gateway is healthy. It also risks making future Codex app-server resume behavior depend on an invalid provider-owned transcript until someone manually edits the rollout JSONL.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions