Summary

Fixes #95474.

• Treat Codex missing_tool_result / "native Codex tool.call without a matching tool.result" failures as local non-provider runtime/tool-execution failures.
• Reuse the existing non-provider abort path in model failover so these errors are surfaced on the current model instead of consuming configured fallback providers.
• Preserve provider-failure precedence: explicit provider metadata such as HTTP 503/429 still remains failover-eligible even if a nested local sentinel is present.

Maintainer checklist

• Fix classification: Root-cause bug fix in provider/model failover classification.
• Maintainer-ready confidence: High; the source path, sentinel string, failover gate, and regression tests line up directly with the reported missing_tool_result provider-jump behavior.
• Root cause: Codex synthetic missing_tool_result errors were local tool-execution failures, but the model failover error boundary did not recognize that source invariant, so unknown-error fallback advanced to another provider.
• Why this is root-cause fix: The change teaches the source-of-truth non-provider runtime gate to identify the stable Codex missing-tool-result sentinel before the generic unknown-error failover branch can consume fallback candidates; it does not merely hide the final error text.
• Patch quality notes: The added guard/default returns are intentional negative checks: no sentinel means the existing behavior is unchanged, while explicit provider metadata remains authoritative over nested local context. This addresses the patch-quality warning without adding a broad catch-all.
• Exact steps or command run after this patch: CI=1 node scripts/run-vitest.mjs --reporter=basic src/agents/model-fallback.test.ts src/agents/failover-error.test.ts src/agents/embedded-agent-runner/run/failover-policy.test.ts src/agents/embedded-agent-runner/result-fallback-classifier.test.ts; CI=1 node scripts/run-vitest.mjs --reporter=basic extensions/codex/src/app-server/event-projector.test.ts extensions/codex/src/app-server/run-attempt.test.ts; git diff --check HEAD~1..HEAD; PROOF_HEAD=04469ace26ffdbdf8122a6b975b23251551addba TASK_WORKTREE=/e/worktree/openclaw-worktrees/issue-95474 pnpm exec tsx /e/worktree/openclaw-issue-95474-evidence/runtime-missing-tool-result-proof.mjs.
• Observed result after fix: All commands exited 0; the new model-fallback regression rejects the original missing-tool-result error and proves only one provider/model attempt is made. The runtime-boundary proof imported the real runWithModelFallback implementation, configured openai/gpt-5.4 plus fallback anthropic/claude-opus-4-8, and observed only openai/gpt-5.4, originalErrorRethrown: true, and fallbackAttempted: false.
• What was not tested: I did not run a live 15-minute hanging native shell command against a real Codex OAuth session; the behavior is locked with source-level failover-boundary tests, Codex app-server sentinel coverage, and a runtime-boundary proof against the real fallback loop.
• Why it matters / User impact: A local hung bash or missing native tool result should not silently switch a user's active session from the requested Codex model to a different provider, because the alternate provider cannot repair the local tool failure.
• What did NOT change: No per-tool timeout policy, Codex synthetic-result creation, provider auth/billing/rate-limit/overload/server/transport classification, public API, schema, protocol payload, migration, default config, or unrelated fallback candidate ordering changed.
• Architecture / source-of-truth check: The canonical source-of-truth is isNonProviderRuntimeCoordinationError, the existing model failover boundary for local runtime conditions; this PR extends that boundary instead of adding downstream string checks in individual provider loops.
• Target test file: src/agents/failover-error.test.ts and src/agents/model-fallback.test.ts, plus existing guard coverage in src/agents/embedded-agent-runner/run/failover-policy.test.ts, src/agents/embedded-agent-runner/result-fallback-classifier.test.ts, extensions/codex/src/app-server/event-projector.test.ts, and extensions/codex/src/app-server/run-attempt.test.ts.
• Scenario locked in: Direct and nested Codex missing_tool_result errors are non-provider/local; runWithModelFallback does not make a second provider attempt; explicit provider HTTP metadata remains failover-eligible.
• Why this is the smallest reliable guardrail: A unit regression at the failover boundary proves the exact candidate-consumption invariant without needing a slow live hanging-shell fixture, while Codex tests cover generation of the sentinel this boundary consumes.
• Risk labels considered: auth-provider, session-state, message-delivery, availability, compatibility.
• Risk explanation: Merge risk is limited to the provider/model failover boundary. The compatibility/default behavior for public contracts is unchanged because no config, schema, protocol, SDK, wire format, migration, or default value is modified; the related open PR scan found other issue-linked PRs (fix(agents): prevent missing_tool_result from triggering cross-provider model fallback (#95474) #95494, fix(agents): classify missing_tool_result as tool_error to prevent cross-provider failover #95488, fix(agents): prevent Codex missing_tool_result from triggering cross-provider model fallback #95520), but this diff remains scoped to the same canonical non-provider classification invariant.
• Why acceptable: The detector is intentionally narrow to missing_tool_result and the stable Codex missing native tool-result sentence, and tests prove provider metadata still wins, so real provider failures continue to rotate/fail over as before.

What Problem This Solves

When Codex synthesizes missing_tool_result for a hung/missing local tool result, the failure is caused by local tool execution, not by the selected model provider. Before this change, the failover loop treated the resulting thrown error as an unknown candidate failure while fallback candidates remained, so a local shell/tool problem could silently switch a session from the requested model to the next configured provider.

This PR keeps that failure on the current provider/model: the user sees the local tool failure instead of OpenClaw spending cross-provider fallback candidates that cannot fix the missing local tool result.

Root Cause

Codex can synthesize a failed tool result with reason: "missing_tool_result" when a native tool call (for example bash) never returns a matching tool.result. That is a local tool-execution failure: switching from openai/gpt-5.4 to a fallback provider cannot make the missing local shell result appear.

The existing model failover guard already aborts failover for non-provider runtime coordination failures such as session write-lock contention and embedded session takeover. However, the missing_tool_result sentinel was not recognized by that guard, so runWithModelFallback reached the generic unknown-error branch and advanced to the next configured candidate.

What Changed

• Added a narrow detector for the stable Codex missing-tool-result sentinel:
  • missing_tool_result
  • native Codex tool.call without a matching tool.result
• Included that detector in the existing non-provider runtime error gate used by runWithModelFallback.
• Added regression coverage for:
  • direct and nested missing-tool-result errors being non-provider/local;
  • missing-tool-result errors not triggering a second fallback model attempt;
  • explicit provider failover metadata remaining authoritative.

Real behavior proof

• Behavior or issue addressed: A Codex missing_tool_result local tool-execution failure no longer consumes model failover candidates or jumps to a different provider; it is rethrown on the current provider/model path.
• Real environment tested: Local OpenClaw repository worktree on Windows Git Bash with Node v22.17.0 and pnpm 11.2.2. In addition to unit regression coverage, I ran a runtime-boundary proof that imports the real src/agents/model-fallback.ts from this worktree and configures a primary plus fallback provider.
• Evidence after fix: The runtime proof configured primary openai/gpt-5.4 and fallback anthropic/claude-opus-4-8, then threw the real Codex missing-tool-result-style error from the callback passed to runWithModelFallback. The observed JSON was:

{
  "proof": "missing_tool_result runtime failover boundary",
  "head": "04469ace26ffdbdf8122a6b975b23251551addba",
  "primary": "openai/gpt-5.4",
  "configuredFallback": "anthropic/claude-opus-4-8",
  "attempts": ["openai/gpt-5.4"],
  "originalErrorRethrown": true,
  "fallbackAttempted": false
}

This exits 0 and shows the local error stops after one candidate, rethrows the original error, and does not switch to the configured fallback provider. The focused failover regression tests and Codex app-server tests also exit 0 and cover the classification and sentinel production paths.

Review findings addressed

• RF-001 / RF-002 / RF-003: Fixed with the runtime-boundary log proof above. It imports the real src/agents/model-fallback.ts, configures primary openai/gpt-5.4 plus fallback anthropic/claude-opus-4-8, throws the Codex missing-tool-result-style local error through runWithModelFallback, and exits 0 with attempts: ["openai/gpt-5.4"], originalErrorRethrown: true, and fallbackAttempted: false. This is redacted terminal/log-style output for the after-fix fallback boundary; I still did not run a live long-running Codex OAuth shell hang because that live environment is unavailable in this local PR repair context.
• RF-004: Disclosed for maintainer decision. The routing change is intentional for this sentinel: missing_tool_result is local tool execution, so surfacing the current-model local error is the expected behavior instead of consuming configured fallback providers that cannot repair the missing local tool result.
• RF-005: Addressed with the narrow detector and provider-precedence regression. Matching is limited to missing_tool_result and the stable native Codex missing tool-result sentence, and the tests keep explicit provider metadata such as HTTP 503 failover-eligible even when nested context contains the local sentinel.
• RF-006: Disclosed for maintainer decision. The related open PR scan found fix(agents): prevent missing_tool_result from triggering cross-provider model fallback (#95474) #95494, fix(agents): classify missing_tool_result as tool_error to prevent cross-provider failover #95488, and fix(agents): prevent Codex missing_tool_result from triggering cross-provider model fallback #95520; this PR remains a focused source-of-truth failover-boundary fix with proof added, so maintainers can compare it against competing fixes without extra scope in this branch.

Verification

Run from the repository worktree after formatting and committing:

CI=1 node scripts/run-vitest.mjs --reporter=basic src/agents/model-fallback.test.ts src/agents/failover-error.test.ts src/agents/embedded-agent-runner/run/failover-policy.test.ts src/agents/embedded-agent-runner/result-fallback-classifier.test.ts
# exit 0

CI=1 node scripts/run-vitest.mjs --reporter=basic extensions/codex/src/app-server/event-projector.test.ts extensions/codex/src/app-server/run-attempt.test.ts
# exit 0

git diff --check HEAD~1..HEAD
# exit 0; no whitespace errors

PROOF_HEAD=04469ace26ffdbdf8122a6b975b23251551addba TASK_WORKTREE=/e/worktree/openclaw-worktrees/issue-95474 pnpm exec tsx /e/worktree/openclaw-issue-95474-evidence/runtime-missing-tool-result-proof.mjs
# exit 0; attempts=["openai/gpt-5.4"], originalErrorRethrown=true, fallbackAttempted=false

Regression Test Plan

• Target tests: src/agents/failover-error.test.ts and src/agents/model-fallback.test.ts cover the new invariant; src/agents/embedded-agent-runner/run/failover-policy.test.ts, src/agents/embedded-agent-runner/result-fallback-classifier.test.ts, extensions/codex/src/app-server/event-projector.test.ts, and extensions/codex/src/app-server/run-attempt.test.ts guard adjacent policy and Codex behavior.
• Regression scenario: Direct string/Error and nested { result: { reason: "missing_tool_result" } } shapes are classified as local/non-provider, and a configured fallback chain does not make a second provider attempt for that local failure.
• Guardrail rationale: This is the smallest reliable guardrail because it proves the exact source-of-truth failover boundary that caused provider jumping, while existing Codex tests cover sentinel production.

Merge risk

• Risk labels considered: auth-provider, session-state, message-delivery, availability, compatibility.
• Risk explanation: The changed code sits in provider/model failover classification, so a bad match could suppress a real provider failure. The implementation limits matching to missing_tool_result and the stable Codex missing native tool-result sentence, and tests keep explicit HTTP/provider metadata authoritative.
• Why acceptable: No public contract, config, schema, protocol, SDK, wire format, migration, or default value changes. The related open PR scan found other issue-linked PRs (fix(agents): prevent missing_tool_result from triggering cross-provider model fallback (#95474) #95494, fix(agents): classify missing_tool_result as tool_error to prevent cross-provider failover #95488, fix(agents): prevent Codex missing_tool_result from triggering cross-provider model fallback #95520), but this PR remains scoped to the same canonical non-provider classification invariant.

Not Changed

• No changes to Codex's synthetic missing_tool_result creation path.
• No per-tool timeout policy changes; this PR only prevents local tool failures from changing provider/model.
• No changes to failover behavior for real provider/auth/billing/rate-limit/overload/server/transport failures.

Risk

Low/medium. The detection is intentionally narrow to Codex's stable missing-tool-result sentinel and only affects the already-existing non-provider abort path. The main regression risk would be accidentally suppressing a provider failure that happens to wrap a local sentinel, so the tests keep explicit provider metadata authoritative.