Skip to content

fix(failover): fallback on replay-safe prompt timeouts#96142

Merged
RomneyDa merged 1 commit into
openclaw:mainfrom
brokemac79:codex/recreate-95576-failover-timeout
Jun 25, 2026
Merged

fix(failover): fallback on replay-safe prompt timeouts#96142
RomneyDa merged 1 commit into
openclaw:mainfrom
brokemac79:codex/recreate-95576-failover-timeout

Conversation

@brokemac79

@brokemac79 brokemac79 commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #95574

Recreates and repairs #95576.

What Problem This Solves

Fixes an issue where replay-safe prompt-stage provider timeouts could surface as terminal errors even when model fallbacks were configured for the run.

Why This Change Was Made

The failover policy now allows model fallback for harness-owned timeout errors only when the timeout is from the prompt request, the attempt is replay-safe, the harness has not marked the timeout replay-invalid, and the failure is not a Codex app-server transport failure. Plugin harness lifecycle and abandoned-turn timeouts stay inside the existing surface-error path.

User Impact

Runs with configured fallback models can recover from safe provider prompt timeouts without replaying abandoned or side-effectful work.

Evidence

Focused local proof:

node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/failover-policy.test.ts src/agents/embedded-agent-runner/run.prompt-timeout-fallback.test.ts src/agents/embedded-agent-runner/run.codex-app-server-recovery.test.ts

Result: passed, 46 tests across 3 files.

git diff --check origin/main..HEAD

Result: passed.

Autoreview initially found a replay-invalid timeout gap; fixed by gating on attempt.promptTimeoutOutcome?.replayInvalid !== true and adding a regression test.

Final autoreview:

python .agents/skills/autoreview/scripts/autoreview --mode commit --commit HEAD --codex-bin %TEMP%/codex-fast-autoreview.cmd

Result: clean, no accepted/actionable findings.

Crabbox/Testbox changed-scope proof:

Exact command run on the Testbox after checking out brokemac79:codex/recreate-95576-failover-timeout:

env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 CI=1 corepack pnpm check:changed

Result: passed, TESTBOX_EXIT=0.

Relevant output:

[check:changed] lanes=core, coreTests
[check:changed] src/agents/embedded-agent-runner/run.prompt-timeout-fallback.test.ts: core test
[check:changed] src/agents/embedded-agent-runner/run.ts: core production
[check:changed] src/agents/embedded-agent-runner/run/failover-policy.test.ts: core test
[check:changed] src/agents/embedded-agent-runner/run/failover-policy.ts: core production
PASS direct dependency pin guard: checked 470 directly declared dependency specs across 158 tracked package manifests; 0 violations.
PASS package patch guard: no new pnpm patches; 4 legacy patches allowlisted.
Found 0 warnings and 0 errors.
Finished in 4.4s on 4 files with 242 rules using 1 threads.
Database-first legacy-store guard passed.
runtime-sidecar-loaders: local runtime sidecar loaders look OK.
Import cycle check: 0 runtime value cycle(s).
TESTBOX_EXIT=0

Azure Crabbox changed gate also passed on the pushed head 365bfa1e92eec80256e9f028cc9e7db2095c37ec:

Provider: azure
Lease: cbx_7b3f808c9cbd / slug harbor-barnacle
Run: run_ebe808a9b08d
Machine: Standard_D32ads_v6
Command: env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 CI=1 corepack pnpm check:changed
Result: exitCode=0
Timing: sync 5m7.189s, command 5m29.249s, total 12m51.318s
Cleanup: one-shot lease stopped by Crabbox (leaseStopped=true)

The Azure changed gate included guarded extension wildcard re-export checks, plugin-sdk wildcard re-export checks, dependency pin guard, npm shrinkwrap guard, runtime sidecar baseline/owner test, package patch guard, database-first legacy-store guard, media download helper guard, runtime sidecar loader guard, all changed-scope typecheck, lint shards, and runtime import cycle check.

Real behavior proof: prompt-stage timeout fallback

Behavior or issue addressed: after a replay-safe prompt-stage timeout from the primary model, runWithModelFallback retries the configured fallback model instead of ending the run with the timeout error.

Real environment tested: Crabbox provider=local-container on this Windows desktop, Docker image mcr.microsoft.com/playwright:v1.60.0-noble, lease cbx_667e40085f52, slug quick-prawn, PR head 365bfa1e92eec80256e9f028cc9e7db2095c37ec.

Exact steps or command run after this patch:

C:\Users\marti\.local\bin\crabbox.exe run `
  --provider local-container `
  --local-container-image mcr.microsoft.com/playwright:v1.60.0-noble `
  --no-hydrate `
  --timing-json `
  --script C:\oc-work\lsr911-recreate\95576\.tmp\proof-96142-local-container.sh

Evidence after fix: the proof harness uses the production runWithModelFallback loop and production runEmbeddedAgent path with a registered plugin harness. The primary configured model returns a prompt-stage timeout outcome; the fallback configured model completes.

PROOF_96142_RESULT={"scenario":"prompt-stage timeout handoff to configured model fallback","primary":{"provider":"openai","model":"timeout-primary"},"fallback":{"provider":"openai","model":"fallback-ok"},"promptErrorSource":"prompt","promptTimeoutOutcome":{"message":"Harness observed prompt request timeout before assistant/tool output.","replayInvalid":false,"livenessState":"blocked","providerStarted":true,"timeoutPhase":"prompt"},"replaySafe":true,"result":{"provider":"openai","model":"fallback-ok","outcome":"completed"},"attemptCount":2}
[agent/embedded] embedded run failover decision: runId=proof-96142-handoff stage=prompt decision=fallback_model reason=timeout from=openai/timeout-primary profile=- rawError=LLM request timed out before assistant output.
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai/timeout-primary candidate=openai/timeout-primary reason=timeout next=openai/fallback-ok detail=LLM request timed out before assistant output.
[model-fallback/decision] model fallback decision: decision=candidate_succeeded requested=openai/timeout-primary candidate=openai/fallback-ok reason=unknown next=none

Observed result after fix: the prompt timeout is classified as prompt-stage and replay-safe, the configured fallback openai/fallback-ok is attempted, and the run completes on the fallback model after two attempts.

Supplemental validation:

Crabbox changed gate: env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 CI=1 corepack pnpm check:changed
Result: exitCode=0
Timing: sync 2m15.308s, command 13m52.512s, total 16m7.919s
Phases: install 1m12.547s, behavior-proof 15.98s, changed-gate 12m23.327s
Cleanup: one-shot local-container lease stopped by Crabbox

Supplemental desktop Ollama proof:

  • Environment: Windows desktop local OpenClaw embedded agent, Ollama 0.30.8 on http://127.0.0.1:11434, model qwen3.5:4b, PR head 365bfa1e92eec80256e9f028cc9e7db2095c37ec.
  • Config shape: default primary ollama-timeout/qwen3.5:4b pointed at http://10.255.255.1:11434 with a 1 second timeout; configured fallback ollama/qwen3.5:4b pointed at the local Ollama server with localModelLean: true and a 262144-token declared context window.

Exact command run after this patch:

$env:OPENCLAW_STATE_DIR = "C:\Users\marti\AppData\Local\Temp\openclaw-96142-ollama-proof-1a1883d5daf84d9cb6ae44833d5b84df"
$env:OPENCLAW_CONFIG_PATH = "C:\oc-work\lsr911-recreate\95576\.tmp\proof-96142-ollama-openclaw.json"
$env:OLLAMA_API_KEY = "ollama-local"
$env:OPENCLAW_SKIP_CHANNELS = "1"
$env:OPENCLAW_DEBUG_MODEL_TRANSPORT = "1"
$env:OPENCLAW_DEBUG_SSE = "events"
$env:OPENCLAW_DEBUG_MODEL_PAYLOAD = "summary"
node scripts/run-node.mjs agent --local --agent main --session-key proof-96142-ollama-live --message "Reply with exactly: proof-ok" --timeout 150 --json

Evidence after fix:

[fetch-timeout] fetch timeout after 1000ms (elapsed 1006ms) operation=fetchWithSsrFGuard url=http://10.255.255.1:11434/api/chat
[agent/embedded] embedded run failover decision: runId=b67a0fdf-ccba-4296-9dd8-08dfe02765f1 stage=assistant decision=fallback_model reason=timeout from=ollama-timeout/qwen3.5:4b profile=- rawError=LLM idle timeout (1s): no response from model
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=ollama-timeout/qwen3.5:4b candidate=ollama-timeout/qwen3.5:4b reason=timeout next=ollama/qwen3.5:4b detail=LLM idle timeout (1s): no response from model
[model-fallback/decision] model fallback decision: decision=candidate_succeeded requested=ollama-timeout/qwen3.5:4b candidate=ollama/qwen3.5:4b reason=unknown next=none

Observed result after fix:

{
  "payloads": [{ "text": "proof-ok", "mediaUrl": null }],
  "meta": {
    "agentMeta": {
      "provider": "ollama",
      "model": "qwen3.5:4b",
      "contextTokens": 262144,
      "fallbackAttempts": [
        {
          "provider": "ollama-timeout",
          "model": "qwen3.5:4b",
          "error": "LLM idle timeout (1s): no response from model",
          "reason": "timeout",
          "status": 408
        }
      ]
    },
    "finalAssistantVisibleText": "proof-ok"
  }
}

Proof limitations or environment constraints: the local-container proof uses a deterministic registered OpenClaw plugin harness. The supplemental desktop proof exercises a real local Ollama server and a real fetch timeout to an unreachable Ollama endpoint from the same Windows host, but it does not prove a remote Crabbox VM can reach this desktop's loopback Ollama service.

What was not tested: live third-party cloud provider auth, an actual external provider outage, and remote Crabbox-to-desktop Ollama tunneling were not exercised.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jun 23, 2026
@brokemac79
brokemac79 marked this pull request as ready for review June 23, 2026 16:51
@clawsweeper

clawsweeper Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 25, 2026, 5:22 PM ET / 21:22 UTC.

Summary
Adds a replay-safety-gated prompt-timeout fallback path for embedded-agent harness-owned prompt timeouts, plus focused failover-policy and runner tests.

PR surface: Source +17, Tests +137. Total +154 across 4 files.

Reproducibility: yes. at source level: current main and v2026.6.10 return surface_error before fallback for the implicated harness-owned prompt-timeout branch, and the linked issue includes a concrete Ollama trajectory.

Review metrics: 1 noteworthy metric.

  • Fallback policy inputs: 1 input added, 2 prompt decision call sites updated. The new predicate changes the shared provider/model fallback decision point for harness-owned prompt timeouts.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P2] Merging intentionally changes existing behavior for replay-safe harness-owned prompt timeouts with configured fallbacks: runs that currently surface a timeout may now advance to another provider/model.
  • [P1] The same canonical bug has multiple open candidate PRs, so maintainers should choose one landing path and close or supersede the sibling branches after review.

Maintainer options:

  1. Land This Replay-Safe Candidate (recommended)
    After maintainer review, use this branch as the canonical fix because it preserves replay-invalid and Codex app-server timeout boundaries that simpler sibling PRs miss.
  2. Ask For More Live Provider Proof
    If deterministic production-loop proof and supplemental Ollama logs are not enough for this compatibility-sensitive path, ask for redacted live provider outage logs before merge.
  3. Pause For Sibling Selection
    If maintainers prefer another open candidate, pause this PR until the selected branch has equivalent replay and Codex isolation coverage.

Next step before merge

  • [P2] Maintainer review should choose this branch or a sibling as the canonical fix; no ClawSweeper code repair is indicated.

Security
Cleared: The diff only changes TypeScript failover control flow and tests; it does not touch secrets, dependencies, workflows, lockfiles, package metadata, or other supply-chain surfaces.

Review details

Best possible solution:

Land this replay-safe implementation or an equivalent maintainer-chosen patch as the canonical fix while preserving Codex app-server and replay-invalid timeout isolation.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main and v2026.6.10 return surface_error before fallback for the implicated harness-owned prompt-timeout branch, and the linked issue includes a concrete Ollama trajectory.

Is this the best way to solve the issue?

Yes: this is narrower than a simple fallbackConfigured guard because it also requires prompt source, timeout classification, replay safety, no replay-invalid timeout outcome, and no Codex app-server failure.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 13ecca5408cb.

Label changes

Label justifications:

  • P1: The PR targets a broken configured fallback path that can terminate real agent, chat, or cron workflows when the primary provider times out.
  • merge-risk: 🚨 compatibility: The diff changes established harness-owned timeout behavior from always surfacing an error to falling back for a replay-safe prompt-timeout subset.
  • merge-risk: 🚨 auth-provider: The changed branch decides whether a failed provider/model attempt stays inside its harness or advances to configured model fallback routing.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The PR body includes after-fix logs from a Crabbox local-container run through the production fallback loop and supplemental desktop Ollama timeout logs.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix logs from a Crabbox local-container run through the production fallback loop and supplemental desktop Ollama timeout logs.
Evidence reviewed

PR surface:

Source +17, Tests +137. Total +154 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 2 17 0 +17
Tests 2 137 0 +137
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 154 0 +154

Acceptance criteria:

  • [P2] Review the existing PR proof logs for PROOF_96142_RESULT and the supplemental Ollama fallback attempt.
  • [P2] If maintainers need a fresh local check, run the PR's focused node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/failover-policy.test.ts src/agents/embedded-agent-runner/run.prompt-timeout-fallback.test.ts src/agents/embedded-agent-runner/run.codex-app-server-recovery.test.ts command on the PR branch.
  • [P1] Before landing, ensure the chosen canonical PR has current CI or changed-gate proof on its final head SHA.

What I checked:

Likely related people:

  • pashpashpash: Authored the merged timeout-isolation change that added harness-owned timeout policy and tests later narrowed by this PR. (role: introduced behavior; confidence: high; commits: 3a64dc762385; files: src/agents/pi-embedded-runner/run/failover-policy.ts, src/agents/pi-embedded-runner/run/failover-policy.test.ts, src/agents/pi-embedded-runner/run.codex-app-server-recovery.test.ts)
  • steipete: Merged the timeout-isolation work and authored the runtime internalization commit that carried the failover policy into the current embedded-agent-runner path. (role: merger and refactor carrier; confidence: high; commits: 3a64dc762385, bb46b79d3c14; files: src/agents/embedded-agent-runner/run/failover-policy.ts, src/agents/embedded-agent-runner/run/failover-policy.test.ts)
  • vincentkoc: Recent live GitHub history shows multiple agent failover and embedded-runner commits on the affected helper, runner, and model-fallback paths. (role: recent area contributor; confidence: medium; commits: 0804901c11d9, 31a65e0647ae, 80805ad7a583; files: src/agents/embedded-agent-runner/run/failover-policy.ts, src/agents/embedded-agent-runner/run.ts, src/agents/model-fallback.ts)
  • rosenlo: Local shallow blame for the current prompt-timeout branch points to a broad current-main snapshot commit, useful for routing current-file questions but not original policy authorship. (role: current-file blame signal; confidence: low; commits: f9e0dce73109; files: src/agents/embedded-agent-runner/run/failover-policy.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jun 24, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 25, 2026
Closed
@brokemac79

brokemac79 commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 25, 2026
@RomneyDa
RomneyDa merged commit 0da2649 into openclaw:main Jun 25, 2026
251 of 267 checks passed
@ruomuxydt ruomuxydt mentioned this pull request Jun 26, 2026
Closed
@Haderach-Ram Haderach-Ram mentioned this pull request Jun 26, 2026
Open
@github-actions github-actions Bot mentioned this pull request Jun 26, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 26, 2026
@brokemac79
fix: fallback on safe prompt timeouts (openclaw#96142)
99a0d79
This was referenced Jun 26, 2026
Closed
Open
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
@brokemac79 @QiuYuang
fix: fallback on safe prompt timeouts (openclaw#96142)
c4415e3
@clawsweeper clawsweeper Bot mentioned this pull request Jul 2, 2026
Closed
amittell pushed a commit to amittell/openclaw that referenced this pull request Jul 2, 2026
@brokemac79 @amittell
fix: fallback on safe prompt timeouts (openclaw#96142)
e60ac8b 
(cherry picked from commit 0da2649)
@clawsweeper clawsweeper Bot mentioned this pull request Jul 3, 2026
Open
wheakerd pushed a commit to wheakerd/clawdbot that referenced this pull request Jul 15, 2026
@brokemac79 @RomneyDa
fix: fallback on safe prompt timeouts (openclaw#96142)
3aa5229 
(cherry picked from commit 0da2649)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

agents Agent runtime and tooling merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Behavior bug (incorrect output/state without crash)

2 participants

@brokemac79 @RomneyDa