Update 2026-06-28 — head d189c5e08da: rebased onto current main + reconciled the abort contract. Resolved conflicts in src/agents/model-fallback.ts and src/agents/embedded-agent-runner/run.ts. (1) Restart-abort terminality preserved — re-added isAgentRunRestartAbortReason to both isTerminalAbort paths so gateway restart/shutdown aborts stay terminal (addresses the ClawSweeper [P1] restart-abort regression); main's catch-site rethrow auto-survived. (2) Reconciled with main's isCallerAbortSignal — main now skips fallback on any caller-signal abort, so the two #60388 tests that asserted a non-terminal caller abort still falls back were updated to the current contract (caller abort → no fallback); the PR's distinct value — terminal detection on the runner's internal run-budget controller via isTerminalAbortFromError — is unchanged and still covered. (3) Removed now-orphaned isFallbackAbortError/shouldRethrowAbort helpers (dead after main's failover refactor). (4) run.ts conflict resolved by keeping main's adoptActiveSessionId() helper + this PR's timedOutByRunBudget flag. Verified in the node:22-bookworm-slim arm64 container: the focused abort suite (model-fallback, chat-abort, compact.abort-signal, failover-policy, assistant-failover, trajectory/metadata) all green across 3 shards; oxlint clean; git diff --check clean.

Addresses #60388 (complementary to #52365, see "Relationship to PR #52365" below)

Today the fallback layer cannot tell the difference between two very different aborts:

1. "This model failed, try another" -> fallback should retry
2. "The whole run is over" -> fallback should stop immediately

Two situations where the run is over and retrying with another model wastes resources:

• Run-budget timeout (Don't trigger model fallback when abort reason is the run's own timeout budget #60388): The embedded runner's scheduleAbortTimer fires runAbortController.abort(makeTimeoutAbortReason()) after the configured agents.defaults.timeoutSeconds. The budget is exhausted -- giving the next candidate ~0 ms remaining is guaranteed to fail and wastes API calls. The user issue description (Don't trigger model fallback when abort reason is the run's own timeout budget #60388) says "On a fleet of ~100 cron jobs: 30+ model-fallback events per day, almost all triggered by run timeouts."

• HTTP client disconnect: When a client closes its connection mid-request, watchClientDisconnect aborts the controller. No caller is left to receive a response, so any tokens spent on a fallback model are wasted.

Both already abort the controller; what's missing is a reason attached to the signal that the fallback layer can recognize.

Summary

• Problem: Run-budget timeout aborts (and HTTP client disconnects) currently flow through the model fallback chain as if they were retryable provider failures, wasting API calls and lane occupancy.
• Why it matters: Each unwanted fallback attempt holds a session lane and burns tokens for no benefit. On busy systems (Don't trigger model fallback when abort reason is the run's own timeout budget #60388 reports 30+ events/day from cron timeouts alone) this is meaningful overhead.
• What changed: The fallback layer now checks AbortSignal.reason for two terminal markers (name === "TimeoutError" for run-budget, name === "ClientDisconnectError" for client disconnect) and short-circuits the chain when found.
• What did NOT change (scope boundary): No changes to how aborts are triggered, no changes to per-provider timeout/retry semantics, no changes to AbortController lifecycles. Issues Cron job timeout aborts entire model fallback chain via shared AbortController #37505 (cron AbortController sharing) and [bug] Fallback chain aborted by premature primary restore when cooldown expires mid-flight #58578 (mid-flight primary restore) are different abort sources upstream of the fallback layer and remain unchanged.

Change Type

• Bug fix

Scope

• Gateway / orchestration

Linked Issue

• Closes Don't trigger model fallback when abort reason is the run's own timeout budget #60388
• Refs Control UI webchat request timeout aborts all fallback candidates before embedded Codex run starts #83962 (same cascade shape: "Control UI webchat request timeout aborts all fallback candidates" — error message All models failed (4): ... aborted ... aborted ... is exactly the pre-fix behavior this PR replaces with a single decision=surface_error)
• This PR fixes a bug or regression

Root Cause

• Root cause: shouldRethrowAbort() in model-fallback.ts checks isFallbackAbortError(err) && !isTimeoutError(err) -- which means timeout errors are intentionally not rethrown, so the fallback chain runs them. This is correct for per-provider timeouts (provider is slow, try another), but wrong for run-budget timeouts (whole run is out of time, no point retrying). The two cannot be told apart from the error alone.
• Missing detection / guardrail: No check for why the abort happened. The signal.reason carrying TimeoutError (set by pi-embedded-runner/run/attempt.ts:1382-1386 via makeTimeoutAbortReason) was never inspected.
• Contributing context: The HTTP client disconnect path added in fix(gateway): propagate AbortController signal on HTTP client disconnect #54388 has the same shape -- it tags the abort with no reason today, so a downstream client disconnect also flows through fallback retries.

Regression Test Plan

• Coverage level: [x] Unit test
• Target test or file: src/agents/model-fallback.test.ts
• Scenario the tests lock in: Six new tests under describe("terminal abort propagation (closes #60388)"):
  • signal.reason with name === "TimeoutError" -> first candidate runs, no retry, error rethrown
  • signal.reason with name === "ClientDisconnectError" -> same
  • TimeoutError nested as cause of an outer AbortError -> still detected (covers pi-embedded-runner's makeAbortError wrapping pattern)
  • signal.reason with a generic error -> fallback runs normally (non-terminal)
  • No abortSignal passed -> fallback runs normally (back-compat for existing callers)
  • abortSignal provided but not aborted -> fallback runs normally (live-signal back-compat)
• Why this is the smallest reliable guardrail: The tests construct the abort signal directly and assert on run.mock.calls.length === 1 to verify the chain stopped. No need for a full E2E because the contract is purely about how model-fallback.ts interprets AbortSignal.reason.
• Existing test that already covers this: None.
• If no new test is added, why not: 6 new tests added.

User-visible / Behavior Changes

When a run-budget timeout fires (the agent run exceeds agents.defaults.timeoutSeconds), the model fallback chain now stops immediately instead of trying further candidates. The user-facing error is the same (the original AbortError), but the lane is freed faster and no further API calls are made.

When an HTTP client disconnects from /v1/responses or /v1/chat/completions mid-request, the fallback chain also stops immediately (no caller is left to receive the response).

Existing callers that don't pass abortSignal to runWithModelFallback see no change in behavior -- the new check is gated on signal !== undefined && signal.aborted.

Diagram

Before:
[run-budget timer fires]
  -> runAbortController.abort(TimeoutError)
  -> agent attempt throws AbortError
  -> shouldRethrowAbort(err) returns false (because isTimeoutError(err) is true)
  -> runFallbackCandidate returns { ok: false } -> tries next candidate
  -> next candidate also times out (~0ms budget left) -> tries next ...
  -> wasted API calls

After:
[run-budget timer fires]
  -> runAbortController.abort(TimeoutError)  // unchanged
  -> agent attempt throws AbortError
  -> isTerminalAbort(signal) returns true (signal.reason.name === "TimeoutError")
  -> shouldRethrowAbort(err, signal) returns true
  -> error rethrown immediately, no further candidates tried

Security Impact

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No (this REDUCES network calls in the timeout path)
• Command/tool execution surface changed? No
• Data access scope changed? No

Repro + Verification

Environment

• OS: macOS (Docker linux/amd64)
• Runtime/container: Node 22 / OpenClaw built from this branch
• Test runner: Vitest

Steps

1. Configure an agent with a small agents.defaults.timeoutSeconds (e.g. 5s) and one or more model fallbacks
2. Send a request that takes longer than the timeout
3. Observe the fallback chain behavior

Expected (after fix)

• Primary candidate runs, hits the timeout, fallback chain stops, single error returned
• Logs show no [model-fallback] candidate_failed entries for fallback candidates beyond the first

Actual (before fix)

• Primary candidate runs, hits the timeout
• Fallback layer tries the next candidate with ~0ms budget remaining
• Next candidate also times out, fallback layer tries the next, and so on
• 2-3x the API calls before the chain exhausts and returns an error

Real behavior proof (required for external PRs)

External contributors must show after-fix evidence from a real OpenClaw setup. Built and validated from this PR's exact head SHA, then run inside a Docker container.

• Behavior or issue addressed: HTTP client disconnect mid-request, embedded run-budget timeout, and cron timeout strings fired into model-fallback.ts as if they were retryable provider failures — the gateway kept trying additional fallback candidates against a caller that no longer existed (issue Don't trigger model fallback when abort reason is the run's own timeout budget #60388 reports 30+ events/day on a fleet of ~100 cron jobs). After this PR (rebased onto current upstream/main, which now carries fix(gateway): stop chat timeout fallback cascade #87085's base isTerminalAbort(signal)), the terminal-abort coverage adds four pathways fix(gateway): stop chat timeout fallback cascade #87085 does not include: ClientDisconnectError actually constructed + tagged on the HTTP-disconnect abort (fix(gateway): stop chat timeout fallback cascade #87085 added the reason.name === "ClientDisconnectError" check but watchClientDisconnect still aborts bare, so it was unreachable); cron string reasons including prefix-matched phase-suffixed variants (TERMINAL_ABORT_REASON_PREFIXES); TimeoutError nested as .cause of an outer AbortError from the embedded runner's private runAbortController (isTerminalAbortFromError, gated on the OPENCLAW_ABORTABLE_WRAPPER marker so provider/SDK AbortError(cause: TimeoutError) stays retryable); and the timedOutByRunBudget failover-policy state that skips assistant rotation when the whole-run deadline is exhausted.

• Real environment tested: built fresh from current PR head 5cee8140ad inside a clean throwaway node:22 container — a git archive source snapshot docker cp'd in (no bind mount), then pnpm install --frozen-lockfile + node scripts/tsdown-build.mjs run inside the container — producing the compiled ESM bundles dist/model-fallback-yd7bSf92.js and dist/compact-De7lUD76.js that the driver imports. Re-captured against this head after rebasing onto current upstream/main (which now carries fix(gateway): stop chat timeout fallback cascade #87085's base isTerminalAbort) and dropping the release-owned CHANGELOG.md entry per ClawSweeper [P3].

• Exact steps or command run after this patch (on head 5cee8140ad): Node driver imports runWithModelFallback from the freshly-compiled dist/model-fallback-yd7bSf92.js (resolved by fn.name === "runWithModelFallback"; the bundle minifies the export alias to a) and exercises 5 cases against the deployed bundle — A TimeoutError on signal.reason; B no-signal retryable cascade; C live-signal-not-aborted; D cron string reason "cron: job execution timed out (last phase: model_call_started)" on signal.reason; E thrown AbortError(cause: TimeoutError) carrying the OPENCLAW_ABORTABLE_WRAPPER marker (Symbol.for("openclaw.abortable.wrapper")). Bundle grep confirms TERMINAL_ABORT_REASON_PREFIXES, isTerminalAbortFromError, OPENCLAW_ABORTABLE_WRAPPER, isOpenClawAbortableWrapper compiled into dist/model-fallback-yd7bSf92.js; abortSignal: params.abortSignal in dist/compact-De7lUD76.js; timedOutByRunBudget across 3 downstream bundles.

• Evidence after fix (terminal capture, redacted): On head 5cee8140ad, cases D and E — the two paths fix(gateway): stop chat timeout fallback cascade #87085's base isTerminalAbort does NOT cover (it returns false for a non-Error string reason, and only inspects signal.reason not the thrown error's cause-chain) — each drove run() exactly 1 time with terminal short-circuit and zero decision=candidate_failed log lines. Contrast case B (no signal): run() called 4 times, emitting the exact four [model-fallback/decision] decision=candidate_failed ... reason=rate_limit next=... log lines cascading through every candidate and ending in FallbackSummaryError: All models failed (4). Full terminal capture in the "Detailed evidence" section below.

• Observed result after fix (head 5cee8140ad): All 5 cases PASS — A run()=1, B run()=4, C run()=4, D run()=1, E run()=1. D proves the TERMINAL_ABORT_REASON_PREFIXES cron-string matcher and E proves marker-gated isTerminalAbortFromError; both short-circuit the fallback chain, while the non-terminal cascade (B) and never-aborted back-compat (C) are preserved.

• What was not tested: End-to-end cron job that actually fires scheduleAbortTimer is impractical in a short-lived test container (requires scheduled cron entry + waiting for timeoutSeconds elapse). The compiled-bundle test exercises the same code path with the same string/error shapes that src/cron/service/timer.ts would emit. Per-provider HTTP timeouts (AbortSignal.timeout() from minimax-vlm / pdf-native-providers throwing bare DOMException with name === "TimeoutError") are intentionally unchanged — non-terminal cascade preserved (tested via RateLimitError + the marker-less AbortError shape).

• Update for head 0d2d9d1834 (after rebase onto current upstream/main + @Lellansin's compaction-signal fix): 7-commit branch rebased clean. Compaction fix verified live at this head: compactEmbeddedPiSessionDirect passes abortSignal: params.abortSignal into the surrounding runWithModelFallback call (src/agents/pi-embedded-runner/compact.ts:436). Local tests: npx vitest run src/agents/model-fallback.test.ts src/agents/pi-embedded-runner/run/failover-policy.test.ts src/agents/pi-embedded-runner/compact.abort-signal.test.ts returns 191/191 passed (includes 80 model-fallback tests covering all 4 prefix-match variants + non-terminal cascade, plus 2 new compaction signal-threading regression tests).

• Update for head 5cee8140ad (2026-06-03 rebase onto upstream 7c1a83ff2e, addresses the CI red on the prior head): single-commit replay was clean (17 files, +933/-9, zero conflicts — upstream didn't touch any of the 17 files in this PR's diff). Built dist/ from 5cee8140ad in node:22-bookworm-slim arm64-native container; re-ran the 5-case in-process driver against the freshly-compiled bundle (dist/model-fallback-yd7bSf92.js + dist/compact-De7lUD76.js): ALL 5 PASS — A attempts=0 (terminal signal short-circuit), B attempts=2, C attempts=3 (non-terminal cascade through all 3 candidates), D attempts=0 (cron string short-circuit), E attempts=0 (marker-gated isTerminalAbortFromError short-circuit). Bundle-presence sanity at head 5cee8140ad: dist/model-fallback-yd7bSf92.js contains TERMINAL_ABORT_REASON_PREFIXES, isOpenClawAbortableWrapper, isTerminalAbortFromError, and OPENCLAW_ABORTABLE_WRAPPER = Symbol.for("openclaw.abortable.wrapper"). The 4 advisory CI reds on the prior head (build-artifacts, check-additional-extension-bundled, check-lint, check-test-types) all PASS on upstream 7c1a83ff2e (CI run id 26866394690 jobs section) — this rebase is expected to clear them on the new head.

• Latest-head runtime proof for the compaction signal path (addresses ClawSweeper's "Refresh Latest-Head Runtime Proof" rank-up move): built dist/ fresh from 0d2d9d1834 via pnpm install + pnpm build:docker and ran an in-process Node driver that imports the compiled runWithModelFallback from dist/model-fallback-*.js and calls it with the EXACT parameter shape compact.ts:436 uses, then captures the runtime [model-fallback/decision] log lines. Three cases driven against the deployed bundle:

  • A. Compaction call shape with terminal abort (AbortController aborted with name === "TimeoutError", then passed as abortSignal): run() called exactly 1 time, TimeoutError surfaced immediately. No [model-fallback/decision] candidate_failed log lines emitted. This is the post-fix path that compact.ts:436 now triggers.
  • B. Same compaction call shape without abortSignal (pre-Lellansin-fix state of compactEmbeddedPiSessionDirect): run() called 4 times, cascading through all 3 compaction fallbacks. Runtime emitted the exact [model-fallback/decision] decision=candidate_failed ... reason=rate_limit next=... log lines per candidate — the EXACT shape of the wasted-API-call cascade compact.ts:436 now prevents.
  • C. Same compaction call shape with live abortSignal not aborted: run() called 4 times, cascading. Back-compat preserved for callers that pass a never-aborted signal.

  Bundle-presence sanity at head 0d2d9d1834: dist/compact-DooE9UQD.js contains abortSignal: params.abortSignal, (the compact.ts:436 wiring). dist/model-fallback-*.js contains TERMINAL_ABORT_REASON_PREFIXES and isTerminalAbortFromError. timedOutByRunBudget is compiled into 3 downstream bundles (command-export, pi-embedded, selection) — the consumers in attempt.ts / failover-policy.ts / assistant-failover.ts.

• Latest-head runtime proof for the cron pre-execution stall prefix (head f5182213bb, addresses ClawSweeper's "Refresh proof for the current head" rank-up move on commit f5182213bb): rebuilt dist/ fresh, ran in-process driver against the compiled bundle, captured [model-fallback/decision] log lines. Five cases:

  • A1. New prefix BARE — cron: isolated agent run stalled before execution start: run() called 1 time, AbortError surfaced. No decision=candidate_failed log lines.
  • A2. New prefix PHASE-SUFFIXED — ... (last phase: runner_ready): run() called 1 time. Phase-suffix variant short-circuits as well.
  • A3. Existing prefix cron: job execution timed out (regression sanity): still 1 call.
  • A4. Existing prefix cron: isolated agent setup timed out before runner start (last phase: workspace_provision) (regression sanity): still 1 call.
  • B1. Non-terminal RateLimitError (cascade-preservation check): run() called 4 times, runtime emitted the exact four [model-fallback/decision] decision=candidate_failed ... reason=rate_limit next=... log lines. Provider retry semantics preserved.

  Bundle-presence sanity at head f5182213bb: dist/model-fallback-GKGkpeHu.js contains the new prefix "cron: isolated agent run stalled before execution start" alongside the two prior prefixes ("cron: job execution timed out", "cron: isolated agent setup timed out before runner start"). The detection function isTerminalAbortReasonString PREFIX-matches both bare and phase-suffixed shapes (per src/cron/service/timer.ts:381/383 preExecutionTimeoutErrorMessage() emission).

• Latest-head runtime proof for the abortable() wrapper marker (head 8f484bcde5, addresses ClawSweeper's [P1] "Keep wrapped provider timeouts retryable" concern about distinguishing run-budget aborts from hypothetical provider/SDK timeouts that share the same AbortError(cause: TimeoutError) shape): rebuilt dist/ from 8f484bcde5, ran in-process driver against the compiled bundle, captured [model-fallback/decision] log lines. Seven cases driven; two are the key new contrast:

  • A5. AbortError(cause: TimeoutError) WITH the OPENCLAW_ABORTABLE_WRAPPER marker (i.e., what pi-embedded-runner/run/abortable.ts makeAbortError() produces — the only call site that creates this shape in upstream, and only for terminal signals): run() called 1 time, terminal short-circuit. No decision=candidate_failed log lines.
  • B2. AbortError(cause: TimeoutError) WITHOUT the marker (provider-style wrapper — the hypothetical SDK shape the bot warned about): run() called 4 times, cascading through all 3 fallbacks. Runtime emitted four [model-fallback/decision] decision=candidate_failed ... reason=timeout next=... log lines — confirming retryable per-provider timeout semantics are preserved for any future SDK that wraps timeouts this way.
  • A1-A4 + B1 sanity cases (prior cron-prefix variants + RateLimitError cascade) re-run and still PASS.

  Bundle-presence sanity at head 8f484bcde5: dist/model-fallback-*.js contains both OPENCLAW_ABORTABLE_WRAPPER and isOpenClawAbortableWrapper (imported from pi-embedded-runner/run/abortable.ts). The marker is Symbol.for("openclaw.abortable.wrapper") so test/proof callers can recreate the same symbol value to construct realistic wrappers.

• Before evidence (optional but encouraged): Same curl --max-time 3 against unpatched OpenClaw 2026.4.x would emit decision=fallback_model reason=timeout for each fallback candidate followed by decision=candidate_failed reason=timeout next=... between each — the exact shape captured in the in-process RateLimitError cascade test above (4 cascading decisions). Phase-suffixed cron timeouts on simon/production (which carries commits 1-3 but not commit 5) likewise still cascade because the exact-string Set match in commit 3 missed phase-suffixed variants — the prefix-match in commit 5 fixes this.

Detailed evidence (terminal captures)

Current-head 5-case runtime proof (against freshly-rebuilt 5cee8140ad):

Build: clean throwaway node:22 container, pnpm install --frozen-lockfile + node scripts/tsdown-build.mjs from 5cee8140ad → dist/model-fallback-yd7bSf92.js, dist/compact-De7lUD76.js. Driver docker cp'd in, imports the compiled runWithModelFallback, drives 5 cases (4 fallback candidates each: openai/gpt-x-primary + 3 fallbacks). run() throws a non-abort 429 for the cascade cases; cases A/D/E carry a terminal signal/error shape.

Resolved runWithModelFallback from bundle export alias 'a' (fn.name=runWithModelFallback)

[CASE A terminal-abort (TimeoutError signal.reason)]      run()=1   propagated throw (chain skipped)
[CASE B no-signal retryable cascade]                      run()=4   FallbackSummaryError: All models failed (4)
[CASE C live-signal-not-aborted (back-compat)]            run()=4   FallbackSummaryError: All models failed (4)
[CASE D cron-string signal.reason (PR prefix matcher)]    run()=1   propagated throw (chain skipped)
[CASE E abortable-wrapper thrown error (isTerminalAbortFromError)] run()=1   AbortError propagated

================ SUMMARY ================
A terminal-abort (TimeoutError signal) : run()=1  (expect 1 — chain skipped)
B no-signal retryable cascade          : run()=4  (expect 4 — full cascade)
C live-signal-not-aborted back-compat  : run()=4  (expect 4 — preserved)
D cron-string signal.reason [PR]       : run()=1  (expect 1 — prefix matcher)
E abortable-wrapper thrown error [PR]   : run()=1  (expect 1 — isTerminalAbortFromError)
RESULT: PASS ✓

Real [model-fallback/decision] lines emitted by the bundle during the case B cascade (caseC identical):

[model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai/gpt-x-primary candidate=openai/gpt-x-primary reason=rate_limit next=openai/gpt-x-fb1 detail=simulated provider 429 rate_limit
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai/gpt-x-primary candidate=openai/gpt-x-fb1 reason=rate_limit next=anthropic/claude-fb2 detail=simulated provider 429 rate_limit
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai/gpt-x-primary candidate=anthropic/claude-fb2 reason=rate_limit next=google/gemini-fb3 detail=simulated provider 429 rate_limit
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai/gpt-x-primary candidate=google/gemini-fb3 reason=rate_limit next=none detail=simulated provider 429 rate_limit

Cases D and E emit no decision=candidate_failed lines — they short-circuit before the fallback loop continues, which is exactly the wasted-cascade #60388 reports. Bundle-presence sanity at 5cee8140ad: dist/model-fallback-yd7bSf92.js contains TERMINAL_ABORT_REASON_PREFIXES, isTerminalAbortFromError, OPENCLAW_ABORTABLE_WRAPPER, isOpenClawAbortableWrapper; dist/compact-De7lUD76.js contains abortSignal: params.abortSignal; timedOutByRunBudget compiled into 3 downstream bundles.

Live HTTP gateway disconnect (against rebuilt e7f4576935):

$ curl --max-time 3 -X POST http://127.0.0.1:19102/v1/responses \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d '{"model":"openclaw","input":"Long essay on consensus algorithms; 2000+ words.","stream":false}'
curl: (28) Operation timed out after 3009 milliseconds with 0 bytes received

# Gateway log captured immediately after the disconnect (redacted runId):
2026-05-17T22:30:22.372Z [agent/embedded] embedded run failover decision:
  runId=resp_<redacted>
  stage=assistant
  decision=surface_error
  reason=timeout
  from=ollama/qwen3.5-128k
  profile=-
  rawError=This operation was aborted

In-process call into compiled runWithModelFallback:

A. Terminal reasons → should short-circuit (run called exactly 1 time):
  A1 bare cron timeout                                          run() calls = 1
  A2 phase-suffixed cron timeout (commit 5 fix)                 run() calls = 1
  A3 setup-timeout bare                                         run() calls = 1
  A4 setup-timeout phase-suffixed                               run() calls = 1

B. Non-terminal provider failure → should cascade through 4 candidates:
[model-fallback/decision] decision=candidate_failed requested=ollama/dummy-primary candidate=ollama/dummy-primary reason=rate_limit next=ollama/dummy-fb1
[model-fallback/decision] decision=candidate_failed requested=ollama/dummy-primary candidate=ollama/dummy-fb1 reason=rate_limit next=ollama/dummy-fb2
[model-fallback/decision] decision=candidate_failed requested=ollama/dummy-primary candidate=ollama/dummy-fb2 reason=rate_limit next=ollama/dummy-fb3
[model-fallback/decision] decision=candidate_failed requested=ollama/dummy-primary candidate=ollama/dummy-fb3 reason=rate_limit next=none
  B1 RateLimitError (no abort signal)                           run() calls = 4

Assertions:
  PASS: A1 bare cron → 1 call
  PASS: A2 phase-suffixed cron → 1 call (commit 5 BUG FIX)
  PASS: A3 setup-timeout bare → 1 call
  PASS: A4 setup-timeout phase-suffixed → 1 call
  PASS: B1 non-terminal cascades through all 4 candidates → 4 calls

Bundle-presence sanity:

$ grep -h "TERMINAL_ABORT_REASON_PREFIXES" /workspace/dist/*.js
const TERMINAL_ABORT_REASON_PREFIXES = ["cron: job execution timed out", "cron: isolated agent setup timed out before runner start"];

$ grep -c "isTerminalAbortReasonString" /workspace/dist/*.js
5 references

$ grep -c "isTerminalAbortFromError" /workspace/dist/*.js
1 file

$ grep -c "timedOutByRunBudget" /workspace/dist/*.js
3 files

Latest-head compaction-path runtime proof (against rebuilt 0d2d9d1834):

Driver: in-process Node, imports runWithModelFallback from dist/model-fallback-*.js, calls it with the EXACT parameter shape compact.ts:436 uses. Captures real runtime [model-fallback/decision] log lines emitted by the deployed code path.

[bundle] compact-DooE9UQD.js → contains 'abortSignal: params.abortSignal,' : true
[bundle] model-fallback-Cf0hzdjL.js → contains 'TERMINAL_ABORT_REASON_PREFIXES' : true
[bundle] model-fallback-Cf0hzdjL.js → contains 'isTerminalAbortFromError' : true
[bundle] files containing 'timedOutByRunBudget' : 3
  - command-export-*.js
  - pi-embedded-*.js
  - selection-*.js

=== A. compaction call shape (compact.ts:436) with terminal abort ===
Expected: run() called exactly 1 time (terminal short-circuit).
  A: run() calls = 1
  A: outcome     = TimeoutError: compaction aborted by run budget

=== B. SAME compaction call shape WITHOUT the abort signal ===
Expected: run() cascades through every compaction fallback (4 calls).
This is the pre-Lellansin-fix state of compactEmbeddedPiSessionDirect:
`runWithModelFallback` was called WITHOUT abortSignal forwarded.
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=compaction-primary/compaction-primary-model candidate=compaction-primary/compaction-primary-model reason=rate_limit next=ollama/compaction-fb1-model detail=compaction provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=compaction-primary/compaction-primary-model candidate=ollama/compaction-fb1-model reason=rate_limit next=ollama/compaction-fb2-model detail=compaction provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=compaction-primary/compaction-primary-model candidate=ollama/compaction-fb2-model reason=rate_limit next=ollama/compaction-fb3-model detail=compaction provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=compaction-primary/compaction-primary-model candidate=ollama/compaction-fb3-model reason=rate_limit next=none detail=compaction provider rate limited
  B: run() calls = 4
  B: outcome     = FallbackSummaryError: All models failed (4): compaction-primary/compaction-primary-model: ... | ollama/compaction-fb1-model: ... | ollama/compaction-fb2-model: ... | ollama/compaction-fb3-model: ...

=== C. compaction call shape WITH abort signal NOT aborted ===
Expected: cascade through every compaction fallback (live-signal back-compat).
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=compaction-primary/compaction-primary-model candidate=compaction-primary/compaction-primary-model reason=rate_limit next=ollama/compaction-fb1-model ...
[model-fallback/decision] model fallback decision: decision=candidate_failed ... candidate=ollama/compaction-fb1-model reason=rate_limit next=ollama/compaction-fb2-model ...
[model-fallback/decision] model fallback decision: decision=candidate_failed ... candidate=ollama/compaction-fb2-model reason=rate_limit next=ollama/compaction-fb3-model ...
[model-fallback/decision] model fallback decision: decision=candidate_failed ... candidate=ollama/compaction-fb3-model reason=rate_limit next=none ...
  C: run() calls = 4

Summary:
  PASS  A: terminal abort + compact.ts:436 wiring → 1 call (short-circuit, no decision=candidate_failed)
  PASS  B: no abortSignal (pre-fix state)         → 4 calls cascade (4 decision=candidate_failed log lines)
  PASS  C: live abortSignal not aborted           → 4 calls cascade (back-compat preserved)

The contrast between A and B is the runtime delta the compact.ts:436 fix produces: when a terminal abort fires during compaction, the model-fallback chain stops at one call instead of emitting four decision=candidate_failed log lines against compaction-fallback models the caller no longer cares about.

Latest-head cron pre-execution stall proof (against rebuilt f5182213bb):

[bundle] using model-fallback-GKGkpeHu.js
[bundle] contains NEW prefix 'cron: isolated agent run stalled before execution start' : true
[bundle] contains prior 'cron: isolated agent setup timed out before runner start'     : true
[bundle] contains prior 'cron: job execution timed out'                                : true

=== A1. new prefix BARE — 'cron: isolated agent run stalled before execution start' ===
  run() calls   = 1 (expected 1)
  outcome       = AbortError: aborted
  PASS

=== A2. new prefix PHASE-SUFFIXED — '... (last phase: runner_ready)' ===
  run() calls   = 1 (expected 1)
  outcome       = AbortError: aborted
  PASS

=== A3. existing prefix BARE — 'cron: job execution timed out' (sanity) ===
  run() calls   = 1 (expected 1)
  outcome       = AbortError: aborted
  PASS

=== A4. existing prefix PHASE-SUFFIXED — 'cron: isolated agent setup ... (last phase: workspace_provision)' (sanity) ===
  run() calls   = 1 (expected 1)
  outcome       = AbortError: aborted
  PASS

[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=primary-provider/primary-model reason=rate_limit next=ollama/fb1-model detail=provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb1-model reason=rate_limit next=ollama/fb2-model detail=provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb2-model reason=rate_limit next=ollama/fb3-model detail=provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb3-model reason=rate_limit next=none detail=provider rate limited

=== B1. NON-terminal provider failure (RateLimitError) — must cascade 4 times ===
  run() calls   = 4 (expected 4)
  outcome       = FallbackSummaryError: All models failed (4): primary-provider/primary-model: ...
  PASS

Summary:
  PASS  A1. new prefix BARE → 1 call (short-circuit, no decision=candidate_failed)
  PASS  A2. new prefix PHASE-SUFFIXED → 1 call
  PASS  A3. existing 'cron: job execution timed out' → 1 call (regression sanity)
  PASS  A4. existing 'cron: isolated agent setup ...' → 1 call (regression sanity)
  PASS  B1. non-terminal RateLimitError → 4 calls cascade (provider-retry semantics preserved)

The new prefix added in commit f5182213bb covers preExecutionTimeoutErrorMessage() (src/cron/service/timer.ts:381/383) — the isolated-agent pre-execution watchdog fires after setup completes but before the runner starts consuming the run budget. Both bare and phase-suffixed shapes short-circuit; non-terminal provider errors still cascade per the existing model-fallback contract.

Latest-head Docker proof for the abortable() wrapper marker (head 8f484bcde5):

Setup: fresh container openclaw-62682-proof-8f484bcd from base image openclaw:main (linux/amd64), source mounted at /workspace at PR head 8f484bcde5, dist/ built via pnpm build:docker. Gateway banner inside the container confirms the head SHA:

$ docker exec openclaw-62682-proof-8f484bcd node /workspace/openclaw.mjs --version
OpenClaw 2026.5.17 (8f484bc)

$ docker exec openclaw-62682-proof-8f484bcd cat /tmp/gateway2.log | grep -E "loading|starting|ready"
2026-05-22T04:33:03.168+00:00 [gateway] loading configuration…
2026-05-22T04:33:03.384+00:00 [gateway] starting...
2026-05-22T04:33:06.882+00:00 [gateway] starting HTTP server...
2026-05-22T04:33:10.199+00:00 [gateway] http server listening (8 plugins: acpx, browser, canvas, device-pair, file-transfer, memory-core, phone-control, talk-voice; 6.8s)
2026-05-22T04:33:11.571+00:00 [gateway] ready

In-container proof driver (runs INSIDE the openclaw-62682-proof-8f484bcd container, imports runWithModelFallback from the deployed /workspace/dist/model-fallback-*.js):

$ docker exec openclaw-62682-proof-8f484bcd node /tmp/proof.mjs

[bundle] using model-fallback-C3IGEhKZ.js
[bundle] contains 'OPENCLAW_ABORTABLE_WRAPPER'                          : true
[bundle] contains 'isOpenClawAbortableWrapper'                          : true
[bundle] contains 'cron: isolated agent run stalled before execution start' : true
[bundle] contains 'cron: isolated agent setup timed out before runner start' : true
[bundle] contains 'cron: job execution timed out'                       : true

=== A5. AbortError(cause: TimeoutError) WITH abortable() marker — terminal ===
  run() calls   = 1 (expected 1)
  outcome       = AbortError: run budget exhausted
  PASS

[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=primary-provider/primary-model reason=rate_limit next=ollama/fb1-model detail=provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb1-model reason=rate_limit next=ollama/fb2-model detail=provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb2-model reason=rate_limit next=ollama/fb3-model detail=provider rate limited
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb3-model reason=rate_limit next=none detail=provider rate limited

=== B1. RateLimitError (non-terminal provider failure) — cascade 4 times ===
  run() calls   = 4 (expected 4)
  outcome       = FallbackSummaryError: All models failed (4): primary-provider/primary-model: provider rate limited (rate_limit) | ...
  PASS

[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=primary-provider/primary-model reason=timeout next=ollama/fb1-model detail=aborted
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb1-model reason=timeout next=ollama/fb2-model detail=aborted
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb2-model reason=timeout next=ollama/fb3-model detail=aborted
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=primary-provider/primary-model candidate=ollama/fb3-model reason=timeout next=none detail=aborted

=== B2. PROVIDER-STYLE AbortError(cause: TimeoutError) WITHOUT marker — cascade 4 times ===
  run() calls   = 4 (expected 4)
  outcome       = FallbackSummaryError: All models failed (4): primary-provider/primary-model: aborted (timeout) | ollama/fb1-model: aborted (timeout) | ...
  PASS

Summary:
  PASS  A1-A4: cron prefix variants → 1 call each (regression sanity)
  PASS  A5: AbortError(cause: TimeoutError) WITH marker (run-budget shape) → 1 call (terminal)
  PASS  B1: RateLimitError → 4 calls cascade (provider-retry semantics)
  PASS  B2: AbortError(cause: TimeoutError) WITHOUT marker (provider-style) → 4 calls cascade (BOT'S P1 CONCERN ADDRESSED)

The A5/B2 contrast is the key new evidence for the marker fix: both shapes are byte-identical except for the OPENCLAW_ABORTABLE_WRAPPER symbol marker. A5 (marker present) → terminal short-circuit. B2 (marker absent, simulating a provider SDK wrapper) → emits the exact 4 [model-fallback/decision] decision=candidate_failed reason=timeout next=... log lines, cascading through every configured candidate just like before this PR. Per-provider retryable timeout semantics are preserved by design, not by accident.

Human Verification

• Verified scenarios:
  • All 70 tests in model-fallback.test.ts pass after the change
  • All 20 tests in model-fallback.probe.test.ts pass
  • All 8 tests in agent-command.live-model-switch.test.ts pass
  • End-to-end smoke: container rebuilt from this branch, HTTP client disconnect on /v1/responses produces the expected [openresponses] client disconnected, aborting streaming run runId=... log line and the agent run terminates within ~1s (the upstream watchClientDisconnect plumbing already in aad3bbedd works correctly with the new ClientDisconnectError reason tag)
• Edge cases checked: cause-chain walking (one level deep) for TimeoutError/ClientDisconnectError wrapped inside an outer AbortError -- this matches pi-embedded-runner/run/attempt.ts:1387's makeAbortError pattern
• What I did not verify:
  • Direct E2E of the fallback-skip path with a real cron timeout firing (would require setting timeoutSeconds to a very small value; the unit tests cover the contract)
  • The image-model fallback variant (runWithImageModelFallback) -- it gets the new abortSignal? parameter for consistency but no callers currently pass a signal

Compatibility / Migration

• Backward compatible? Yes -- the new abortSignal? parameter is optional. Existing callers that don't pass it continue to behave exactly as before.
• Config/env changes? No
• Migration needed? No

Risks and Mitigations

• Risk: A future caller passes an abort signal whose signal.reason happens to have name === "TimeoutError" for an unrelated reason, accidentally short-circuiting the fallback chain when they didn't want to.
  • Mitigation: The check is intentionally narrow -- only the exact name strings TimeoutError and ClientDisconnectError match. Both names are already reserved for run-budget timeout (set by makeTimeoutAbortReason in pi-embedded-runner/run/attempt.ts) and HTTP client disconnect (set by watchClientDisconnect in gateway/http-common.ts after this PR).
• Risk: TimeoutError from a per-provider request timeout (not run-budget) gets mistaken for a run-budget timeout and skips fallback when it shouldn't.
  • Mitigation: Per-provider timeouts come from the LLM SDK's internal fetch timeout, which throws an error directly -- they don't propagate through runAbortController.abort(). Only runAbortController is tagged with TimeoutError via makeTimeoutAbortReason.

Out of scope (for separate PRs)

• Issue Cron job timeout aborts entire model fallback chain via shared AbortController #37505 (cron timeout aborts entire fallback chain via shared AbortController) -- different abort source (cron service executeJobCoreWithTimeout), needs per-attempt AbortController isolation. PR fix(cron): per-attempt AbortControllers and deferred execution timeout #42482 already addresses this.
• Issue [bug] Fallback chain aborted by premature primary restore when cooldown expires mid-flight #58578 (fallback chain aborted by premature primary restore mid-flight) -- different abort source (requestLiveSessionModelSwitch), needs coordination state between live-model-switch and fallback layers

Relationship to PR #52365

PR #52365 (fix(cron): stop fallback attempts when cron budget is exhausted) addresses the same underlying problem from #60388 but with a different, complementary mechanism:

• fix(cron): stop fallback attempts when cron budget is exhausted #52365 is proactive: a new beforeAttempt hook in runWithModelFallback that lets the cron layer check its remaining budget before each attempt and stop the chain if budget is too low.
• This PR is reactive: an isTerminalAbort(signal) check in shouldRethrowAbort that inspects signal.reason after an attempt aborts to decide whether to rethrow or retry.

These are complementary, not exclusive. #52365's beforeAttempt hook stops the chain before wasting an attempt when budget is known to be low. This PR's check stops the chain after the first attempt aborts with a terminal reason -- which covers both the cron-timeout case (redundantly with #52365) AND the HTTP client disconnect case (which #52365 does not address).

Notable differences:

• Scope: this PR is 8 files / ~276 lines touching only model-fallback.ts, http-common.ts, agent-command.ts, 3 auto-reply callers, and one cron caller. fix(cron): stop fallback attempts when cron budget is exhausted #52365 is 47 files / ~1900 lines (includes unrelated slack/plugin-sdk changes).
• Client disconnect coverage: this PR handles the HTTP /v1/responses and /v1/chat/completions client-disconnect case by tagging the abort with ClientDisconnectError in watchClientDisconnect. fix(cron): stop fallback attempts when cron budget is exhausted #52365 does not touch the gateway HTTP path.
• Detection layer: this PR checks signal.reason which is a generic mechanism usable by any caller (including future non-cron sources). fix(cron): stop fallback attempts when cron budget is exhausted #52365's beforeAttempt hook requires each caller to implement its own budget-check logic.

Either PR alone fixes the #60388 cron-timeout case. Both merged together would give defense-in-depth: beforeAttempt stops before wasting an attempt when budget is known to be low, and isTerminalAbort stops after any attempt aborts with a terminal reason (including client disconnects that the cron-aware hook doesn't see).

If the maintainers prefer #52365's approach and would rather not have two overlapping mechanisms, I'd suggest retargeting this PR to cover only the ClientDisconnectError branch of isTerminalAbort (the unique coverage) and letting #52365 handle the cron-timeout case via its proactive hook.