fix(cron): engage model fallback on embedded result-level failures (reasoning-only / empty / incomplete_turn)#96529
Conversation
|
Thanks for the context here. I did a careful shell check against current Current main already implements the central isolated-cron result-level fallback fix through the merged reliability batch, with wiring coverage in the existing cron fallback tests; this branch is now redundant even though the latest stable release still lacks the fix. Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. So I’m closing this as already implemented rather than keeping a duplicate issue open. Review detailsBest possible solution: Keep the merged current-main implementation from #100483 and close this redundant branch; track any future CLI-specific cron classifier concern separately with a concrete repro. Do we have a high-confidence way to reproduce the issue? Yes for the pre-fix behavior, at source level: v2026.6.11 lacks the classifier hooks and runWithModelFallback requires a classifier to advance returned result-level failures. Current main no longer reproduces because the hooks are wired in. Is this the best way to solve the issue? Yes, current main is the best current solution for the reported gap: it reuses the canonical embedded classifier and merge helper in isolated cron, matching sibling fallback surfaces. The PR's extra CLI guard is not needed to close the embedded result-level failure report. Security review: Security review cleared: No security or supply-chain concern found; the PR only changes TypeScript cron fallback wiring and tests, with no dependencies, workflows, credentials, packaging, or permissions touched. AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against bf04d049fa36; fix evidence: commit aaf5ab910cbe, main fix timestamp 2026-07-06T00:08:51+01:00. |
|
Real-world reproduction confirming this gap. We're running OpenClaw 2026.6.10 with a local llama-server (Qwen3-8B Q5_K_M) as primary and deepseek/deepseek-v4-pro as fallback. Our exact failure pattern:
Gateway log evidence: The Our workaround (not a fix): Added This PR would fix that gap. Happy to provide more logs or run a specific test if it helps with the proof the maintainers are requesting. |
|
Added real-behavior proof: loopback real-HTTP transport reproduces the gap and verifies the fix end-to-end Per the "needs real behavior proof from a real setup" gate — an end-to-end run where every layer is real production code: the real A — real transport: empty B — real classifier flags it as fallback-eligible C — real
The cron @nessim-liamani — thanks for the production confirmation, that's genuinely helpful. One precise note for maintainers reading the thread: the pasted log line ( |
|
@clawsweeper re-review The "needs real behavior proof from a real setup" item is now addressed: the PR body's new Real-behavior proof section (and the comment above) captures an end-to-end run over a loopback real-HTTP |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Cross-checking #97115 against this PR: the returned-result part of #97115 appears covered here, and the timeout/abort part should stay separate from this landing decision. Evidence I checked:
So for maintainer triage, I would treat #97115's returned-result failure as a duplicate/partial duplicate of #96525 and keep this PR as the canonical fix. The separate "shared AbortController kills later fallbacks" claim in #97115 is a timeout/abort path and is not proven or covered by this PR. |
Isolated cron agentTurn runs called runWithModelFallback without a result classifier or merge helper, so returned fallback-safe failures (reasoning-only, empty-visible, incomplete_turn) were treated as the first candidate's success. The configured fallback model was never called and the run was recorded as an error, contradicting the documented fallback contract honored by the interactive and auto-reply paths. Wire classifyEmbeddedAgentRunResultForModelFallback and mergeEmbeddedAgentRunResultForModelFallbackExhaustion into the cron runWithModelFallback call, scoped to the embedded branch so the CLI runner keeps owning its own terminal classification.
3802123 to
4af4e70
Compare
|
ClawSweeper applied the proposed close for this PR.
|
|
🦞👀 Reason: structured ClawSweeper close marker: close-required (sha=4af4e70c22d0924a14e62194dc87e823b2b0d439) Usage: |
What Problem This Solves
Isolated cron / scheduled
agentTurnjobs on embedded (non-CLI) providers with a configured model fallback chain silently drop the answer when the primary model returns a result-level terminal failure (reasoning-only, empty-visible, orincomplete_turn). The configured fallback model is never called, the run is recorded asstatus: "error", and the operator receives the generic "Agent couldn't generate a response." payload — even though a healthy fallback model was configured and would have produced the real answer.This contradicts the documented contract (
docs/automation/cron-jobs.md) that configured fallback chains still apply when the job primary fails. ThrownFailoverErrors (auth, rate-limit, overload) already engage the chain via the catch path; only the returned result-level failure class was missed. Because reasoning-only / empty output is a routine outcome for reasoning models and cron fires repeatedly, the failure recurs every run until a human notices.Why This Change Was Made
The cron embedded-run path (
src/cron/isolated-agent/run-executor.ts) calledrunWithModelFallbackwithout aclassifyResultormergeExhaustedResult. Every other answer-bearing caller passes both (src/agents/agent-command.ts,src/auto-reply/reply/agent-runner-execution.ts).runEmbeddedAgentdoes not throw for a fallback-safe incomplete turn — it returns a result carryingmeta.error.kind = "incomplete_turn",fallbackSafe: true, andmeta.agentHarnessResultClassification(e.g."reasoning-only"). WithclassifyResultundefined,runWithModelFallbackresolves no classification, returns the first candidate as a success, and never advances to candidate 2+. Cron then turns the returnedmeta.errorinto a fatal error payload and recordsstatus: "error".This change wires
classifyEmbeddedAgentRunResultForModelFallbackandmergeEmbeddedAgentRunResultForModelFallbackExhaustioninto the cron call, exactly as the interactive and auto-reply paths do. Classification is scoped to the embedded branch (a sharedresolveCronExecutionProviderhelper reused by the run path and the classifier), so the CLI runner keeps owning its own terminal classification and CLI fallback semantics are unchanged.User Impact
Scheduled/cron
agentTurnjobs on embedded providers with a configured fallback chain (model.fallbacksor per-job payloadfallbacks) now advance to the healthy fallback model when the primary returns a reasoning-only / empty-visible /incomplete_turnresult, delivering the real answer instead of recurring "Agent couldn't generate a response." errors. CLI-provider cron jobs are unaffected.Evidence
Regression test
New regression test
src/cron/isolated-agent/run.result-fallback-gap.test.tsdrives the realrun-executorwiring:classifyResult/mergeExhaustedResultthe cron path now passes; the primary embedded run returns a reasoning-onlyincomplete_turnresult and the fallback returns a healthy answer. The embedded runner is reached twice (second call = the configured fallback model) and the cron status isok.classifyResultreturns a classification for an embedded reasoning-only result but returnsnullfor a CLI provider;mergeExhaustedResultis the embedded merge helper.Both assertions are RED on
main(classifyResultisundefined; the loop stops on the first reasoning-only result) and GREEN with this change. Verified locally:Adjacent cron suites stay green (
run.payload-fallbacks,run.meta-error-status,run-fallback-policy,run.cron-model-override,run.live-session-model-switch,run.interim-retry— 45 tests). Changed files formatted withoxfmt.Real-behavior proof (loopback real-HTTP transport)
End-to-end run where every layer is real production code — the real
openai-completionstransport, the realclassifyEmbeddedAgentRunResultForModelFallback, and the realrunWithModelFallbackloop. The only injected piece is a loopback OpenAI-compatible model server, which is the only way to deterministically force the reasoning-only / empty-content primary response (the Qwen3 thinking-mode leak independently reported in production on this thread).A — real transport: empty
content+ populatedreasoning_contentyields empty visible text:B — real classifier flags the empty / reasoning-only result as fallback-eligible:
C — real
runWithModelFallbackover real HTTP, without vs with the classifier the cron path now passes:before(= cron onmain, called with noclassifyResult) treats the empty primary as success and never contacts the fallback (serverHitshas only the primary).after(this PR) advances to the configured fallback over real HTTP (serverHitsincludeshealthy-fallback) and delivers the real answer. Theincomplete_turn/fallbackSaferesult shape used in B/C is what the embedded runner already produces for this case and is unchanged by this PR.AI-assisted.