Skip to content

fix(cron): engage model fallback on embedded result-level failures (reasoning-only / empty / incomplete_turn)#96529

Closed
ZengWen-DT wants to merge 2 commits into
openclaw:mainfrom
ZengWen-DT:fix/96525-cron-result-fallback-classifier
Closed

fix(cron): engage model fallback on embedded result-level failures (reasoning-only / empty / incomplete_turn)#96529
ZengWen-DT wants to merge 2 commits into
openclaw:mainfrom
ZengWen-DT:fix/96525-cron-result-fallback-classifier

Conversation

@ZengWen-DT

@ZengWen-DT ZengWen-DT commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Isolated cron / scheduled agentTurn jobs on embedded (non-CLI) providers with a configured model fallback chain silently drop the answer when the primary model returns a result-level terminal failure (reasoning-only, empty-visible, or incomplete_turn). The configured fallback model is never called, the run is recorded as status: "error", and the operator receives the generic "Agent couldn't generate a response." payload — even though a healthy fallback model was configured and would have produced the real answer.

This contradicts the documented contract (docs/automation/cron-jobs.md) that configured fallback chains still apply when the job primary fails. Thrown FailoverErrors (auth, rate-limit, overload) already engage the chain via the catch path; only the returned result-level failure class was missed. Because reasoning-only / empty output is a routine outcome for reasoning models and cron fires repeatedly, the failure recurs every run until a human notices.

Why This Change Was Made

The cron embedded-run path (src/cron/isolated-agent/run-executor.ts) called runWithModelFallback without a classifyResult or mergeExhaustedResult. Every other answer-bearing caller passes both (src/agents/agent-command.ts, src/auto-reply/reply/agent-runner-execution.ts).

runEmbeddedAgent does not throw for a fallback-safe incomplete turn — it returns a result carrying meta.error.kind = "incomplete_turn", fallbackSafe: true, and meta.agentHarnessResultClassification (e.g. "reasoning-only"). With classifyResult undefined, runWithModelFallback resolves no classification, returns the first candidate as a success, and never advances to candidate 2+. Cron then turns the returned meta.error into a fatal error payload and records status: "error".

This change wires classifyEmbeddedAgentRunResultForModelFallback and mergeEmbeddedAgentRunResultForModelFallbackExhaustion into the cron call, exactly as the interactive and auto-reply paths do. Classification is scoped to the embedded branch (a shared resolveCronExecutionProvider helper reused by the run path and the classifier), so the CLI runner keeps owning its own terminal classification and CLI fallback semantics are unchanged.

User Impact

Scheduled/cron agentTurn jobs on embedded providers with a configured fallback chain (model.fallbacks or per-job payload fallbacks) now advance to the healthy fallback model when the primary returns a reasoning-only / empty-visible / incomplete_turn result, delivering the real answer instead of recurring "Agent couldn't generate a response." errors. CLI-provider cron jobs are unaffected.

Evidence

Regression test

New regression test src/cron/isolated-agent/run.result-fallback-gap.test.ts drives the real run-executor wiring:

  • Engages the fallback: a faithful minimal fallback loop exercises the real classifyResult/mergeExhaustedResult the cron path now passes; the primary embedded run returns a reasoning-only incomplete_turn result and the fallback returns a healthy answer. The embedded runner is reached twice (second call = the configured fallback model) and the cron status is ok.
  • Scoped to embedded: the captured request's classifyResult returns a classification for an embedded reasoning-only result but returns null for a CLI provider; mergeExhaustedResult is the embedded merge helper.

Both assertions are RED on main (classifyResult is undefined; the loop stops on the first reasoning-only result) and GREEN with this change. Verified locally:

node scripts/run-vitest.mjs src/cron/isolated-agent/run.result-fallback-gap.test.ts
 Test Files  1 passed (1)
      Tests  2 passed (2)

Adjacent cron suites stay green (run.payload-fallbacks, run.meta-error-status, run-fallback-policy, run.cron-model-override, run.live-session-model-switch, run.interim-retry — 45 tests). Changed files formatted with oxfmt.

Real-behavior proof (loopback real-HTTP transport)

End-to-end run where every layer is real production code — the real openai-completions transport, the real classifyEmbeddedAgentRunResultForModelFallback, and the real runWithModelFallback loop. The only injected piece is a loopback OpenAI-compatible model server, which is the only way to deterministically force the reasoning-only / empty-content primary response (the Qwen3 thinking-mode leak independently reported in production on this thread).

A — real transport: empty content + populated reasoning_content yields empty visible text:

[A] real-transport primary: {"text":"","thinking":"Let me think..."} serverHits: ["empty-primary-qwen"]

B — real classifier flags the empty / reasoning-only result as fallback-eligible:

[B] classification: {"message":"Agent couldn't generate a response.","reason":"format","code":"incomplete_result","preserveResultOnExhaustion":true,"preserveResultPriority":0}

C — real runWithModelFallback over real HTTP, without vs with the classifier the cron path now passes:

[C-before no classifier]  winnerModel: empty-primary-qwen  visible: (none)  serverHits: ["empty-primary-qwen"]
[C-after  with classifier] winnerModel: healthy-fallback   visible: Workspace cleaned up: removed 12 stale files.  serverHits: ["empty-primary-qwen","healthy-fallback"]

before (= cron on main, called with no classifyResult) treats the empty primary as success and never contacts the fallback (serverHits has only the primary). after (this PR) advances to the configured fallback over real HTTP (serverHits includes healthy-fallback) and delivers the real answer. The incomplete_turn / fallbackSafe result shape used in B/C is what the embedded runner already produces for this case and is unchanged by this PR.

AI-assisted.

@clawsweeper

clawsweeper Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Current main already implements the central isolated-cron result-level fallback fix through the merged reliability batch, with wiring coverage in the existing cron fallback tests; this branch is now redundant even though the latest stable release still lacks the fix.

Root-cause cluster
Relationship: superseded
Canonical: #100483
Summary: The current PR and related candidates target the isolated cron embedded result-level fallback classifier gap; the merged reliability batch is now the canonical implementation on main.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Keep the merged current-main implementation from #100483 and close this redundant branch; track any future CLI-specific cron classifier concern separately with a concrete repro.

Do we have a high-confidence way to reproduce the issue?

Yes for the pre-fix behavior, at source level: v2026.6.11 lacks the classifier hooks and runWithModelFallback requires a classifier to advance returned result-level failures. Current main no longer reproduces because the hooks are wired in.

Is this the best way to solve the issue?

Yes, current main is the best current solution for the reported gap: it reuses the canonical embedded classifier and merge helper in isolated cron, matching sibling fallback surfaces. The PR's extra CLI guard is not needed to close the embedded result-level failure report.

Security review:

Security review cleared: No security or supply-chain concern found; the PR only changes TypeScript cron fallback wiring and tests, with no dependencies, workflows, credentials, packaging, or permissions touched.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • steipete: Merged the reliability batch that landed the cron fallback classification fix and appears repeatedly in recent history for the cron executor, fallback helper, and classifier files. (role: recent merger and fallback/cron history owner; confidence: high; commits: aaf5ab910cbe, 0314819f918a, be5906e19dd7; files: src/cron/isolated-agent/run-executor.ts, src/cron/isolated-agent/run.payload-fallbacks.test.ts, src/agents/model-fallback.ts)
  • vincentkoc: GitHub path history shows recent fallback/error-normalization and cron runtime work around the same execution boundary. (role: recent area contributor; confidence: medium; commits: 80805ad7a583, aa3797c8d0d7, 14e448e0e13d; files: src/agents/model-fallback.ts, src/cron/isolated-agent/run-executor.ts, src/agents/agent-command.ts)
  • jincheng-xydt: The merged reliability batch credits the cron fallback classification slice to the source branch in fix #97115: classify isolated cron fallback results #99913. (role: source-fix contributor; confidence: medium; commits: aaf5ab910cbe; files: src/cron/isolated-agent/run-executor.ts, src/cron/isolated-agent/run.payload-fallbacks.test.ts)

Codex review notes: model internal, reasoning high; reviewed against bf04d049fa36; fix evidence: commit aaf5ab910cbe, main fix timestamp 2026-07-06T00:08:51+01:00.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jun 24, 2026
@nessim-liamani

Copy link
Copy Markdown

Real-world reproduction confirming this gap. We're running OpenClaw 2026.6.10 with a local llama-server (Qwen3-8B Q5_K_M) as primary and deepseek/deepseek-v4-pro as fallback.

Our exact failure pattern:

  1. Isolated cron agentTurn session starts (wa-revision-request-check, every 5 min)
  2. Session loads bootstrap, calls llama-server for the initial response
  3. llama-server returns content="" with reasoning_content populated (Qwen3-8B build b9638 thinking-mode leak — the model outputs reasoning tokens, the primary response is empty)
  4. The gateway receives this empty result, classifies it as an incomplete turn, and logs "Agent couldn't generate a response. Note: some tool actions may have already been executed"
  5. The cron reports lastRunStatus: ok, lastDurationMs: ~100s, lastDiagnosticSummary: "⚠️ Agent couldn't generate a response... verify before retrying"
  6. The fallback model (DeepSeek) is never called despite being configured
  7. The revision request remains stuck in status='processing' permanently
  8. Subsequent cron runs return "already-running" — the stuck session blocks all future runs until a disable/enable cycle

Gateway log evidence:

14:45:00 [model-fetch] error provider=llama-server model=qwen3-8b elapsedMs=5767 name=AbortError
14:45:00 cron: cleaned up timed-out agent run
14:45:00 embedded run failover decision: stage=assistant decision=surface_error

The model_fallback_decision events show the gateway correctly caught the error at provider-fetch level, but the embedded result-level failure (empty content) was never routed through the fallback classifier — it went straight to "couldn't generate a response" with no fallback attempt.

Our workaround (not a fix): Added --reasoning off to llama-server. Now the model returns actual content, bypassing the empty-response problem entirely. But the underlying fallback gap remains — any other model that returns reasoning-only/empty/incomplete output would hit the same dead end without an operator noticing.

This PR would fix that gap. Happy to provide more logs or run a specific test if it helps with the proof the maintainers are requesting.

@ZengWen-DT

Copy link
Copy Markdown
Contributor Author

Added real-behavior proof: loopback real-HTTP transport reproduces the gap and verifies the fix end-to-end

Per the "needs real behavior proof from a real setup" gate — an end-to-end run where every layer is real production code: the real openai-completions transport, the real classifyEmbeddedAgentRunResultForModelFallback, and the real runWithModelFallback loop. The only injected piece is a loopback OpenAI-compatible model server, which is the only way to deterministically force the reasoning-only / empty-content primary response (the exact Qwen3 thinking-mode leak @nessim-liamani hit in production).

A — real transport: empty content + populated reasoning_content → empty visible text

[A] real-transport primary: {"text":"","thinking":"Let me think..."} serverHits: ["empty-primary-qwen"]

B — real classifier flags it as fallback-eligible

[B] classification: {"message":"Agent couldn't generate a response.","reason":"format","code":"incomplete_result","preserveResultOnExhaustion":true,"preserveResultPriority":0}

C — real runWithModelFallback over real HTTP, without vs with the classifier the cron path now passes

[C-before no classifier]  winnerModel: empty-primary-qwen  visible: (none)  serverHits: ["empty-primary-qwen"]
[C-after  with classifier] winnerModel: healthy-fallback   visible: Workspace cleaned up: removed 12 stale files.  serverHits: ["empty-primary-qwen","healthy-fallback"]
  • before (= cron on main, which calls runWithModelFallback with no classifyResult): the loop treats the empty primary as success, the fallback model is never contacted (serverHits has only the primary), and no visible answer is produced.
  • after (this PR wires classifyResult + mergeExhaustedResult): the loop advances to the configured fallback over real HTTP (serverHits now includes healthy-fallback) and delivers the real answer.

The cron run-executor wiring (embedded branch passes the classifier, CLI scoped out) is covered by the committed regression run.result-fallback-gap.test.ts, which drives the real runCronIsolatedAgentTurn. The incomplete_turn / fallbackSafe result shape in B/C is what the embedded runner already produces for this case — unchanged by this PR — and matches both #96525 and the production report below.


@nessim-liamani — thanks for the production confirmation, that's genuinely helpful. One precise note for maintainers reading the thread: the pasted log line (name=AbortError + cron: cleaned up timed-out agent run) is the timeout/abort path, which throws and is a sibling of — not identical to — the returned result-level failure this PR targets (empty content / reasoning_content, no throw). Your step 3 ("returns content:"" with reasoning_content populated … classified as an incomplete turn") is exactly the returned path this PR fixes. Both are real; flagging the distinction so the proof above is read against the right failure mode.

@ZengWen-DT

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

The "needs real behavior proof from a real setup" item is now addressed: the PR body's new Real-behavior proof section (and the comment above) captures an end-to-end run over a loopback real-HTTP openai-completions transport — the real transport, the real classifyEmbeddedAgentRunResultForModelFallback, and the real runWithModelFallback loop. serverHits shows the configured fallback is never contacted without the classifier (cron on main) and is reached over real HTTP with it (this PR). Please re-evaluate the proof gate.

@clawsweeper

clawsweeper Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 25, 2026
@harjothkhara

Copy link
Copy Markdown
Contributor

Cross-checking #97115 against this PR: the returned-result part of #97115 appears covered here, and the timeout/abort part should stay separate from this landing decision.

Evidence I checked:

  • Current main still calls runWithModelFallback from src/cron/isolated-agent/run-executor.ts without classifyResult / mergeExhaustedResult, while src/agents/model-fallback.ts only retries an ok returned result when a classifier returns a classification.
  • The Bug: Cron fallback chain broken — models return zero-token success or killed by shared AbortController #97115 zero-token / empty-response slice maps to the existing embedded classifier's incomplete_turn, empty, and reasoning-only handling in src/agents/embedded-agent-runner/result-fallback-classifier.ts.
  • PR head 3802123dffe3 wires that classifier and exhaustion merge helper into the cron fallback call, scopes the classifier away from CLI candidates with resolveCronExecutionProvider(...).isCli, and adds src/cron/isolated-agent/run.result-fallback-gap.test.ts coverage for reaching the configured fallback and skipping CLI classification.

So for maintainer triage, I would treat #97115's returned-result failure as a duplicate/partial duplicate of #96525 and keep this PR as the canonical fix. The separate "shared AbortController kills later fallbacks" claim in #97115 is a timeout/abort path and is not proven or covered by this PR.

Isolated cron agentTurn runs called runWithModelFallback without a result
classifier or merge helper, so returned fallback-safe failures (reasoning-only,
empty-visible, incomplete_turn) were treated as the first candidate's success.
The configured fallback model was never called and the run was recorded as an
error, contradicting the documented fallback contract honored by the
interactive and auto-reply paths.

Wire classifyEmbeddedAgentRunResultForModelFallback and
mergeEmbeddedAgentRunResultForModelFallbackExhaustion into the cron
runWithModelFallback call, scoped to the embedded branch so the CLI runner
keeps owning its own terminal classification.
@ZengWen-DT
ZengWen-DT force-pushed the fix/96525-cron-result-fallback-classifier branch from 3802123 to 4af4e70 Compare July 6, 2026 03:34
@clawsweeper

clawsweeper Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@clawsweeper clawsweeper Bot closed this Jul 6, 2026
@clawsweeper

clawsweeper Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper could not autoclose this item.

Reason: structured ClawSweeper close marker: close-required (sha=4af4e70c22d0924a14e62194dc87e823b2b0d439)

Usage: /autoclose <maintainer close reason>. I will close this item and any open same-repo items explicitly referenced in the command text.

@ZengWen-DT
ZengWen-DT deleted the fix/96525-cron-result-fallback-classifier branch July 6, 2026 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants