Problem
Isolated cron / scheduled agentTurn jobs on embedded (non-CLI) providers with a configured model fallback chain silently drop the answer when the primary model returns a result-level terminal failure (reasoning-only, empty-visible, or incomplete_turn). The configured fallback model is never called, the cron run is recorded as status: "error", and the operator receives the generic "Agent couldn't generate a response." payload, even though a healthy fallback model was configured and would have produced the real answer.
This contradicts the documented contract. docs/automation/cron-jobs.md:179 and docs/automation/cron-jobs.md:486 both state that configured fallback chains still apply when the job primary fails. Thrown FailoverErrors (auth, rate-limit, overload) do engage the chain via the catch path. Only the returned result-level failure class is missed.
Reasoning-only / empty-visible output is a routine outcome for reasoning models, and a cron job fires repeatedly, so the failure recurs every run until a human notices.
Pinned to HEAD 3217165be77071339f599cd1f5f0f341bc5af8dd.
Root cause
The cron embedded-run path calls runWithModelFallback without a result classifier:
-
src/cron/isolated-agent/run-executor.ts:276-454 builds and awaits runWithModelFallback({...}) for both the CLI and embedded branches, but passes no classifyResult and no mergeExhaustedResult. Every other answer-bearing caller passes both: src/agents/agent-command.ts:1870 / :1876 and src/auto-reply/reply/agent-runner-execution.ts:2143 / :2158. grep -rn "classifyResult\|mergeExhaustedResult" src/cron/ returns zero hits.
-
runEmbeddedAgent does not throw for a fallback-safe incomplete turn. It returns a result carrying meta.error.kind = "incomplete_turn", fallbackSafe: true, an isError payload, and meta.agentHarnessResultClassification (for example "reasoning-only"): see src/agents/embedded-agent-runner/run.ts:3871-3909 and :3960-3988.
-
In runWithModelFallback, with classifyResult undefined, the runResult.ok branch at src/agents/model-fallback.ts:419-471 resolves classification = undefined, so classifiedError = null, and the first candidate is returned as a success. The loop never advances to candidate 2+, and the exhaustion-preserve branch (src/agents/model-fallback.ts:1845-1865, gated on mergeExhaustedResult) is dead for cron.
-
Cron then reads runResult = fallbackResult.result (src/cron/isolated-agent/run-executor.ts:455), and the returned meta.error is turned into hasFatalErrorPayload = true at src/cron/isolated-agent/helpers.ts:331-333, so the run is recorded as status: "error" and the "Agent couldn't generate a response." payload is delivered.
The interactive and auto-reply paths, which do pass the classifier and merge helper, classify the same returned result and advance to the healthy fallback, delivering the real answer.
Reproduction
The following repro drives the real runWithModelFallback from src/agents/model-fallback.js with the real classifier (classifyEmbeddedAgentRunResultForModelFallback) and merge helper (mergeEmbeddedAgentRunResultForModelFallbackExhaustion), twice: once with interactive wiring (classifier + merge passed) and once with the exact cron wiring from run-executor.ts:276 (neither passed). The primary returns a fallback-safe incomplete_turn / reasoning-only result; the fallback returns a real answer.
Place at src/cron/isolated-agent/cron-result-fallback-gap.repro.test.ts and run with node scripts/run-vitest.mjs src/cron/isolated-agent/cron-result-fallback-gap.repro.test.ts.
import {
createContractRunResult,
OUTCOME_FALLBACK_RUNTIME_CONTRACT,
} from "openclaw/plugin-sdk/agent-runtime-test-contracts";
import { describe, expect, it, vi } from "vitest";
import {
classifyEmbeddedAgentRunResultForModelFallback,
mergeEmbeddedAgentRunResultForModelFallbackExhaustion,
} from "../../agents/embedded-agent-runner/result-fallback-classifier.js";
import { runWithModelFallback } from "../../agents/model-fallback.js";
vi.mock("../../agents/auth-profiles/source-check.js", () => ({
hasAnyAuthProfileStoreSource: () => false,
}));
const fallbackOverride = [
`${OUTCOME_FALLBACK_RUNTIME_CONTRACT.fallbackProvider}/${OUTCOME_FALLBACK_RUNTIME_CONTRACT.fallbackModel}`,
];
function buildPrimaryReasoningOnlyResult() {
return createContractRunResult({
payloads: [{ text: "Agent couldn't generate a response.", isError: true }],
meta: {
durationMs: 1,
agentHarnessResultClassification: "reasoning-only",
error: {
kind: "incomplete_turn",
message: "Agent couldn't generate a response.",
fallbackSafe: true,
terminalPresentation: false,
},
},
});
}
function buildHealthyFallbackResult() {
return createContractRunResult({
payloads: [{ text: "Workspace cleaned up: removed 12 stale files." }],
meta: { durationMs: 1, finalAssistantVisibleText: "Workspace cleaned up: removed 12 stale files." },
});
}
describe("cron result-level fallback gap", () => {
it("INTERACTIVE wiring engages the fallback (classifier + merge passed)", async () => {
const run = vi
.fn()
.mockResolvedValueOnce(buildPrimaryReasoningOnlyResult())
.mockResolvedValueOnce(buildHealthyFallbackResult());
const result = await runWithModelFallback<ReturnType<typeof createContractRunResult>>({
cfg: undefined,
provider: OUTCOME_FALLBACK_RUNTIME_CONTRACT.primaryProvider,
model: OUTCOME_FALLBACK_RUNTIME_CONTRACT.primaryModel,
fallbacksOverride: fallbackOverride,
run,
classifyResult: ({ provider, model, result: resultValue }) =>
classifyEmbeddedAgentRunResultForModelFallback({ provider, model, result: resultValue }),
mergeExhaustedResult: mergeEmbeddedAgentRunResultForModelFallbackExhaustion,
skipAuthProfileRuntime: true,
});
expect(run).toHaveBeenCalledTimes(2);
expect(result.provider).toBe(OUTCOME_FALLBACK_RUNTIME_CONTRACT.fallbackProvider);
expect(result.result.payloads).toEqual([
{ text: "Workspace cleaned up: removed 12 stale files." },
]);
});
it("CRON wiring (run-executor.ts:276) drops the fallback (no classifier, no merge)", async () => {
const run = vi
.fn()
.mockResolvedValueOnce(buildPrimaryReasoningOnlyResult())
.mockResolvedValueOnce(buildHealthyFallbackResult());
const result = await runWithModelFallback<ReturnType<typeof createContractRunResult>>({
cfg: undefined,
provider: OUTCOME_FALLBACK_RUNTIME_CONTRACT.primaryProvider,
model: OUTCOME_FALLBACK_RUNTIME_CONTRACT.primaryModel,
fallbacksOverride: fallbackOverride,
run,
skipAuthProfileRuntime: true,
});
expect(run).toHaveBeenCalledTimes(2);
expect(result.provider).toBe(OUTCOME_FALLBACK_RUNTIME_CONTRACT.fallbackProvider);
expect(result.result.payloads).toEqual([
{ text: "Workspace cleaned up: removed 12 stale files." },
]);
});
});
RED output on main at HEAD 3217165be77071339f599cd1f5f0f341bc5af8dd. The INTERACTIVE test passes (fallback engaged, run called twice). The CRON test fails (fallback dropped, run called only once):
RUN v4.1.8
❯ |cron| src/cron/isolated-agent/cron-result-fallback-gap.repro.test.ts (2 tests | 1 failed)
× CRON wiring (run-executor.ts:276) drops the fallback (no classifier, no merge)
FAIL src/cron/isolated-agent/cron-result-fallback-gap.repro.test.ts > cron result-level fallback gap > CRON wiring (run-executor.ts:276) drops the fallback (no classifier, no merge)
AssertionError: expected "vi.fn()" to be called 2 times, but got 1 times
❯ src/cron/isolated-agent/cron-result-fallback-gap.repro.test.ts:84:17
84| expect(run).toHaveBeenCalledTimes(2);
Test Files 1 failed (1)
Tests 1 failed | 1 passed (2)
The single failing assertion (run called once instead of twice) is the gap: the cron wiring never advances to the configured fallback model for a returned result-level failure, while the interactive wiring does.
Trigger
- An isolated cron / scheduled
agentTurn job on an embedded (non-CLI) provider.
- A configured fallback chain (agent
model.fallbacks or per-job payload fallbacks).
- The primary model returns a fallback-safe result-level terminal failure: reasoning-only, empty-visible, or
incomplete_turn (a routine outcome for reasoning models, not a thrown error).
The fallback model is never called; the cron is marked error and delivers the generic failure text.
Impact / Severity
P1. Cron is a core path and the failure is recurring (it repeats every run until a human intervenes). Reasoning-only / empty-visible output is a routine outcome for reasoning models, so the trigger is common, not exotic. The user gets recurring cron failures and never receives the answer a configured fallback chain would have produced, which is exactly the availability guarantee the documented fallback contract promises.
Relationship to existing issues
This is distinct from the closest existing items:
Suggested direction
Not prescriptive. Pass the embedded result classifier and the merge helper into the cron runWithModelFallback call as the interactive and auto-reply paths do (classifyResult: classifyEmbeddedAgentRunResultForModelFallback, mergeExhaustedResult: mergeEmbeddedAgentRunResultForModelFallbackExhaustion), and have the cron path honor outcome === "exhausted" when deciding the final cron status. Scope this to the embedded branch of run-executor.ts, not the runCliAgent branch, since CLI providers already surface their own terminal classification.
Problem
Isolated cron / scheduled
agentTurnjobs on embedded (non-CLI) providers with a configured model fallback chain silently drop the answer when the primary model returns a result-level terminal failure (reasoning-only, empty-visible, orincomplete_turn). The configured fallback model is never called, the cron run is recorded asstatus: "error", and the operator receives the generic "Agent couldn't generate a response." payload, even though a healthy fallback model was configured and would have produced the real answer.This contradicts the documented contract.
docs/automation/cron-jobs.md:179anddocs/automation/cron-jobs.md:486both state that configured fallback chains still apply when the job primary fails. ThrownFailoverErrors (auth, rate-limit, overload) do engage the chain via the catch path. Only the returned result-level failure class is missed.Reasoning-only / empty-visible output is a routine outcome for reasoning models, and a cron job fires repeatedly, so the failure recurs every run until a human notices.
Pinned to HEAD
3217165be77071339f599cd1f5f0f341bc5af8dd.Root cause
The cron embedded-run path calls
runWithModelFallbackwithout a result classifier:src/cron/isolated-agent/run-executor.ts:276-454builds and awaitsrunWithModelFallback({...})for both the CLI and embedded branches, but passes noclassifyResultand nomergeExhaustedResult. Every other answer-bearing caller passes both:src/agents/agent-command.ts:1870/:1876andsrc/auto-reply/reply/agent-runner-execution.ts:2143/:2158.grep -rn "classifyResult\|mergeExhaustedResult" src/cron/returns zero hits.runEmbeddedAgentdoes not throw for a fallback-safe incomplete turn. It returns a result carryingmeta.error.kind = "incomplete_turn",fallbackSafe: true, anisErrorpayload, andmeta.agentHarnessResultClassification(for example"reasoning-only"): seesrc/agents/embedded-agent-runner/run.ts:3871-3909and:3960-3988.In
runWithModelFallback, withclassifyResultundefined, therunResult.okbranch atsrc/agents/model-fallback.ts:419-471resolvesclassification = undefined, soclassifiedError = null, and the first candidate is returned as a success. The loop never advances to candidate 2+, and the exhaustion-preserve branch (src/agents/model-fallback.ts:1845-1865, gated onmergeExhaustedResult) is dead for cron.Cron then reads
runResult = fallbackResult.result(src/cron/isolated-agent/run-executor.ts:455), and the returnedmeta.erroris turned intohasFatalErrorPayload = trueatsrc/cron/isolated-agent/helpers.ts:331-333, so the run is recorded asstatus: "error"and the "Agent couldn't generate a response." payload is delivered.The interactive and auto-reply paths, which do pass the classifier and merge helper, classify the same returned result and advance to the healthy fallback, delivering the real answer.
Reproduction
The following repro drives the real
runWithModelFallbackfromsrc/agents/model-fallback.jswith the real classifier (classifyEmbeddedAgentRunResultForModelFallback) and merge helper (mergeEmbeddedAgentRunResultForModelFallbackExhaustion), twice: once with interactive wiring (classifier + merge passed) and once with the exact cron wiring fromrun-executor.ts:276(neither passed). The primary returns a fallback-safeincomplete_turn/reasoning-onlyresult; the fallback returns a real answer.Place at
src/cron/isolated-agent/cron-result-fallback-gap.repro.test.tsand run withnode scripts/run-vitest.mjs src/cron/isolated-agent/cron-result-fallback-gap.repro.test.ts.RED output on main at HEAD
3217165be77071339f599cd1f5f0f341bc5af8dd. The INTERACTIVE test passes (fallback engaged,runcalled twice). The CRON test fails (fallback dropped,runcalled only once):The single failing assertion (
runcalled once instead of twice) is the gap: the cron wiring never advances to the configured fallback model for a returned result-level failure, while the interactive wiring does.Trigger
agentTurnjob on an embedded (non-CLI) provider.model.fallbacksor per-job payloadfallbacks).incomplete_turn(a routine outcome for reasoning models, not a thrown error).The fallback model is never called; the cron is marked
errorand delivers the generic failure text.Impact / Severity
P1. Cron is a core path and the failure is recurring (it repeats every run until a human intervenes). Reasoning-only / empty-visible output is a routine outcome for reasoning models, so the trigger is common, not exotic. The user gets recurring cron failures and never receives the answer a configured fallback chain would have produced, which is exactly the availability guarantee the documented fallback contract promises.
Relationship to existing issues
This is distinct from the closest existing items:
classifyResultis wired (agent-command line 711 in their build), and hypothesizes a guard inside the classifier short-circuiting to null. This finding is the cron surface, where the classifier and merge helper are entirely absent from therunWithModelFallbackcall (zero hits forclassifyResult/mergeExhaustedResultinsrc/cron/). Different surface, different provable root cause: the wiring is simply missing, not mis-guarded.upstream_error/LLM request failed): that is about a provider error payload on the general embedded path. This finding is the cron-specific missing classifier wiring for the reasoning-only / empty-visible /incomplete_turnresult class.agentTurnas non-fatal whenfinalAssistantVisibleTextexists): that PR edits cron outcome classification inhelpers.tsfor a run that already completed and recovered; it does not touchrunWithModelFallbackand does not engage the fallback chain. This finding is upstream of that: the configured fallback model is never even called.Suggested direction
Not prescriptive. Pass the embedded result classifier and the merge helper into the cron
runWithModelFallbackcall as the interactive and auto-reply paths do (classifyResult: classifyEmbeddedAgentRunResultForModelFallback,mergeExhaustedResult: mergeEmbeddedAgentRunResultForModelFallbackExhaustion), and have the cron path honoroutcome === "exhausted"when deciding the final cron status. Scope this to the embedded branch ofrun-executor.ts, not therunCliAgentbranch, since CLI providers already surface their own terminal classification.