Skip to content

model.fallbacks chain does not engage on reasoning-only / empty-visible-reply failures #85422

Description

@leatherneck-33

OpenClaw version observed: First observed on 2026.5.7; re-verified still present in 2026.5.18 (installed at ~/.npm-global/lib/node_modules/openclaw). On 2026.5.18 the symbols are at dist/result-fallback-classifier-Czg3Eu15.js (classifyEmbeddedPiRunResultForModelFallback), dist/agent-command-CbVp9Pf5.js (runWithModelFallback wire-in), and dist/pi-embedded-ydXicp1O.js (reasoningOnlyRetriesExhausted).
Filed by: SKP Team (Shawn Powers, skpteam.casa)
Date observed: First reasoning-only retries-exhausted in our gateway.err.log at 2026-05-04T06:03:51-05:00 (provider pace-openrouter/google/gemini-2.5-flash); continued daily; full agent outage 2026-05-14 18:00 → 2026-05-15 17:24 CT (~24h) on openai-codex/gpt-5.4.
Severity (from user perspective): High — the agent's configured model.fallbacks chain exists exactly so the agent stays available when the primary fails. When the primary fails by emitting reasoning-only or empty-visible turns, the fallback chain is silently skipped and the user-visible agent is unreachable until a human notices.


Summary

The gateway's reasoning-model retry path correctly classifies an empty-visible / reasoning-only turn (see dist/result-fallback-classifier-CrVa7J1V.js returning code: "reasoning_only_result" / "empty_result"). That classifier is also correctly passed as the classifyResult callback into runWithModelFallback (see dist/agent-command-CYAAR35V.js ~line 711). So at the architectural level, the wiring exists for reasoning-only outcomes to traverse the configured model.fallbacks chain.

But in practice, on our host, reasoning-only failures never traverse the fallback chain. The primary model retries twice with visible-answer continuation instructions, then surfaces incomplete-turn error to the user. The fallback chain (pace-openrouter/anthropic/claude-haiku-4-5, pace-openrouter/google/gemini-2.5-pro) is never attempted. We have ~24 hours of reasoning-only retries exhausted log entries for one provider with the same outcome at the top of every hour: error surfaced, no fallback step recorded.

Our hypothesis (without runtime trace logs) is that one of the early guards in classifyEmbeddedPiRunResultForModelFallback short-circuits to null on this code path — most likely hasOutboundDeliveryEvidence(params.result) returning true based on side-channel evidence (e.g., a typing indicator or a non-content delivery event) even though no real visible reply landed. We can't pinpoint exactly which guard tripped without source-side logging.

Reproduction

Easily observed against any reasoning model that occasionally emits a hidden-reasoning-only response. Our crossing of the failure was:

  1. Configure an agent with a reasoning model as model.primary and a non-reasoning model as the first entry of model.fallbacks.
  2. Send a prompt that the primary handles by producing reasoning content but no visible content (which the gateway then classifies internally as "reasoning-only" and retries with nextReasoningOnlyRetryInstruction).
  3. Continue until both retry attempts also return reasoning-only.
  4. Expected: After reasoningOnlyRetriesExhausted fires, the run result is classified as reasoning_only_result, runWithModelFallback advances to the next provider/model in model.fallbacks, and the agent's reply is produced by the fallback.
  5. Actual: runWithModelFallback surfaces the incomplete-turn error and exits without trying any fallback model. No model.fallback_step event is emitted.

Two independent providers exhibited this on our host:

  • pace-openrouter/google/gemini-2.5-flash (first observed 2026-05-04, sporadic)
  • openai-codex/gpt-5.4 via the built-in openai-codex auth provider (continuous from 2026-05-14 18:00 to 2026-05-15 17:24 CT — every hourly agent turn for ~24h)

In both cases, the agent (Pace) was effectively unreachable until we hand-swapped model.primary to a non-reasoning provider. The fallback chain in model.fallbacks was configured the whole time and never engaged.

Affected Source

Grep across ~/.npm-global/lib/node_modules/openclaw/dist/:

  • dist/pi-embedded-Bcz04p2i.js ~line 3159 — increments reasoningOnlyRetryAttempts, retries with nextReasoningOnlyRetryInstruction.
  • Same file ~line 3165 — sets reasoningOnlyRetriesExhausted = true when reasoningOnlyRetryAttempts >= maxReasoningOnlyRetryAttempts.
  • Same file ~line 3223 — on reasoningOnlyRetriesExhausted && !finalAssistantVisibleText, returns a result with a single {isError: true, text: STRICT_AGENTIC_BLOCKED_TEXT} payload, marking the attempt terminal.
  • dist/result-fallback-classifier-CrVa7J1V.js line 33 (classifyEmbeddedPiRunResultForModelFallback):
    • Line 35-39: early returns null if params.result.meta.aborted, hasDirectlySentBlockReply, hasBlockReplyPipelineOutput, or hasVisibleAgentPayload(..., includeReasoningPayloads:false).
    • Line 39: also early returns null if hasOutboundDeliveryEvidence(params.result) is true.
    • Line 40-45: harness classification path returns reasoning_only_result / empty_result correctly when reached.
  • dist/agent-command-CYAAR35V.js line 711 — wires the above classifier into runWithModelFallback({ classifyResult: ... }).

The wiring is correct. The behavior we observe is consistent with one of the guards at lines 35-39 returning truthy when it shouldn't, given a reasoning-only-exhausted run.

Sample Log Lines

From our ~/.openclaw/logs/gateway.err.log:

2026-05-15T16:34:39 [agent/embedded] reasoning-only assistant turn detected: runId=b1a75342... provider=openai-codex/gpt-5.4 — retrying 1/2 with visible-answer continuation
2026-05-15T16:34:43 [agent/embedded] reasoning-only assistant turn detected: runId=b1a75342... provider=openai-codex/gpt-5.4 — retrying 2/2 with visible-answer continuation
2026-05-15T16:34:46 [agent/embedded] reasoning-only retries exhausted: runId=b1a75342... provider=openai-codex/gpt-5.4 attempts=2/2 — surfacing incomplete-turn error

Same shape, every hour, for ~24 hours. No model.fallback_step event in the transcript or trajectory recorder for any of these runs.

What Helpful Trace Output Would Look Like

To self-diagnose without source modifications, it would help if the gateway emitted (at debug level) the actual return value of classifyEmbeddedPiRunResultForModelFallback per fallback evaluation, including which guard short-circuited it to null. Right now the run just ends and there's no way to tell why the chain didn't advance.

Ideally:

[agent/embedded] fallback-classifier: provider=A/M result=null reason=hasOutboundDeliveryEvidence
[agent/embedded] fallback-classifier: provider=A/M result=reasoning_only_result -> attempting fallback A2/M2

Proposed Fix Direction

We don't know enough about your guard semantics to propose a one-line patch, but a couple of directions:

  1. Audit the line 35-39 guards in classifyEmbeddedPiRunResultForModelFallback for whether they fire spuriously when the only payload in the result is the synthetic STRICT_AGENTIC_BLOCKED_TEXT error payload returned from the exhausted-retries branch in pi-embedded. In particular, hasOutboundDeliveryEvidence and hasDirectlySentBlockReply are good first suspects — both are concepts that an exhausted-retries run could plausibly trip without actually having delivered a real reply.

  2. Or, on the call-site side in pi-embedded-Bcz04p2i.js ~line 3223, when returning the exhausted-retries result, ensure meta.agentHarnessResultClassification is explicitly set to "reasoning-only" and that no spurious outbound-evidence / block-reply flags are set on the result. The classifier's harness-classification path (line 14-31) would then catch this case unambiguously and return reasoning_only_result, which runWithModelFallback should honor.

  3. Add the debug trace described above so operators can self-diagnose future fallback-not-firing situations.

Our Workaround

  1. Hand-swap model.primary in openclaw.json to a non-reasoning provider, kickstart the gateway. (We're currently running Pace on pace-openrouter/anthropic/claude-sonnet-4-6 with pace-openrouter/google/gemini-2.5-pro as the fallback.)
  2. Added a userspace monitor that tails gateway.err.log for reasoning-only retries exhausted / incomplete turn detected.*payloads=0 / context overflow detected.*attempt [2-9], extracts the provider, and posts a 60-min-cooldown alert to a designated Discord channel. We were running blind on this failure mode for ~24 hours before noticing via downstream symptoms; the monitor surfaces it within minutes now.

We'd switch back to relying on model.fallbacks as soon as the fallback path engages on reasoning-only / empty-visible failures.

Cross-Reference

  • KI-2026-05-15-01 in our local known-issues registry (~/.openclaw/knowledge-base/processes/known-issues.md).
  • Related to the same architectural constraint as KI-2026-05-14-02 (minified npm dist; userspace workarounds are durable but the upstream patch is the real cure).

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions