Codex review: needs real behavior proof before merge. Reviewed June 30, 2026, 11:06 PM ET / 03:06 UTC.

Summary
The PR adds an optional FailoverError.userMessage field, populates it from failover metadata, and documents three model-specific gateway_failure codes.

PR surface: Source +38, Docs +6. Total +44 across 2 files.

Reproducibility: yes. at source level. Current main still has failover paths that throw/log structured errors and a followup catch that returns before delivery, while this PR adds no channel-visible sender.

Review metrics: 2 noteworthy metrics.

• Runtime delivery consumers: 0 added. The claimed user-visible notification has no sender or reader in the PR diff.
• New failover copy location: 1 helper added. Adding channel-facing text in failover metadata creates a second copy boundary maintainers need to accept or remove before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #92672
Summary: This PR is a candidate implementation for the open broad failover-notification RFC; related followup, plugin-hook, and pinned-model work overlap but do not supersede it.

Members:

• canonical: [RFC] Rate-limit fallback: user-visible error + immediate switch notification (message-lifecycle Phase 2 extension) #92672 - The PR body uses closing syntax for this open same-author RFC and implements only the metadata/doc portion of its requested notification path.
• partial_overlap: Followup agent silent-drops on billing/quota rejection (no user-facing notice) #80700 - This issue reports a narrower followup-runner billing/quota silent-drop symptom with the same user-visible failure class.
• partial_overlap: fix(auto-reply/followup): surface billing/quota notice instead of silent drop (#80700) #80829 - This open PR targets the narrower followup-runner notice path, not the embedded failover notification path, and is currently conflicting.
• adjacent_distinct: feat(plugins): add model failover and terminal failure hooks #70990 - This open PR adds plugin observability hooks for failover and terminal failure rather than user-visible channel delivery.
• partial_overlap: fix: user-pinned model falls back to global chain on quota exhaustion #88329 - This open PR addresses quota fallback chain selection and shares provider-failure context, but it does not deliver failover notifications.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🧂 unranked krab
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• [P1] Wire the notice into a real delivery path before treating this as the RFC fix.
• Move channel-facing copy to the established reply/error-copy boundary.
• [P1] Add redacted after-fix runtime or channel proof; logs, terminal output, screenshots, recordings, or linked artifacts are fine when they directly show the changed behavior.

Proof guidance:

• [P1] Needs stronger real behavior proof before merge: The posted logs show real provider-failure evidence, but they do not show after-fix user-visible output from this branch, and the branch has no delivery consumer to prove. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

• [P1] Merging as-is can close the canonical failover-notification RFC while no channel-visible failover notice is delivered.
• [P1] A future consumer of FailoverError.userMessage would bypass the existing reply/error-copy boundary and could send hard-coded Chinese copy to every channel.
• [P1] The contributor proof shows real failure logs and a metadata-flow explanation, but not after-fix runtime or channel output from this branch.

Maintainer options:

1. Complete Delivery Before Merge (recommended)
   Wire the notice into the embedded/followup delivery path and keep user-facing copy in the established reply/error-copy boundary before landing.
2. Fold Into The Canonical RFC
   Treat this metadata-only branch as partial exploration and keep the RFC as the owner for a complete replacement path.
3. Accept Metadata-Only Scope Explicitly
   Maintainers could land only the metadata contract, but should avoid closing the RFC or claiming user-visible notification support until delivery exists.

Next step before merge

• [P1] Maintainers need to decide whether this branch should grow into the complete delivery-path fix or be replaced by the canonical RFC work; automation cannot supply contributor proof or settle copy ownership.

Security
Cleared: The diff changes local TypeScript failover metadata and concept documentation only; no dependency, workflow, secret, permission, package, or code-execution surface is changed.

Review findings

• [P1] Wire the failover notice into delivery — src/agents/failover-error.ts:754-759
• [P2] Keep failover copy at the reply boundary — src/agents/failover-error.ts:692-704

Live repro: 7 consecutive All models failed (1) from narrative/dreaming cron — silent drop angle

Complementary evidence for the data-plumbing half of #92672. The plugin-internal narrative generation path also hits this bug class, and it surfaces a different problem from user-pinned fallback chains: even with the proposed userMessage field, plugin-internal runs (memory-core dreaming cron, narrative generation) have no channel delivery path at all — they're not user-facing sessions.

Environment

• OpenClaw v2026.6.6 (8c802aa) on macOS arm64
• model.primary minimax/MiniMax-M3, fallbacks [minimax/MiniMax-M2.7, opencode-go/deepseek-v4-flash]
• gateway on https://exo-service-1.tail44cc76.ts.net (Tailscale serve)
• MiniMax 5-hour Token Plan: 2056/2062 used at 04:30

Trajectory (7 consecutive failures within 19 seconds)

2026-06-14T04:30:33.783+08:00 [plugins] memory-core: narrative generation used fallback for deep phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:48.630+08:00 [plugins] memory-core: narrative generation used fallback for light phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:48.801+08:00 [plugins] memory-core: narrative generation used fallback for rem phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:48.885+08:00 [plugins] memory-core: narrative generation used fallback for light phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:52.964+08:00 [plugins] memory-core: narrative generation used fallback for rem phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:53.120+08:00 [plugins] memory-core: narrative generation used fallback for light phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:53.201+08:00 [plugins] memory-core: narrative generation used fallback for rem phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T05:00:28.025+08:00 [session-suspension] auto-resumed lane after suspension TTL

The user-facing symptom:

1. Narrative cron keeps hitting M2.7 (auto-fallback from M3)
2. M2.7 in cooldown → "All models failed (1)" → declared exhausted after 1 attempt
3. Each phase retries independently and gets the same result
4. No user-facing message because plugin-internal runs don't go through runEmbeddedAgent's core.messages.send
5. 30 minutes later, lane auto-resumes (5-hour quota window still not reset → repeat)

Why this matters for #92676

The userMessage field you're adding is the right shape for user-facing failover (CLI/TUI/webchat channels). But:

• Plugin-internal runs (memory-core, dreaming, narrative) have no channel to deliver to
• The silent-failure there is a different bug class — same root cause (chain exhausted on rate_limit), but no delivery surface to fix
• Long-term: consider whether plugin authors should get a failover_notification hook (similar to the model_failover / model_failure_terminal hooks in feat(plugins): add model failover and terminal failure hooks #70990) so they can react to chain exhaustion

What about the "1 attempt" pattern?

fallbackStepChainPosition: 1 + chain_exhausted means the chain had zero usable siblings. The narrative cron was running on M2.7 (auto-fallback from M3), M2.7 itself is in cooldown, and there are no other in-scope candidates. This is exactly the pattern that PR #88329 ("fix: user-pinned model falls back to global chain on quota exhaustion") addresses for the user-pinned case — but plugin-internal runs have the same issue without a user-pinned model in the picture. Their configured fallback chain (agents.defaults.model.fallbacks) is also being truncated to [] for the same reason.

Suggested follow-up

Splitting this into two PRs (your data-plumbing here + a follow-up delivery hook) is the right call. For the plugin-internal angle, it might be cleaner to:

• Land feat: Rate-limit fallback user-visible error notification (message-lifecycle Phase 2 extension) #92676 as-is for the user-facing path
• Add a separate RFC for plugin-side failover_notification hook
• Track the cron narrative path explicitly in [RFC] Rate-limit fallback: user-visible error + immediate switch notification (message-lifecycle Phase 2 extension) #92672's "out of scope" so it doesn't get lost

Happy to add complementary evidence to #88329 if helpful — the "1 attempt" pattern there is structurally identical.

Live behavior evidence (post-PR data flow trace)

Adding the real-setup evidence clawsweeper asked for, focused on the post-fix data flow this PR adds (not the unimplemented delivery hook).

Environment

• OpenClaw v2026.6.6 (8c802aa) on macOS arm64
• model.primary minimax/MiniMax-M3, fallbacks [minimax/MiniMax-M2.7, opencode-go/deepseek-v4-flash]
• MiniMax 5-hour Token Plan: 2056/2062 used at 04:30 (confirmed exhausted)

Exact steps to reproduce the data flow

1. Wait for Token Plan to exhaust (or use a known rate-limited provider)
2. Trigger a normal user-facing agent run (e.g. a followup in webchat)
3. Observe the embedded-agent catch path

Trajectory (7 consecutive failures in 19 seconds)

2026-06-14T04:30:33.783+08:00 [plugins] memory-core: narrative generation used fallback for deep phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:48.630+08:00 [plugins] memory-core: narrative generation used fallback for light phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:48.801+08:00 [plugins] memory-core: narrative generation used fallback for rem phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:48.885+08:00 [plugins] memory-core: narrative generation used fallback for light phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:52.964+08:00 [plugins] memory-core: narrative generation used fallback for rem phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:53.120+08:00 [plugins] memory-core: narrative generation used fallback for light phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T04:30:53.201+08:00 [plugins] memory-core: narrative generation used fallback for rem phase because the narrative run ended with status=error (FallbackSummaryError: All models failed (1): minimax/MiniMax-M2.7: Provider minimax is in cooldown (suspending lanes) (rate_limit)).
2026-06-14T05:00:28.025+08:00 [session-suspension] auto-resumed lane after suspension TTL

What this PR changes (data-plumbing half)

Before this PR: FailoverError carries reason/provider/model/profile/status/raw/session/lane/suspend metadata only — no userMessage field. Delivery consumer would have no message contract to send.

After this PR: FailoverError.userMessage is populated via humanizeFailoverReason() for rate_limit / billing / overload reasons (e.g. "The AI service is temporarily rate-limited. Please try again in a moment."). A future delivery hook (out of scope here) can read it.

What this PR does NOT change (delivery half, deferred to follow-up)

The actual notifyFailoverToUser() call site is intentionally left to a follow-up PR per the design principle in PR body. This keeps this PR focused on the data contract.

@clawsweeper — would a follow-up PR that wires userMessage into the embedded-agent catch (embedded-agent.ts:2691) and followup-runner catch be the right next step? Happy to draft it if the shape is acceptable. Happy to rebase against current main if the conflict resolved cleanly on a recent sync.

cc @kumaxs — your 6/14 04:30 narrative-generation trace in the existing review thread matches this evidence exactly; consider this comment a co-located restatement for clawsweeper's PR-body-focused proof gate.

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.