feat(plugins): add model failover and terminal failure hooks by mantisai-bot · Pull Request #70990 · openclaw/openclaw

mantisai-bot · 2026-04-24T07:13:30Z

Summary

Adds two plugin observability hooks for model failure handling:

model_failover: observes embedded-runner failover decisions, including failed source refs, selected target refs, stage, decision, reason metadata, recoverability, timeout/abort/status flags, and fallback availability.
model_failure_terminal: observes terminal before-reply model failures after all configured attempts are exhausted.

This PR does not change fallback policy, provider retry behavior, context re-injection behavior, command/tool execution, network access, or default user-facing failure copy.

Current implementation

Emits model_failover from the embedded-runner failover decision path.
Propagates the selected fallback candidate through direct agent-command runs, Gateway auto-reply runs, and queued follow-up runs before running the embedded agent.
Emits model_failure_terminal after resolving the actual delivered failure reply, so finalMessage matches user-visible failure copy.
Registers both hook names and payload types in the Plugin SDK surface and refreshes the SDK API baseline hash.
Bounds both failure-path void hooks with default 30s timeout budgets and fail-open behavior.

Security / compatibility

New public Plugin SDK hook contract: model_failover, model_failure_terminal.
model_failure_terminal is conversation-access gated for non-bundled plugins because it can expose delivered failure copy.
Attempt summaries expose provider/model/reason metadata, not raw provider error blobs.
Maintainers still need to accept the public hook contract and terminal-failure data boundary before release.

Real behavior proof

Behavior addressed: A real embedded OpenClaw model run should emit model_failover when the primary model fails and the runtime selects the configured fallback target. A separate exhausted-model run should emit model_failure_terminal after all configured model attempts fail, with finalMessage matching the user-visible failure copy and attempts summarizing provider/model/reason metadata.
Environment tested: Local OpenClaw source checkout on macOS Darwin 25.3.0 arm64, Node v25.6.0, pnpm 11.2.2, PR head 5cd9b674c3b9.
Exact steps or command run after this patch: The proof runner created a temporary workspace, temporary state dir, and a real workspace plugin with openclaw.plugin.json. OpenClaw loaded that plugin through the real plugin loader; the plugin registered api.on("model_failover", ...) and api.on("model_failure_terminal", ...) and wrote hook payloads to a redacted JSONL artifact. Model providers were deterministic local OpenAI-compatible Responses endpoints. Scenario 1 returned HTTP 429 from the primary endpoint and a successful SSE canary from fallback. Scenario 2 returned HTTP 429 from both primary and fallback to exhaust all configured model attempts. No external provider token was used.

OPENCLAW_MODEL_FAILOVER_PROOF_HEAD=$(git rev-parse --short=12 HEAD) node --import tsx .artifacts/model-failover-real-proof.mjs

Evidence after fix: The proof command produced copied live terminal output and a redacted JSONL artifact from the runtime-loaded plugin. The runtime output confirmed model_failover fired after proof-primary/proof-primary-model returned HTTP 429, selected proof-fallback/proof-fallback-model, and then delivered PROOF_FALLBACK_CANARY_OK. The same proof run confirmed the exhausted-model scenario emitted model_failure_terminal after both configured attempts failed.
Observed result after fix:
- model_failover reported selected target proof-fallback/proof-fallback-model, failed source proof-primary/proof-primary-model, decision: fallback_model, failoverReason: rate_limit, status: 429, and sourceRecoverable: true.
- The fallback endpoint was hit and the run delivered PROOF_FALLBACK_CANARY_OK.
- The terminal run hit both proof-terminal-primary/proof-terminal-primary-model and proof-terminal-fallback/proof-terminal-fallback-model, exhausted all attempts, and emitted model_failure_terminal with kind: all_models_failed.
- model_failure_terminal.finalMessage was exactly the delivered terminal reply: ⚠️ All models are temporarily rate-limited. Please try again in a few minutes.
- Terminal attempts contained both failed provider/model summaries with rate_limit reasons.
Not tested: Live third-party provider outages were not tested; provider failures were deterministic local HTTP endpoints. Secret-bearing headers were redacted in proof output.

Regression coverage for the latest review findings

New source tests cover the ClawSweeper findings directly:

src/auto-reply/reply/agent-runner-execution.test.ts: passes the selected next fallback candidate into embedded auto-reply runs
src/auto-reply/reply/agent-runner-execution.test.ts: emits model_failure_terminal with the delivered terminal failure copy
src/auto-reply/reply/followup-runner.test.ts: passes selected next fallback candidates to embedded follow-up runs

The unrelated Anthropic config-default change is no longer part of this PR diff.

Validation

Commands run on head 5cd9b674c3b9:

node scripts/run-vitest.mjs run src/auto-reply/reply/followup-runner.test.ts src/auto-reply/reply/agent-runner-execution.test.ts src/plugins/hooks.model-failover.test.ts src/agents/model-fallback.test.ts src/agents/embedded-agent-runner/run/failover-observation.model-failover-hook.test.ts
pnpm plugin-sdk:api:check
pnpm config:docs:check
pnpm tsgo:test:src
pnpm check:architecture
git diff --check origin/main...HEAD && git diff --check
git merge-tree --write-tree HEAD origin/main
npx oxfmt --check src/auto-reply/reply/agent-runner-execution.ts src/auto-reply/reply/agent-runner-execution.test.ts src/plugins/hooks.model-failover.test.ts src/agents/model-fallback.test.ts src/agents/embedded-agent-runner/run/failover-observation.model-failover-hook.test.ts
OPENCLAW_MODEL_FAILOVER_PROOF_HEAD=$(git rev-parse --short=12 HEAD) node --import tsx .artifacts/model-failover-real-proof.mjs

Observed results:

Focused Vitest passed 3 shards / 363 tests.
Plugin SDK API baseline check passed.
Config docs baseline check passed.
tsgo:test:src passed.
pnpm check:architecture passed.
git diff --check origin/main...HEAD && git diff --check passed.
git merge-tree --write-tree HEAD origin/main passed.
Focused oxfmt --check passed.
Exact-head real behavior proof passed on 5cd9b674c3b9, with the runtime-loaded plugin observing model_failover, selected fallback target proof-fallback/proof-fallback-model, delivered fallback canary PROOF_FALLBACK_CANARY_OK, and terminal model_failure_terminal.finalMessage matching the delivered all-models-failed reply.

Latest rebase/proof refresh (2026-06-16)

Current head: 566a985e2773e7890782801386191837ee102f90

Merged current origin/main (395a0823484f) into the PR branch. Conflict resolution kept both current-main lifecycle terminal error emission and this PR's model_failure_terminal plugin hook emission. Queued follow-up fallback execution now keeps both exposeNextCandidateToRun: true and current-main exhausted-result merging. The Plugin SDK API baseline hash was regenerated from the merged tree. A follow-up commit also bumps the plugin SDK public export budget from 10,270 to 10,274 for the intentional hook surface expansion, fixing the post-push checks-node-core-fast failure in test/scripts/plugin-sdk-surface-report.test.ts.

Validation on the current head:

node scripts/run-vitest.mjs run src/auto-reply/reply/agent-runner-execution.test.ts src/auto-reply/reply/followup-runner.test.ts src/plugins/hooks.model-failover.test.ts src/agents/model-fallback.test.ts src/agents/embedded-agent-runner/run/failover-observation.model-failover-hook.test.ts
node scripts/run-vitest.mjs run test/scripts/plugin-sdk-surface-report.test.ts --reporter=verbose
node --max-old-space-size=8192 --import tsx scripts/generate-plugin-sdk-api-baseline.ts --check
OPENCLAW_PLUGIN_SDK_MAX_PUBLIC_DEPRECATED_EXPORTS_BY_ENTRYPOINT='{"core":2}' node scripts/plugin-sdk-surface-report.mjs --check
git diff --check origin/main...HEAD && git diff --check
npx oxfmt --check src/auto-reply/reply/agent-runner-execution.ts src/auto-reply/reply/followup-runner.ts docs/.generated/plugin-sdk-api-baseline.sha256 scripts/plugin-sdk-surface-report.mjs
OPENCLAW_MODEL_FAILOVER_PROOF_HEAD=$(git rev-parse --short=12 HEAD) node --import tsx .artifacts/model-failover-real-proof.mjs

Observed results:

Focused model-failover Vitest passed 3 shards / 387 tests.
Plugin SDK surface report test passed 1 shard / 4 tests.
Plugin SDK API baseline check passed.
Plugin SDK surface report --check passed with the exact deprecated-export override used by the failing CI test.
git diff --check origin/main...HEAD && git diff --check passed.
Focused oxfmt --check passed.
Exact-head real behavior proof passed on 566a985e2773, with the runtime-loaded plugin observing model_failover, selected fallback target proof-fallback/proof-fallback-model, delivered fallback canary PROOF_FALLBACK_CANARY_OK, and terminal model_failure_terminal.finalMessage matching the delivered all-models-failed reply.

Notes:

pnpm plugin-sdk:api:gen got stuck in pnpm's dependency-status install wrapper on this host, so the same generator was run directly with node --max-old-space-size=8192 --import tsx scripts/generate-plugin-sdk-api-baseline.ts --write.
I did not rerun pnpm config:docs:check, pnpm tsgo:test:src, or pnpm check:architecture on this refresh.

Latest head refresh (2026-06-22)

Current head: 6a1b11fb758bb1c7ed0f45a9e840ef54e05ff8fc

Merged current origin/main (3ff59df960) into the PR branch, resolved the Plugin SDK surface-budget conflict by keeping this PR's intentional higher hook-surface budgets, and aligned the slash-command browser-import expectation with the current shared-registry import order.

Validation on the current head:

OPENCLAW_MODEL_FAILOVER_PROOF_HEAD=$(git rev-parse --short=12 HEAD) node --import tsx .artifacts/model-failover-real-proof.mjs
node scripts/run-vitest.mjs run src/auto-reply/reply/followup-runner.test.ts src/auto-reply/reply/agent-runner-execution.test.ts src/plugins/hooks.model-failover.test.ts src/agents/model-fallback.test.ts src/agents/embedded-agent-runner/run/failover-observation.model-failover-hook.test.ts
pnpm exec vitest run ui/src/ui/chat/slash-commands.browser-import.test.ts test/scripts/plugin-sdk-surface-report.test.ts

Observed results:

Exact-head real behavior proof passed on 6a1b11fb758b: the runtime-loaded proof plugin observed model_failover, the local primary provider returned HTTP 429, the fallback provider delivered PROOF_FALLBACK_CANARY_OK, and the exhausted terminal run emitted model_failure_terminal with finalMessage matching the delivered all-models-failed reply.
Focused model-failover Vitest passed 3 shards / 396 tests.
Focused slash-command import + Plugin SDK surface tests passed 2 files / 7 tests.

Limitations: provider failures were deterministic local HTTP endpoints, not real third-party outages; secret-bearing headers were redacted in proof output. I did not rerun full repository CI locally on this refresh.

Latest head refresh (2026-06-30)

Current head: 0ffca7e11f6f

Merged current origin/main (738b2be4b4) into the PR branch. Conflict resolution kept this PR's model_failover / model_failure_terminal hook API additions, accepted current-main Codex app-inventory/cache-key and channel-delivery fixes, regenerated config and Plugin SDK API baselines, raised the Plugin SDK public export budget to 10404 for the intentional hook surface, and removed a duplicate normalizeFastMode import introduced by the merge.

Validation on the current head:

node --import tsx scripts/generate-config-doc-baseline.ts --check
node --max-old-space-size=8192 --import tsx scripts/generate-plugin-sdk-api-baseline.ts --check
node scripts/plugin-sdk-surface-report.mjs --check
node scripts/run-vitest.mjs run src/auto-reply/reply/followup-runner.test.ts src/auto-reply/reply/agent-runner-execution.test.ts src/plugins/hooks.model-failover.test.ts src/agents/model-fallback.test.ts src/agents/embedded-agent-runner/run/failover-observation.model-failover-hook.test.ts
git diff --check
pnpm exec oxfmt --check scripts/plugin-sdk-surface-report.mjs extensions/codex/src/app-server/app-inventory-cache.test.ts extensions/codex/src/app-server/plugin-app-cache-key.ts extensions/msteams/src/monitor-handler.ts extensions/telegram/src/lane-delivery-text-deliverer.ts test/scripts/check-workflows.test.ts src/auto-reply/reply/agent-runner-execution.ts src/auto-reply/reply/followup-runner.ts src/plugins/hook-types.ts src/agents/tools/embedded-gateway-stub.ts
OPENCLAW_MODEL_FAILOVER_PROOF_HEAD=$(git rev-parse --short=12 HEAD) node --import tsx .artifacts/model-failover-real-proof.mjs

Observed results:

Config baseline check passed.
Plugin SDK API baseline check passed.
Plugin SDK surface report passed with public package totals: 322 entrypoints, 10,404 exports, 5,219 callable exports.
Focused model-failover Vitest passed 3 shards / 415 tests.
git diff --check passed.
Focused oxfmt --check passed.
Exact-head source-built proof passed on 0ffca7e11f6f: the runtime-loaded proof plugin observed model_failover, the local primary provider returned HTTP 429, the fallback provider delivered PROOF_FALLBACK_CANARY_OK, and the exhausted terminal run emitted model_failure_terminal with finalMessage matching the delivered all-models-failed reply.

Limitations: provider failures were deterministic local HTTP endpoints, not real third-party outages; secret-bearing headers were redacted in proof output. I did not rerun full repository CI locally on this refresh.

greptile-apps · 2026-04-24T07:15:48Z

Greptile Summary

Adds two fire-and-forget plugin hooks — model_failover and model_failure_terminal — wired at the failover-decision and terminal before-reply failure seams respectively. The plumbing follows the existing runVoidHook/fail-open pattern and the new event types are correctly registered in PLUGIN_HOOK_NAMES and PluginHookHandlerMap.

Confidence Score: 5/5

Safe to merge — both hooks are fire-and-forget with fail-open semantics and no changes to existing failover control flow.

All remaining findings are P2: a minor duplicate type-guard call and an optional field in the attempts type that is never populated. Neither affects correctness or runtime behavior.

No files require special attention.

Comments Outside Diff (1)

src/auto-reply/reply/agent-runner-execution.ts, line 1530-1564 (link)

Duplicate isFallbackSummaryError call

isFallbackSummaryCheck is computed inside the if (terminalHookRunner?.hasHooks(...)) block, but the pre-existing isFallbackSummary at line 1564 calls isFallbackSummaryError(err) again unconditionally. Hoisting the check above the if block and reusing a single binding below would remove the redundancy and make it clear both branches are looking at the same guard.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/auto-reply/reply/agent-runner-execution.ts
Line: 1530-1564

Comment:
**Duplicate `isFallbackSummaryError` call**

`isFallbackSummaryCheck` is computed inside the `if (terminalHookRunner?.hasHooks(...))` block, but the pre-existing `isFallbackSummary` at line 1564 calls `isFallbackSummaryError(err)` again unconditionally. Hoisting the check above the `if` block and reusing a single binding below would remove the redundancy and make it clear both branches are looking at the same guard.



How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: src/auto-reply/reply/agent-runner-execution.ts
Line: 1530-1564

Comment:
**Duplicate `isFallbackSummaryError` call**

`isFallbackSummaryCheck` is computed inside the `if (terminalHookRunner?.hasHooks(...))` block, but the pre-existing `isFallbackSummary` at line 1564 calls `isFallbackSummaryError(err)` again unconditionally. Hoisting the check above the `if` block and reusing a single binding below would remove the redundancy and make it clear both branches are looking at the same guard.

```suggestion
      const isFallbackSummary = isFallbackSummaryError(err);
      const terminalHookRunner = getGlobalHookRunner();
      if (terminalHookRunner?.hasHooks("model_failure_terminal")) {
        const isFallbackSummaryCheck = isFallbackSummary;
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/plugins/hook-types.ts
Line: 237-245

Comment:
**`stage` field declared but never populated**

`PluginHookModelFailureTerminalEvent.attempts` includes an optional `stage?: "prompt" | "assistant"` field, but the emission site in `agent-runner-execution.ts` maps each attempt to only `{ provider, model, reason }` — `stage` is never set. If no current caller can provide it, omitting the field from the type keeps the surface minimal and avoids a permanently-undefined slot that could mislead plugin authors.

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "feat(plugins): add model_failover and mo..." | Re-trigger Greptile}

clawsweeper · 2026-04-27T00:02:27Z

Codex review: needs changes before merge. Reviewed July 3, 2026, 3:05 AM ET / 07:05 UTC.

Summary
The PR adds model_failover and model_failure_terminal Plugin SDK hooks, wires them into embedded failover and terminal failure paths, updates docs/generated SDK baselines, and adds focused hook/fallback tests.

Reproducibility: yes. Source inspection reproduces the review blockers: returned outcome: "exhausted" results are handled as failures without the new hook, and the PR head contains an unused import that corresponds to failing type/lint checks.

Review metrics: 3 noteworthy metrics.

Public Plugin SDK hooks: 2 added. New hook names and payload contracts become third-party plugin API once released.
Conversation-access hook surface: 1 hook added. model_failure_terminal is added to the gated conversation hook set because it can expose delivered failure copy and terminal provider metadata.
Exact-head failing checks: 2 failing. check-prod-types and check-lint are failing on the reviewed head, so green CI does not currently gate the merge.

Stored data model
Persistent data-model change detected: unknown-truncated-pull-files, vector/embedding metadata: src/secrets/target-registry-data.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #70990
Summary: This PR is the canonical open implementation for the narrow Plugin SDK failover/terminal-failure hook gap; broader fallback notification and context re-injection work remains separate.

Members:

fixed_by_candidate: [Feature]: Emit first-class plugin hooks for model failover and terminal all-models-failed outcomes #70976 - This closed issue requested the same two hook surfaces and was explicitly closed as superseded by this PR.
partial_overlap: [Feature] User notification + workspace context re-injection on model fallback #79163 - This open issue is the broader tracker for fallback notification, context re-injection, and hook exposure; this PR only addresses the hook slice.
partial_overlap: [Meta] Feature request bundle: 11 platform gaps from intensive daily use #65824 - The meta issue includes the original silent-fallback hook gap among several broader platform requests.
adjacent_distinct: feat(runtime): report model fallback transitions #98954 - That open PR reports fallback transitions to a configured diagnostics endpoint, while this PR adds first-class Plugin SDK hooks and terminal failure events.
adjacent_distinct: feat: Rate-limit fallback user-visible error notification (message-lifecycle Phase 2 extension) #92676 - That open PR is about user-visible rate-limit fallback notification plumbing, not the plugin hook contract implemented here.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Emit model_failure_terminal for returned exhausted fallback results in both auto-reply and follow-up paths with focused regressions.
Remove the unused import and verify exact-head type/lint checks are clean.
Get maintainer acceptance or requested edits for the public hook contract and terminal-failure data boundary.

Risk before merge

[P1] The PR adds two public Plugin SDK hook names and payload contracts; once released, hook timing and fields become third-party API surface.
[P1] model_failure_terminal exposes delivered failure copy and provider/model/reason attempt summaries to conversation-access plugins, so maintainers need to accept that data boundary before release.
[P2] Returned exhausted fallback results can still produce lifecycle/reply failure without model_failure_terminal, so plugin authors would see incomplete terminal-failure coverage.
[P1] Exact-head check-prod-types and check-lint are failing on the PR head due to an unused import.

Maintainer options:

Repair coverage and clear CI before API approval (recommended)
Add terminal-hook emission for returned exhausted auto-reply and follow-up results, remove the unused import, then have maintainers explicitly accept or revise the public hook names, timing, payloads, and data boundary.
Accept the new trusted-plugin data boundary
Maintainers may decide the conversation-access gate is sufficient for delivered failure copy and attempt summaries, but that acceptance should be visible before release.
Pause for a different fallback API shape
If maintainers want fallback transition reporting through diagnostics, notification policy, or a different hook name/payload, pause this branch and keep the broader decision on the fallback tracker.

Next step before merge

[P2] Two code blockers are narrow automation-repair candidates; maintainer approval of the public API/data boundary remains a separate merge decision after repair.

Security
Needs attention: The diff has no supply-chain change, but it adds a security-relevant trusted-plugin data boundary that maintainers should explicitly approve before release.

Review findings

[P2] Cover returned-exhausted failures with this hook — src/auto-reply/reply/agent-runner-execution.ts:3343-3344
[P2] Remove the unused sandbox policy import — src/security/audit-extra.summary.ts:7

Review details

Best possible solution:

Land a maintainer-approved Plugin SDK hook contract only after the terminal hook covers both thrown and returned exhausted failures, CI is clean, and broader fallback notification/context policy remains on the existing fallback tracker.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection reproduces the review blockers: returned outcome: "exhausted" results are handled as failures without the new hook, and the PR head contains an unused import that corresponds to failing type/lint checks.

Is this the best way to solve the issue?

No, not yet. The hook surface is a plausible owner-boundary solution, but the implementation should share terminal-hook emission across thrown and returned exhausted failures and get maintainer signoff on the public API/data boundary.

Full review comments:

[P2] Cover returned-exhausted failures with this hook — src/auto-reply/reply/agent-runner-execution.ts:3343-3344
runWithModelFallback can return outcome: "exhausted" with a result instead of throwing. This PR emits model_failure_terminal only in the catch path, so the returned-exhausted auto-reply and follow-up branches still mark failure without notifying plugins; add shared terminal-hook emission for those branches too.
Confidence: 0.92
[P2] Remove the unused sandbox policy import — src/security/audit-extra.summary.ts:7
PR head imports pickSandboxToolPolicy but never uses it, and exact-head check-prod-types plus check-lint are failing. Drop the dead import so the branch can pass the required checks.
Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 8ed6c78b7891.

Label changes

Label justifications:

P2: This is a bounded Plugin SDK observability feature with useful proof and concrete repair blockers, not an urgent production regression.
merge-risk: 🚨 compatibility: The PR adds public Plugin SDK hook names, payload fields, generated baselines, and export-budget growth that become compatibility surface after release.
merge-risk: 🚨 security-boundary: The terminal failure hook exposes delivered failure copy and provider/model attempt summaries to trusted conversation-access plugins.
rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body supplies exact-head source-built proof with a runtime-loaded workspace plugin observing both hooks against deterministic local Responses endpoints, including fallback success and terminal exhausted failure output.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies exact-head source-built proof with a runtime-loaded workspace plugin observing both hooks against deterministic local Responses endpoints, including fallback success and terminal exhausted failure output.

Evidence reviewed

Security concerns:

[medium] Approve terminal failure data exposure — src/plugins/hook-types.ts:443
model_failure_terminal exposes delivered terminal failure copy and provider/model/reason attempt summaries to plugins that pass the conversation-access gate; that may be acceptable, but it is a new trusted-plugin data boundary rather than a purely internal hook.
Confidence: 0.86

Acceptance criteria:

[P2] node scripts/run-vitest.mjs run src/auto-reply/reply/agent-runner-execution.test.ts src/auto-reply/reply/followup-runner.test.ts src/plugins/hooks.model-failover.test.ts src/agents/model-fallback.test.ts src/agents/embedded-agent-runner/run/failover-observation.model-failover-hook.test.ts.
[P1] node --max-old-space-size=8192 --import tsx scripts/generate-plugin-sdk-api-baseline.ts --check.
[P1] node scripts/plugin-sdk-surface-report.mjs --check.
[P1] pnpm tsgo:prod.
[P1] pnpm lint --threads=8.

What I checked:

Repository policy applied: Root and scoped AGENTS.md files were read; Plugin SDK additions, fallback behavior, and terminal failure data exposure are compatibility-sensitive review surfaces. (AGENTS.md:31, 8ed6c78b7891)
Current main does not implement the hook request: Current main has no model_failover or model_failure_terminal Plugin SDK hook entries; matching current-main hits are only existing Prometheus metric names, not plugin hooks. (src/plugins/hook-types.ts:417, 8ed6c78b7891)
PR head adds public hook contract: PR head adds PluginHookModelFailoverEvent and PluginHookModelFailureTerminalEvent; the terminal event includes user-facing finalMessage, kind, and provider/model/reason attempt summaries. (src/plugins/hook-types.ts:422, 0ffca7e11f6f)
PR head emits failover hooks from the failover logger: The PR emits model_failover from createFailoverDecisionLogger alongside the existing structured failover log, with selected target and failed source fields. (src/agents/embedded-agent-runner/run/failover-observation.ts:139, 0ffca7e11f6f)
Returned exhausted results bypass terminal hook emission: runWithModelFallback can return outcome: "exhausted" with attempts, while PR-head auto-reply and follow-up returned-exhausted branches mark lifecycle/reply failure without calling runModelFailureTerminal; the new hook call is only in the catch path. (src/auto-reply/reply/agent-runner-execution.ts:3056, 0ffca7e11f6f)
Follow-up runner has the same returned-exhausted gap: PR-head follow-up handling sets fallbackExhaustedFailure and fails the reply operation for returned exhausted results, but no terminal hook is emitted on that branch. (src/auto-reply/reply/followup-runner.ts:1308, 0ffca7e11f6f)

Likely related people:

steipete: Peter Steinberger introduced the original model fallback feature and later refactored model fallback conversion/error handling on the central fallback path. (role: feature introducer and recent fallback contributor; confidence: high; commits: 734bb6b4fd9a, dcc3392a1a40; files: src/agents/model-fallback.ts)
vincentkoc: Vincent Koc has recent history on lightweight model fallback helpers and the split hook contract types that define the public hook surface this PR expands. (role: recent fallback and hook-contract contributor; confidence: high; commits: 93ce76afe310, 7f5a5a34dbf8; files: src/agents/model-fallback.ts, src/plugins/hook-types.ts, src/plugins/hooks.ts)
Dallin Romney: Current-main blame for the returned-exhausted handling and plugin hook-type region points to the recent channel streaming evidence migration commit, so this person is useful routing context for the latest base shape. (role: recent current-main line owner; confidence: medium; commits: 8604dbdc93fb; files: src/agents/model-fallback.ts, src/auto-reply/reply/agent-runner-execution.ts, src/plugins/hook-types.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

smonett · 2026-05-07T12:00:57Z

This directly addresses the hook/event gap from #65824 item 1 — glad to see it moving.

A few notes from running our own local mitigation (canon-guardian, shared in #75750) that might be useful for the implementation:

On hook timing: @kinthaiofficial noted in #65824 that lifecycle:fallback_active fires too late — the new model has already started its first reply. The before_prompt_build hook fires before request construction, which is what let our plugin do a context re-injection that actually lands before the fallback model's first token. Flagging in case the model_failover hook placement hits the same timing question.

On what operators actually need in the event payload: From our production logs, the most actionable fields are: which model failed, which fallback was selected, whether it was a rate-limit/quota failure vs. a hard error (404/invalid config), and whether the primary is expected to recover. The last one is the difference between 'alert now' and 'just log it.'

Scope boundary looks right — keeping context re-injection out of this PR is the right call. That's a bigger design question that belongs in the #65824 thread.

Thanks for picking this up. Running 2026.5.3-1 and will test against the final version when it lands.

mantisai-bot · 2026-05-09T05:20:48Z

Quick follow-up after the recent interest here: I re-checked the current head ef986ebf8a3bba03f1295e3f15829bdf5fef5da6 against the operator needs called out above.

The current model_failover payload covers the actionable fields mentioned:

failed source model: sourceProvider / sourceModel
selected/current target: provider / model
rate-limit/quota vs harder failure signals: failoverReason, profileFailureReason, and optional status
whether the failed source is expected to be worth probing again later: sourceRecoverable

The timing boundary is intentionally observation-only: docs now call out that model_failover is fire-and-forget telemetry/alerting/operator-state, not a synchronous context re-injection hook before fallback generation. Keeping context re-injection out of this PR still looks like the right scope split for #65824.

I also re-ran the requested focused validation locally:

pnpm test src/plugins/hooks.model-failover.test.ts src/agents/pi-embedded-runner/run/failover-observation.model-failover-hook.test.ts
pnpm test src/plugins/loader.test.ts
pnpm check:changed

All passed, current checks are green, and merge state is CLEAN. I also resolved the stale/outdated Greptile thread about the old stage field on terminal attempts; that field is not present in the current public terminal-attempt payload.

mantisai-bot · 2026-06-10T16:51:02Z

@clawsweeper re-review

Rebased onto current origin/main (2e4bf410a2f0) and refreshed exact-head proof in the PR body for head 5cd9b674c3b9. Merge conflict resolution kept both the current lifecycle error event and the new terminal plugin hook, then reran focused tests, generated-baseline checks, TypeScript source check, architecture check, diff/merge-tree checks, focused formatting, and the real plugin-loaded failover proof.

clawsweeper · 2026-06-10T16:51:05Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

…nto feat/70976-model-failover-hooks # Conflicts: # docs/.generated/config-baseline.sha256 # docs/.generated/plugin-sdk-api-baseline.sha256 # src/auto-reply/reply/agent-runner-execution.test.ts # src/auto-reply/reply/agent-runner-execution.ts

…over-hooks # Conflicts: # docs/.generated/config-baseline.sha256 # docs/.generated/plugin-sdk-api-baseline.sha256

…over-hooks # Conflicts: # docs/.generated/plugin-sdk-api-baseline.sha256 # src/auto-reply/reply/agent-runner-execution.ts # src/auto-reply/reply/followup-runner.ts

mantisai-bot · 2026-06-16T04:37:40Z

@clawsweeper re-review

Current head 566a985e2773 is merge-clean and the PR body has exact-head real behavior proof. Also fixed the post-push checks-node-core-fast failure by updating the intentional Plugin SDK public export budget from 10,270 to 10,274; local targeted rerun passed.

clawsweeper · 2026-06-16T04:37:43Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27594530403
Updated: 2026-06-16T05:08:43.831Z

…over-hooks # Conflicts: # docs/.generated/config-baseline.sha256 # docs/.generated/plugin-sdk-api-baseline.sha256 # extensions/memory-core/src/memory/manager-fts-state.test.ts # scripts/plugin-sdk-surface-report.mjs # src/agents/embedded-agent-runner/run/failover-observation.ts # src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts # src/commands/migrate/skill-selection-prompt.ts # src/gateway/session-utils.ts # src/proxy-capture/store.sqlite.test.ts # ui/src/ui/views/workboard.ts

…over-hooks # Conflicts: # docs/.generated/config-baseline.sha256 # docs/.generated/plugin-sdk-api-baseline.sha256 # extensions/slack/src/monitor/send.runtime.ts # scripts/plugin-sdk-surface-report.mjs # src/agents/agent-tools.schema.test.ts # src/agents/model-fallback.ts # src/commands/onboard-channels.e2e.test.ts

…over-hooks # Conflicts: # scripts/plugin-sdk-surface-report.mjs

…over-hooks # Conflicts: # docs/.generated/config-baseline.sha256 # docs/.generated/plugin-sdk-api-baseline.sha256 # extensions/codex/src/app-server/app-inventory-cache.test.ts # extensions/codex/src/app-server/plugin-app-cache-key.ts # extensions/msteams/src/monitor-handler.ts # extensions/telegram/src/lane-delivery-text-deliverer.ts # scripts/plugin-sdk-surface-report.mjs # test/scripts/check-workflows.test.ts

openclaw-barnacle Bot added agents Agent runtime and tooling size: L labels Apr 24, 2026

greptile-apps Bot reviewed Apr 24, 2026

View reviewed changes

Comment thread src/plugins/hook-types.ts Outdated

clawsweeper Bot mentioned this pull request Apr 26, 2026

[Feature]: Emit first-class plugin hooks for model failover and terminal all-models-failed outcomes #70976

Closed

This was referenced Apr 27, 2026

[Meta] Feature request bundle: 11 platform gaps from intensive daily use #65824

Closed

Configurable user-facing message when all models are exhausted #39142

Closed

mantisai-bot force-pushed the feat/70976-model-failover-hooks branch from 4755b57 to ef986eb Compare May 9, 2026 04:02

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 9, 2026

This was referenced May 9, 2026

Feature Request: Proactive user notification on LLM failover/API errors #8130

Closed

Plugins: expose structured finalLlmOutcome on agent_end #49042

Closed

mantisai-bot force-pushed the feat/70976-model-failover-hooks branch from ef986eb to 2d82dfe Compare May 14, 2026 05:23

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 14, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 14, 2026

mantisai-bot force-pushed the feat/70976-model-failover-hooks branch from 2d82dfe to da7929f Compare May 14, 2026 05:36

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 14, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 14, 2026

mantisai-bot force-pushed the feat/70976-model-failover-hooks branch from da7929f to ab573be Compare May 16, 2026 05:53

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026

mantisai-bot force-pushed the feat/70976-model-failover-hooks branch from ab573be to 95dc679 Compare May 16, 2026 05:54

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 16, 2026

mantisai-bot force-pushed the feat/70976-model-failover-hooks branch from 95dc679 to c74ecfc Compare May 20, 2026 03:19

mantisai-bot and others added 9 commits June 10, 2026 19:45

refactor(plugins): tighten terminal failure hook types

ad56a93

fix(plugins): harden model failure hook contract

eb2c1d0

fix(plugins): report selected failover target

36ffd4f

chore(config): refresh generated baseline

c905374

fix(plugins): complete failover hook target propagation

2b90d22

fix(plugins): bound model failure hooks

1859c8c

fix(plugins): align failover hook payloads

6bccb67

chore(migrate): mark skill selection alias deprecated

57dd514

fix(plugins): propagate follow-up fallback targets

5cd9b67

JohnnyBraver added 2 commits June 15, 2026 06:54

Merge remote-tracking branch 'origin/main' into feat/70976-model-fail…

446a509

…over-hooks # Conflicts: # docs/.generated/config-baseline.sha256 # docs/.generated/plugin-sdk-api-baseline.sha256

kumaxs mentioned this pull request Jun 15, 2026

feat: Rate-limit fallback user-visible error notification (message-lifecycle Phase 2 extension) #92676

Open

JohnnyBraver added 2 commits June 16, 2026 07:22

Merge remote-tracking branch 'origin/main' into feat/70976-model-fail…

3af6432

…over-hooks # Conflicts: # docs/.generated/plugin-sdk-api-baseline.sha256 # src/auto-reply/reply/agent-runner-execution.ts # src/auto-reply/reply/followup-runner.ts

test(plugin-sdk): update public export budget

566a985

JohnnyBraver added 4 commits June 17, 2026 10:07

Merge origin/main into feat/70976-model-failover-hooks

ab652a9

ci: retrigger checks after actionlint download 502

7cd9f15

test(gateway): wait for chat completion in history smoke

61e7441

BradGroux mentioned this pull request Jun 21, 2026

WORKING: All Microsoft Issues and PRs (refresh) #74163

Draft

JohnnyBraver added 3 commits June 22, 2026 06:36

test(ui): align slash command import expectation

c935d15

Merge remote-tracking branch 'origin/main' into feat/70976-model-fail…

6a1b11f

…over-hooks # Conflicts: # scripts/plugin-sdk-surface-report.mjs

github-actions Bot mentioned this pull request Jun 22, 2026

🦞 OpenClaw 生态日报 2026-06-22 96loveslife/big_model_radar#49

Open

clawsweeper Bot mentioned this pull request Jun 28, 2026

[Feature] User notification + workspace context re-injection on model fallback #79163

Open

clawsweeper Bot mentioned this pull request Jul 2, 2026

feat(runtime): report model fallback transitions #98954

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(plugins): add model failover and terminal failure hooks#70990

feat(plugins): add model failover and terminal failure hooks#70990
mantisai-bot wants to merge 34 commits into
openclaw:mainfrom
mantisai-bot:feat/70976-model-failover-hooks

mantisai-bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Apr 24, 2026 •

edited

Loading

Comments Outside Diff (1)

Uh oh!

Uh oh!

clawsweeper Bot commented Apr 27, 2026 •

edited

Loading

Uh oh!

smonett commented May 7, 2026

Uh oh!

mantisai-bot commented May 9, 2026

Uh oh!

mantisai-bot commented Jun 10, 2026

Uh oh!

clawsweeper Bot commented Jun 10, 2026 •

edited

Loading

Uh oh!

mantisai-bot commented Jun 16, 2026

Uh oh!

clawsweeper Bot commented Jun 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mantisai-bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Current implementation

Security / compatibility

Real behavior proof

Regression coverage for the latest review findings

Validation

Latest rebase/proof refresh (2026-06-16)

Latest head refresh (2026-06-22)

Latest head refresh (2026-06-30)

Uh oh!

greptile-apps Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Comments Outside Diff (1)

Uh oh!

Uh oh!

clawsweeper Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smonett commented May 7, 2026

Uh oh!

mantisai-bot commented May 9, 2026

Uh oh!

mantisai-bot commented Jun 10, 2026

Uh oh!

clawsweeper Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mantisai-bot commented Jun 16, 2026

Uh oh!

clawsweeper Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mantisai-bot commented Apr 24, 2026 •

edited

Loading

greptile-apps Bot commented Apr 24, 2026 •

edited

Loading

clawsweeper Bot commented Apr 27, 2026 •

edited

Loading

clawsweeper Bot commented Jun 10, 2026 •

edited

Loading

clawsweeper Bot commented Jun 16, 2026 •

edited

Loading