Problem Statement

A single misbehaving plugin can currently degrade the entire gateway by holding session locks too long, timing out repeatedly, or throwing enough errors to trigger restart churn and queue growth. The workspace now has diagnostics for lock-hold overruns, session-store growth, and gateway-log anomaly detection, but the framework still lacks a plugin-scoped circuit breaker that can degrade one plugin gracefully while keeping the rest of the gateway alive.

Proposed Design Summary

Add a framework-level plugin circuit breaker in the gateway/plugin runtime. The breaker should:

• wrap plugin hook execution with timing and error capture,
• attribute lock ownership to the active plugin and hook,
• track rolling lock overruns and failure counts,
• persist breaker state to a small runtime registry,
• skip only the affected plugin hooks when the breaker opens,
• recover automatically through a half-open trial path, with a manual forced-open reset path for operators.

Failure Model

The breaker should evaluate three signal classes:

1. Lock hold overruns
   • Trigger when a plugin-associated lock is held past its configured budget.
   • Requires new lock telemetry that includes plugin id, hook name, and session id.
2. Repeated plugin errors
   • Trigger when the same plugin throws, rejects, or times out repeatedly in a bounded window.
   • Timeout exits should count toward the error budget.
3. Gateway-impact correlation
   • Use restart spikes, queue growth, and repeated lock-timeout patterns as supporting confidence signals.
   • This should boost confidence, not be the primary trigger.

Breaker States

• closed: normal operation.
• open: the plugin is auto-disabled for selected hooks because it exceeded budget.
• half-open: cooldown expired; allow a small number of trial invocations.
• forced-open: manual operator quarantine; no automatic reset.

Configuration / Policy

Suggested starting budgets from the design work:

• context-guard: warn 15s, open 30s, 3 failures in 10m
• mission-control: warn 10s, open 20s, 5 failures in 10m
• moltbot-mem0-memory: warn 20s, open 45s, 4 failures in 15m
• interactive-session-watchdog: warn 10s, open 20s, 3 failures in 10m
• default/unknown plugin: warn 15s, open 30s, 3 failures in 10m

Framework-wide guardrails:

• any single lock over 60s is a critical event,
• any plugin that reopens its breaker 3 times in 1h should move to forced-open until manual reset.

Required operator/config surfaces:

• persisted breaker registry (for example .clawdbot/runtime/plugin-circuit-breakers.json),
• read-only breaker visibility in gateway status output,
• manual reset command such as clawdbot plugins circuit-breaker reset <plugin-id>,
• per-plugin fail mode support for skip, fail-request, or manual-only.

Design Doc

• Full design: https://github.com/XanderXML-Bit/OpenClaw/blob/master/docs/circuit-breaker-design.md

Rollout Recommendation

1. Ship hook timing and error telemetry first.
2. Add lock attribution second.
3. Run passive breaker evaluation in log-only mode.
4. Enable active open/half-open behavior behind a gateway flag.
5. After telemetry confirms budgets, enable by default for non-critical plugins.