Summary

The active-memory plugin (the optional blocking memory sub-agent that runs before eligible conversational sessions) has two related reliability issues:

1. The plugin pins to a single model / modelFallback pair and does not waterfall through configured fallback providers when its model becomes unavailable. When the configured provider has all profiles in cooldown or returns a structured error, the plugin emits FailoverError and drops the recall pass entirely instead of degrading gracefully.
2. The plugin's timeoutMs is the only knob for slow providers, but before_prompt_build simply times out without any structured signal back to the caller. When provider latency exceeds the configured timeout, the plugin silently fails on every turn, with no way for the host to learn that the timeout is the wrong knob.

Environment

• openclaw 2026.5.7
• Plugin: plugins.entries.active-memory
• Config in use:

  {
    "model": "google/gemini-3-flash-preview",
    "modelFallback": "google/gemini-3-flash-preview",
    "queryMode": "message",
    "timeoutMs": 30000,
    "circuitBreakerMaxTimeouts": 1,
    "circuitBreakerCooldownMs": 600000
  }

Evidence

Sub-bug 1 — no failover when provider unavailable

Live journal (today, between 19:16 and 22:30 EDT):

[plugins] active-memory: agent=main session=agent:main:telegram:direct:5192807992
  activeProvider=openai-codex activeModel=gpt-5.5 start timeoutMs=30000

[diagnostic] lane task error: lane=session:agent:main:telegram:direct:5192807992:active-memory:9e809d83a2d6
  durationMs=91 error="FailoverError: No available auth profile for openai-codex (all in cooldown or unavailable)."

[plugins] active-memory: ... activeProvider=openai-codex activeModel=gpt-5.5 failed
  error=No available auth profile for openai-codex (all in cooldown or unavailable).

Note that the host gateway has fallback providers configured (claude-cli, openrouter chain) and the main agent path uses them successfully. Active-memory does not.

Sub-bug 2 — timeout < provider latency

Operator measurement (May 6, 2026, via running gateway probe):

Plugin defaults are documented at 3000–5000 ms. Operator pushed timeoutMs to 30000, then 45000, and the plugin still times out:

2026-05-08T22:19:26.789-04:00 [plugins] [hooks] before_prompt_build handler from active-memory failed: timed out after 45000ms
2026-05-08T22:26:16.550-04:00 [plugins] [hooks] before_prompt_build handler from active-memory failed: timed out after 45000ms

Two timeouts in 7 minutes on a 45 s budget when the underlying probe takes 17–25 s suggests cold-start / queue contention compounds on top of base latency, and the timeout itself is the wrong knob.

Reproduction

1. Configure active-memory with model: openai-codex/gpt-5.5 (or any provider where you can force all profiles into cooldown).
2. Configure host fallback providers (e.g. claude-cli, openrouter).
3. Force the active-memory provider into cooldown (rate-limit all profiles).
4. Send a user message → observe failed: No available auth profile for openai-codex in the journal, and the active-memory recall pass is skipped entirely. The host's fallback chain is not consulted by the plugin.

For sub-bug 2: configure active-memory with a model whose typical first-token latency exceeds timeoutMs (e.g. gemini-3-flash-preview at timeoutMs: 5000). Every turn fails with before_prompt_build handler from active-memory failed: timed out after Nms.

Expected behavior

For #1 — failover

When the active-memory plugin's primary model fails with a recoverable error (rate_limit, usage_limit_reached, auth_invalid, provider 5xx, transport closed), it should:

• Walk the host's fallback_providers chain (or its own configured fallback chain).
• Honor credential_pool profile rotation for each provider it tries.
• Only emit FailoverError after exhausting the chain.
• Mark the timed-out provider in its circuit breaker so subsequent turns skip it cleanly.

For #2 — provider latency vs timeout

timeoutMs should be a budget, but the plugin should additionally:

• Keep a rolling p95 measurement of provider first-token latency.
• If the configured timeoutMs is below observed p95, log a latency_budget_under_provider_p95 warning once per cooldown window with the measured p95 value, instead of silently dropping every turn.
• Optionally short-circuit (skip the recall pass) when the provider's p95 has been above timeoutMs for N consecutive turns, to avoid wasting before_prompt_build time.

Behavioral guarantees

• Plugin should never silently drop a recall pass without surfacing a structured signal in the host journal that distinguishes "timed out", "provider unavailable / fell through fallback", and "circuit breaker tripped".
• Plugin should not block user-visible reply latency by tying up the gateway event loop for the full timeoutMs when the provider is already known to be unhealthy.

Suggested test coverage

In src/plugins/ (active-memory plugin tests):

1. fallback-on-provider-cooldown: configure active-memory with primary provider whose only profile is in cooldown; assert the host's fallback chain is consulted and the recall pass succeeds via fallback.
2. fallback-on-provider-5xx: primary returns 502; assert one fallback attempt and structured signal in observation log.
3. timeout-but-provider-was-slow: provider takes 20 s, timeoutMs: 5000; assert the plugin emits latency_budget_under_provider_p95 and circuit-breaker-trips after N consecutive timeouts.
4. circuit-breaker-cooldown-respects-config: after the breaker trips, subsequent turns within circuitBreakerCooldownMs skip the recall pass entirely without re-attempting.

Impact

• Plugin reliability under provider degradation (rate limits, slow providers, partial outages).
• Eliminates a class of silent failures where active-memory drops every turn for hours and the operator only notices via a before_prompt_build failed line in the journal.
• Aligns active-memory with the host gateway's existing fallback semantics.

Filed by

OpenClaw operator instance with corroborating journal evidence from a live Hermes deployment running openclaw 2026.5.7. Hermes / OpenClaw pair share this plugin via the same openclaw runtime package.