摘要 (Summary)

cooldownProbeUsedProviders is a module-level Set that lives for the entire gateway process. It is populated by every error classified as rate_limit / overloaded / timeout (including the client-side LLM idle timeout (120s): no response from model, which is not a model failure at all) / empty_response / unknown / unclassified / no_error_details. Once a provider is in the set, every subsequent run within the same process skips it — even across sessions and agents. The /model slash command only mutates the agent primary in config; it does NOT touch this Set, so user-side recovery is impossible. The only escape hatch is a full kill of the gateway process.

This is reproducible, has a clear source-level root cause, and the fix is small.

复现步骤 (Reproduction)

Environment: openclaw v2026.5.x (reproduced on 2026.5.28 and 2026.5.3-1), MiniMax-M3 (1.0M context) as agents.defaults.model.primary, MiniMax-M2.7 as first fallback, Qwen/Qwen3.5-397B-A17B second, exo/Qwen3.5-2B-MLX-8bit third. Repro is provider-agnostic — the issue is in runWithModelFallback, not the providers.

1. Configure a primary model that can fail with a timeout / overloaded / empty_response transient error (or just let the agent run a long-running turn that triggers the 120 s streaming idle timeout).
2. Let the primary turn abort / fail with that error.
3. Verify with openclaw gateway status (or any available observability) that the provider's cooldown slot is occupied.
4. From the same gateway process, in the Control UI:
   • run /model <another model on the same provider> — observe that the new model is not tried on the next turn, because the provider is still in the cooldown set.
   • open a brand-new session via /new or a fresh dashboard session — observe the same: the provider is still in the cooldown set.
5. Run openclaw gateway restart (or kill -TERM <gateway-pid> and let it respawn).
6. From the now-fresh process, retry the same turns — they succeed. The only recovery is process-level.

期望 vs 实际 (Expected vs Actual)

Expected

• A failed primary model turn should be retryable, either automatically after the cooldown window expires or via an explicit user action (e.g. /model, /reset-cooldown, openclaw models reset-cooldown).
• The user should be able to see the cooldown state of every provider/model so they understand why the agent is "stuck" on a fallback.
• The OpenClaw client should NOT classify its own 120 s streamWithIdleTimeout as a model-side timeout — that is a client-side measurement and could just as easily be triggered by a slow network or a slow transformers startup.

Actual

• cooldownProbeUsedProviders is populated by transient-client-side errors and is never cleared except by a process restart.
• /model does not clear it.
• The Control UI does not surface the cooldown state at all.
• Manual recovery requires killing the gateway, which the user only discovers by accident after they pkill node and find that everything starts working again.

根因（源码定位）— Root cause with source references

All paths are inside the openclaw package install (e.g. /opt/homebrew/lib/node_modules/openclaw/dist/). Bundle names are abbreviated.

1. The cooldown Set is module-level

dist/model-fallback--*.js

// Module top-level — global to the gateway process
const cooldownProbeUsedProviders = new Set();

// Once a provider is in the set, every subsequent run within the
// same process skips it (per-run + cross-session + cross-agent)
if (isTransientCooldownReason && cooldownProbeUsedProviders.has(candidate.provider)) {
    const error = `Provider ${candidate.provider} is in cooldown (probe already attempted this run)`;
    attempts.push({ provider: candidate.provider, error, reason: decision.reason });
    decision: "skip_candidate",
    // ...
}

if (!shouldPreserveTransientCooldownProbeSlot(probeFailureReason)) {
    cooldownProbeUsedProviders.add(transientProbeProviderForAttempt);
}

2. The reason classifier considers the client-side idle timeout as timeout

dist/selection-*.js (client-side streaming wrapper):

function streamWithIdleTimeout(baseFn, timeoutMs, onIdleTimeout) {
    return (model, context, options) => {
        const createIdleTimeoutError = () =>
            new Error(`LLM idle timeout (${Math.floor(timeoutMs / 1e3)}s): no response from model`);

        const createTimeoutPromise = (setTimer) => {
            return new Promise((_, reject) => {
                const timer = setTimeout(() => {
                    const error = createIdleTimeoutError();
                    abortStream(error);
                    onIdleTimeout?.(error);
                    reject(error);
                }, timeoutMs);
            });
        };
        // ...
    };
}

dist/errors-*.js (reason classification):

// classifyFailoverReasonFromCode:
default: return TIMEOUT_ERROR_CODES.has(normalized) ? "timeout" : null;

dist/model-fallback--*.js (cooldown classifier):

function shouldUseTransientCooldownProbeSlot(reason) {
    return reason === "rate_limit"
        || reason === "overloaded"
        || reason === "timeout"          // <-- LLM idle timeout ends up here
        || reason === "empty_response"
        || reason === "unknown"
        || reason === "unclassified"
        || reason === "no_error_details";
}

The LLM idle timeout (120s): no response from model error is generated by the OpenClaw client. It is not an API response. The provider never had a chance to return overloaded or rate_limit; the socket might have been mid-stream when the 120 s timer fired. Treating it as a provider-side timeout and adding the provider to the cooldown set is a category error: it confuses "the model did not produce anything for 120 s" (which can be slow-but-fine, e.g. a long-thinking turn on MiniMax-M3 with a 32k-token input) with "the provider is degraded and we should back off".

3. /model does not reset the cooldown

dist/get-reply-*.js and dist/agent-scope-*.js:

• runAutoFallbackPrimaryProbe reads from the per-session modelOverride, never from the global cooldownProbeUsedProviders.
• clearAutoFallbackPrimaryProbeSelection clears the session entry, not the module-level Set.

There is no command path that calls cooldownProbeUsedProviders.clear(). Only a process restart reaches Set initialisation.

4. The set is not observable

There is no Control UI panel, no openclaw models list --cooldown, no openclaw status line for it. Users have no way to know why a model is no longer being attempted.

建议修复（按优先级）— Suggested fixes

P0 — Stop the mis-classification

Either

(a) Exclude the client-side idle timeout from shouldUseTransientCooldownProbeSlot — change the timeout branch to be a finer signal that distinguishes "provider reported timeout / network timed out" from "client-side setTimeout fired". The client's LLM idle timeout (Ns): no response from model should be a separate reason: "client_idle_timeout" that does not poison the provider.

or

(b) Default streamWithIdleTimeout's timeoutMs to a much larger value (e.g. 600 s) for slow / thinking models, AND make the default configurable per-agent so a 32k-token M3 turn doesn't trip it.

P1 — Make the cooldown recoverable from the user side

• openclaw models reset-cooldown [provider ...] — clear entries from the set.
• /reset-cooldown slash command (session + cross-session).
• /model should always re-evaluate the cooldown set, not just change the primary in config.

P2 — Make the cooldown observable

• Add a column to the Models tab in Control UI showing the per-provider cooldown state, time-remaining, and the last error that poisoned it.
• Add a cooldownUntil line in openclaw status for the running gateway.
• Emit a one-shot log line (and a CLI flag) the first time a provider enters the cooldown set, including the reason and the source call site, so users can grep their gateway log.

P3 — Optional: stop persisting it across session boundaries

If the design intent really is "single-process global cooldown for safety", document it loudly in the model-selection docs and in the warning text. But based on user pain, the cross-session persistence is a foot-gun, not a feature. Consider scoping the Set to the per-run / per-session scope unless the operator opts in to global cooldown via agents.defaults.model.cooldownScope: "global".

影响范围 — Impact

• Severity: High. Locks the user out of the configured primary model for the rest of the gateway's uptime. In practice this means "until I notice and restart", which is hours to days.
• Scope: Every OpenClaw user with a slow / thinking primary model (MiniMax-M3, o1, Claude-with-extended-thinking, large-context DeepSeek, etc.) and a fallback chain.
• First-observed in production: 2026-06-09 on a MiniMax-M3 + MiniMax-M2.7 + Qwen-397B + Qwen2B setup. Confirmed with MiniMax-M2.7 itself which the user describes as "长期使用稳定" (long-term stable in production) — i.e. the regression is in the OpenClaw client, not in the provider.

复现配置 — Reproducer config

~/.openclaw/openclaw.json (excerpt, sanitised):

{
  "agents": {
    "defaults": {
      "model": {
        "primary": "minimax/MiniMax-M3",
        "fallbacks": [
          "minimax/MiniMax-M2.7",
          "modelscope/Qwen/Qwen3.5-397B-A17B",
          "exo/mlx-community/Qwen3.5-2B-MLX-8bit"
        ]
      },
      "subagents": {
        "model": {
          "primary": "minimax/MiniMax-M2.7",
          "fallbacks": [
            "modelscope/Qwen/Qwen3.5-397B-A17B",
            "exo/mlx-community/Qwen3.5-2B-MLX-8bit"
          ]
        }
      }
      // stream idle timeout: implicit 120s (default), not currently
      // surfaced as a top-level agent option
    }
  }
}

实际观察 — Observed evidence

• Trajectory of the affected session shows model.completed for MiniMax-M3 succeeded, then no further MiniMax-M3 calls in the next 11 minutes despite many user turns. Cooldown set kept the runner off the primary.
• MiniMax-M2.7 aborted exactly once with errorMessage: "LLM idle timeout (120s): no response from model" — a 32k-token input + tool-call chain. After that, minimax provider never recovered within the session.
• modelscope/Qwen/Qwen3.5-397B-A17B then 401'd (already fixed by the user with a key rotation), then 429'd (limit_burst_rate). Fallback chain exhausted.
• openclaw gateway restart immediately restored MiniMax-M3 availability. No config change.

我可以帮的

• 写一个最小复现 script（CLI / one-shot isolated session）让你附在 issue 里
• 提一个对应的 PR patch 给 dist/selection-*.js 把 client_idle_timeout 拆出来
• 提一个对应的 PR patch 给 dist/model-fallback-*.js 暴露 cooldownScope + models.reset-cooldown CLI

需要任何一个就告诉我。