Skip to content

[Bug]: Z.AI Coding-Plan: ECONNRESET triggers model fallback — fallback notice is invisible to the user in async contexts (cron jobs, sub-agents, isolated runs) #94919

Description

@moccassins

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

OpenClaw 2026.6.8 with zai/glm-5-turbo as primary and a different model as fallback swaps the active model to the fallback provider whenever a model_fetch request fails with ECONNRESET against the Z.AI Coding-Plan endpoint. OpenClaw's agent runtime does emit a fallback notice (the format ↪️ Model Fallback: <active> (selected <selected>; <reason>) is built in /app/dist/agent-runner.runtime-BapylDFW.js), but the notice is rendered to the agent's local output stream, not surfaced back to the user who triggered the run. In async contexts (cron jobs, sub-agents, isolated agent runs) the line is buried in the sub-agent's own context; the user only sees the final response and the underlying zai → minimax swap is invisible. The failures are bursty and intermittent, correlate with concurrent fetches, and the end-user symptom is the assistant switching style/voice mid-task with no warning that the configured primary model was abandoned.

Steps to reproduce

  1. Configure a Z.AI provider in openclaw.json:
    {
      "models": {
        "providers": {
          "zai": {
            "api": "openai-completions",
            "baseUrl": "https://api.z.ai/api/coding/paas/v4",
            "apiKey": "***}",
            "models": [{ "id": "glm-5-turbo", "name": "GLM-5 Turbo" }]
          }
        }
      }
    }
  2. Set as primary with a fallback chain that includes a different model (e.g. minimax/MiniMax-M3).
  3. Drive a workload that fires several model_fetch calls within a few hundred ms of each other — e.g. concurrent sub-agent spawns, heartbeat checks, multi-model tool-call chains, or a simple load test using the same API key directly against https://api.z.ai/api/coding/paas/v4/chat/completions.
  4. In the gateway log, look for [model-fetch] error provider=zai ... causeCode=ECONNRESET followed by [model-fallback] decision=candidate_failed ... fallbackStepToModel=<fallback>.
  5. Observe the fallback notice that OpenClaw emits: the format string in /app/dist/agent-runner.runtime-BapylDFW.js is ↪️ Model Fallback: <active> (selected <selected>; <reason>). In a synchronous chat context the user sees this line interleaved with the assistant's response; in an async context (cron job, sub-agent, isolated run) the line is emitted into the sub-agent's own local context, not the user-visible output stream that produced the trigger.

Expected behavior

When model_fetch against the configured Z.AI primary fails and a fallback model is selected, the user who triggered the run should be reliably informed — regardless of whether the run was synchronous (chat) or async (cron, sub-agent, isolated run). The fallback notice should reach the same output stream that the user's final response is delivered on, so a model swap is never invisible.

In addition, the transport should expose a per-provider connection/serialization knob so users with strict providers (Z.AI Coding-Plan, regional or self-hosted endpoints) can opt into a connection policy that respects the provider's limits, instead of relying on whatever default undici dispatcher the gateway shares across all providers. This would reduce the frequency of fallback events in the first place.

Actual behavior

Parallel model_fetch calls against https://api.z.ai/api/coding/paas/v4 fail with ECONNRESET in roughly 50–70% of attempts under bursty load. The fallback model activates on ECONNRESET and the chat thread continues as the new model for the rest of the turn. OpenClaw's agent runtime does emit a fallback notice (↪️ Model Fallback: <active> (selected <selected>; <reason>) — built in /app/dist/agent-runner.runtime-BapylDFW.js), but that notice is written to the agent's local output stream. In a synchronous chat context the user typically sees it; in async contexts (cron jobs, sub-agents, isolated agent runs) the line is buried in the sub-agent's own context and the user only sees the final response, with the underlying zai → minimax swap invisible. The user discovers the swap only by noticing a style/voice change in the response.

Why this happens (root cause, with sources)

The Z.AI Coding-Plan tier enforces per-account rate (concurrency) limits that vary by subscription tier. This is explicitly documented by Z.AI, not an undocumented quirk: [Z.AI Usage Policy](https://docs.z.ai/devpack/usage-policy) states "Rate (concurrency) limits are tied to your plan tier... Max > Pro > Lite", with recommended concurrent projects of Lite = 1, Pro = 1–2, Max = 2+, and higher concurrency during off-peak hours. The same policy explicitly suggests the workaround shape: "For each project, you can use methods like Subagent to make concurrent model calls" — i.e. one main session, with concurrent work delegated to sub-agents.

Z.AI also recommends GLM-5-Turbo specifically for the OpenClaw scenario (https://docs.z.ai/guides/llm/glm-5-turbo).

On the OpenClaw side, the model_fetch transport goes through sanitizeConfiguredProviderRequestfetchWithSsrFGuardcreatePinnedDispatchercreateHttp1Agent, but none of these layers currently consume a per-provider connection-policy field. All providers share the same undici dispatcher configuration, so the transport has no way to opt into a "1 concurrent request per host" behavior for Z.AI. The transport therefore hits the Coding-Plan's per-account limit, fails with ECONNRESET, and triggers the fallback chain.

On the user-visible side, buildFallbackNotice() in /app/dist/agent-runner.runtime-BapylDFW.js returns the notice to the agent-runner, which writes it into the local output of the agent that experienced the fallback. There is no propagation to the originating user's output stream when that agent was spawned as a sub-agent, cron job, or isolated run. As a result, the configured primary model zai/glm-5-turbo is abandoned in favor of the fallback, and the user who triggered the run has no way to know.

OpenClaw version

2026.6.8

Operating system

Linux 6.18.33-Unraid (x64), Docker container

Install method

Docker (openclaw/openclaw image, openclaw/openclaw:2026.6.8)

Model

zai/glm-5-turbo(primary). Fallback chain observed:zai/glm-5-turbo → minimax/MiniMax-M3`. Reproducible with any Z.AI Coding-Plan model against the same endpoint.

Provider / routing chain

openclaw (embedded agent runtime) → zai provider (openai-completions, https://api.z.ai/api/coding/paas/v4) → anthropic-messages fallback (minimax via https://api.minimax.io/anthropic). No intermediate gateway or proxy. Auth profile zai:default (api_key mode).

Additional provider/model setup details

No response

Logs, screenshots, and evidence

Gateway log excerpt (anonymized, from a 30-minute window with 4 parallel sub-agents running):


[model-fetch] start provider=zai api=openai-completions model=glm-5-turbo method=POST url=https://api.z.ai/api/coding/paas/v4/chat/completions timeoutMs=300000 proxy=none policy=custom
[model-fetch] error provider=zai api=openai-completions model=glm-5-turbo elapsedMs=4200 name=SocketError causeCode=ECONNRESET message=read ECONNRESET
[model-fallback] requestedProvider=zai requestedModel=glm-5-turbo decision=candidate_failed attempt=1 fallbackStepFromFailureReason=timeout fallbackStepFromFailureDetail="fetch failed"
[model-fetch] start provider=minimax api=anthropic-messages model=MiniMax-M3 ...
[model-fallback] decision=candidate_succeeded ... fallbackStepToModel=minimax/MiniMax-M3 ...


The fallback notice that the agent-runner would emit (verified in source at `/app/dist/agent-runner.runtime-BapylDFW.js`):


↪️ Model Fallback: minimax/MiniMax-M3 (selected zai/glm-5-turbo; timeout)


This line is rendered into the agent's local output stream. In a synchronous chat context the user typically sees it interleaved with the assistant response. In an async context (cron job, sub-agent, isolated run) it is written into the spawned agent's own context and the user who triggered the run does not see it.

Direct measurement from the same host, same `ZAI_API_KEY`, same model, using Node `fetch`:

| Endpoint                                          | N=4  | N=6    | N=8    |
|---------------------------------------------------|------|--------|--------|
| `https://api.z.ai/api/coding/paas/v4` (Global)    | 1-3/4 | **2/6** | 3-4/8  |
| `https://open.bigmodel.cn/api/coding/paas/v4` (CN) | 4/4  | **6/6** | 8/8    |

The CN endpoint is the same provider, same key, same model, same API; the only difference is the load balancer. This localizes the failure to the Global endpoint's LB and rules out OpenClaw's HTTP transport as the proximate cause. (Note for European users: the CN endpoint is **not** a recommended workaround for GDPR / data-residency reasons. It is used here purely as a technical control to localize the failure to the Global endpoint's LB.)

External corroboration of the per-account concurrency limit on the Z.AI Coding-Plan tier:

- [anomalyco/opencode #8618](https://github.com/anomalyco/opencode/issues/8618) — community confirmation that parallel requests fail on the same key
- [z.ai/subscribe](https://z.ai/subscribe) — official subscribe page FAQ which itself asks "How many concurrent connections does the plan support, and why does it sometimes feel like there's only one concurrent connection?" (acknowledges the issue at the provider level)

Impact and severity

No response

Additional information

Workaround (current)

A user-space globalThis.fetch wrapper hook at hooks/init-tools/zai-connection-limiter.js that intercepts requests to api.z.ai and routes them through a custom undici Agent with connections: 1 and Connection: close header achieves a partial improvement (~50% success rate). It is not production-safe: the wrap happens before fetchWithSsrFGuard and bypasses the SSRF guard's pinned DNS dispatcher (see related issue #87763). The hook also has to be maintained per-host and is fragile across OpenClaw upgrades.

Suggested fix

Two complementary changes would resolve the user-visible problem:

1. Per-provider fetchOptions config (reduce fallback frequency):

Expose a per-provider providerRequest.fetchOptions (or equivalently providerRequest.dispatcher) configuration surface that flows through the existing transport pipeline:

"zai": {
  "api": "openai-completions",
  "baseUrl": "https://api.z.ai/api/coding/paas/v4",
  "providerRequest": {
    "fetchOptions": {
      "connections": 1,
      "pipelining": 0,
      "keepAlive": false,
      "keepAliveTimeout": 0,
      "headersTimeout": 300000,
      "bodyTimeout": 300000
    }
  }
}

Implementation should extend sanitizeConfiguredProviderRequest (whitelist validation, bounded numeric ranges only) and thread the options through buildProviderRequestDispatcherPolicyfetchWithSsrFGuardcreatePinnedDispatchercreateHttp1Agent. Keeping the option inside the existing fetchWithSsrFGuard path preserves the SSRF-guard's DNS-pinning and policy validation (no bypass). The pattern of whitelisted per-provider scalar config is already established in the repo by #65143 (feat(models): add per-provider timeoutSeconds config).

2. Propagate fallback notices to the originating user output (close the visibility gap):

When a sub-agent, cron job, or isolated run falls back from the configured primary to a fallback model, propagate the ↪️ Model Fallback: <active> (selected <selected>; <reason>) notice to the same output stream that the originating user is reading. Either through explicit propagation in resolveFallbackTransition() / buildFallbackNotice() in agent-runner.runtime-BapylDFW.js, or by elevating the notice to a session-level event that the channel/transport layer is expected to surface.

A follow-up enhancement would be a small in-process circuit-breaker on ECONNRESET with exponential backoff and jitter per provider-host (e.g. 3 consecutive failures → open for 30s, then half-open trial) so that providers with strict 1-concurrent policies fail fast into the user-visible error path instead of silently swapping models.

Related issues

Same failure family, different provider or trigger

Different bug, same "silent z.ai" UX pattern

  • Session context silently lost between consecutive turns with z.ai provider (GLM gateway) #76665 — Session context silently lost between consecutive turns with z.ai provider (GLM gateway) (different proximate cause — session-history pruning vs ECONNRESET — but same end-user experience: an OpenClaw + Z.AI interaction silently produces incorrect behavior, and the maintainer labels include impact:data-loss and impact:message-loss. A general "Z.AI providers need explicit user-visible failure paths" theme is visible across both issues.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:needs-security-reviewClawSweeper marked this issue as needing security-sensitive review.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions