You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Bug]: Z.AI Coding-Plan: ECONNRESET triggers model fallback — fallback notice is invisible to the user in async contexts (cron jobs, sub-agents, isolated runs) #94919
Behavior bug (incorrect output/state without crash)
Beta release blocker
No
Summary
OpenClaw 2026.6.8 with zai/glm-5-turbo as primary and a different model as fallback swaps the active model to the fallback provider whenever a model_fetch request fails with ECONNRESET against the Z.AI Coding-Plan endpoint. OpenClaw's agent runtime does emit a fallback notice (the format ↪️ Model Fallback: <active> (selected <selected>; <reason>) is built in /app/dist/agent-runner.runtime-BapylDFW.js), but the notice is rendered to the agent's local output stream, not surfaced back to the user who triggered the run. In async contexts (cron jobs, sub-agents, isolated agent runs) the line is buried in the sub-agent's own context; the user only sees the final response and the underlying zai → minimax swap is invisible. The failures are bursty and intermittent, correlate with concurrent fetches, and the end-user symptom is the assistant switching style/voice mid-task with no warning that the configured primary model was abandoned.
Set as primary with a fallback chain that includes a different model (e.g. minimax/MiniMax-M3).
Drive a workload that fires several model_fetch calls within a few hundred ms of each other — e.g. concurrent sub-agent spawns, heartbeat checks, multi-model tool-call chains, or a simple load test using the same API key directly against https://api.z.ai/api/coding/paas/v4/chat/completions.
In the gateway log, look for [model-fetch] error provider=zai ... causeCode=ECONNRESET followed by [model-fallback] decision=candidate_failed ... fallbackStepToModel=<fallback>.
Observe the fallback notice that OpenClaw emits: the format string in /app/dist/agent-runner.runtime-BapylDFW.js is ↪️ Model Fallback: <active> (selected <selected>; <reason>). In a synchronous chat context the user sees this line interleaved with the assistant's response; in an async context (cron job, sub-agent, isolated run) the line is emitted into the sub-agent's own local context, not the user-visible output stream that produced the trigger.
Expected behavior
When model_fetch against the configured Z.AI primary fails and a fallback model is selected, the user who triggered the run should be reliably informed — regardless of whether the run was synchronous (chat) or async (cron, sub-agent, isolated run). The fallback notice should reach the same output stream that the user's final response is delivered on, so a model swap is never invisible.
In addition, the transport should expose a per-provider connection/serialization knob so users with strict providers (Z.AI Coding-Plan, regional or self-hosted endpoints) can opt into a connection policy that respects the provider's limits, instead of relying on whatever default undici dispatcher the gateway shares across all providers. This would reduce the frequency of fallback events in the first place.
Actual behavior
Parallel model_fetch calls against https://api.z.ai/api/coding/paas/v4 fail with ECONNRESET in roughly 50–70% of attempts under bursty load. The fallback model activates on ECONNRESET and the chat thread continues as the new model for the rest of the turn. OpenClaw's agent runtime does emit a fallback notice (↪️ Model Fallback: <active> (selected <selected>; <reason>) — built in /app/dist/agent-runner.runtime-BapylDFW.js), but that notice is written to the agent's local output stream. In a synchronous chat context the user typically sees it; in async contexts (cron jobs, sub-agents, isolated agent runs) the line is buried in the sub-agent's own context and the user only sees the final response, with the underlying zai → minimax swap invisible. The user discovers the swap only by noticing a style/voice change in the response.
Why this happens (root cause, with sources)
The Z.AI Coding-Plan tier enforces per-account rate (concurrency) limits that vary by subscription tier. This is explicitly documented by Z.AI, not an undocumented quirk: [Z.AI Usage Policy](https://docs.z.ai/devpack/usage-policy) states "Rate (concurrency) limits are tied to your plan tier... Max > Pro > Lite", with recommended concurrent projects of Lite = 1, Pro = 1–2, Max = 2+, and higher concurrency during off-peak hours. The same policy explicitly suggests the workaround shape: "For each project, you can use methods like Subagent to make concurrent model calls" — i.e. one main session, with concurrent work delegated to sub-agents.
On the OpenClaw side, the model_fetch transport goes through sanitizeConfiguredProviderRequest → fetchWithSsrFGuard → createPinnedDispatcher → createHttp1Agent, but none of these layers currently consume a per-provider connection-policy field. All providers share the same undici dispatcher configuration, so the transport has no way to opt into a "1 concurrent request per host" behavior for Z.AI. The transport therefore hits the Coding-Plan's per-account limit, fails with ECONNRESET, and triggers the fallback chain.
On the user-visible side, buildFallbackNotice() in /app/dist/agent-runner.runtime-BapylDFW.js returns the notice to the agent-runner, which writes it into the local output of the agent that experienced the fallback. There is no propagation to the originating user's output stream when that agent was spawned as a sub-agent, cron job, or isolated run. As a result, the configured primary model zai/glm-5-turbo is abandoned in favor of the fallback, and the user who triggered the run has no way to know.
zai/glm-5-turbo(primary). Fallback chain observed:zai/glm-5-turbo → minimax/MiniMax-M3`. Reproducible with any Z.AI Coding-Plan model against the same endpoint.
Provider / routing chain
openclaw (embedded agent runtime) → zai provider (openai-completions, https://api.z.ai/api/coding/paas/v4) → anthropic-messages fallback (minimax via https://api.minimax.io/anthropic). No intermediate gateway or proxy. Auth profile zai:default (api_key mode).
Additional provider/model setup details
No response
Logs, screenshots, and evidence
Gateway log excerpt (anonymized, from a 30-minute window with 4 parallel sub-agents running):
[model-fetch] start provider=zai api=openai-completions model=glm-5-turbo method=POST url=https://api.z.ai/api/coding/paas/v4/chat/completions timeoutMs=300000 proxy=none policy=custom
[model-fetch] error provider=zai api=openai-completions model=glm-5-turbo elapsedMs=4200 name=SocketError causeCode=ECONNRESET message=read ECONNRESET
[model-fallback] requestedProvider=zai requestedModel=glm-5-turbo decision=candidate_failed attempt=1 fallbackStepFromFailureReason=timeout fallbackStepFromFailureDetail="fetch failed"
[model-fetch] start provider=minimax api=anthropic-messages model=MiniMax-M3 ...
[model-fallback] decision=candidate_succeeded ... fallbackStepToModel=minimax/MiniMax-M3 ...
The fallback notice that the agent-runner would emit (verified insource at `/app/dist/agent-runner.runtime-BapylDFW.js`):
↪️ Model Fallback: minimax/MiniMax-M3 (selected zai/glm-5-turbo; timeout)
This line is rendered into the agent's local output stream. In a synchronous chat context the user typically sees it interleaved with the assistant response. In an async context (cron job, sub-agent, isolated run) it is written into the spawned agent's own context and the user who triggered the run does not see it.
Direct measurement from the same host, same `ZAI_API_KEY`, same model, using Node `fetch`:
| Endpoint | N=4 | N=6 | N=8 ||---------------------------------------------------|------|--------|--------||`https://api.z.ai/api/coding/paas/v4` (Global) | 1-3/4 |**2/6**| 3-4/8 ||`https://open.bigmodel.cn/api/coding/paas/v4` (CN) | 4/4 |**6/6**| 8/8 |
The CN endpoint is the same provider, same key, same model, same API; the only difference is the load balancer. This localizes the failure to the Global endpoint's LB and rules out OpenClaw's HTTP transport as the proximate cause. (Note for European users: the CN endpoint is **not** a recommended workaround for GDPR / data-residency reasons. It is used here purely as a technical control to localize the failure to the Global endpoint's LB.)External corroboration of the per-account concurrency limit on the Z.AI Coding-Plan tier:- [anomalyco/opencode #8618](https://github.com/anomalyco/opencode/issues/8618) — community confirmation that parallel requests fail on the same key- [z.ai/subscribe](https://z.ai/subscribe) — official subscribe page FAQ which itself asks "How many concurrent connections does the plan support, and why does it sometimes feel like there's only one concurrent connection?" (acknowledges the issue at the provider level)
Impact and severity
No response
Additional information
Workaround (current)
A user-space globalThis.fetch wrapper hook at hooks/init-tools/zai-connection-limiter.js that intercepts requests to api.z.ai and routes them through a custom undici Agent with connections: 1 and Connection: close header achieves a partial improvement (~50% success rate). It is not production-safe: the wrap happens before fetchWithSsrFGuard and bypasses the SSRF guard's pinned DNS dispatcher (see related issue #87763). The hook also has to be maintained per-host and is fragile across OpenClaw upgrades.
Suggested fix
Two complementary changes would resolve the user-visible problem:
Expose a per-provider providerRequest.fetchOptions (or equivalently providerRequest.dispatcher) configuration surface that flows through the existing transport pipeline:
Implementation should extend sanitizeConfiguredProviderRequest (whitelist validation, bounded numeric ranges only) and thread the options through buildProviderRequestDispatcherPolicy → fetchWithSsrFGuard → createPinnedDispatcher → createHttp1Agent. Keeping the option inside the existing fetchWithSsrFGuard path preserves the SSRF-guard's DNS-pinning and policy validation (no bypass). The pattern of whitelisted per-provider scalar config is already established in the repo by #65143 (feat(models): add per-provider timeoutSeconds config).
2. Propagate fallback notices to the originating user output (close the visibility gap):
When a sub-agent, cron job, or isolated run falls back from the configured primary to a fallback model, propagate the ↪️ Model Fallback: <active> (selected <selected>; <reason>) notice to the same output stream that the originating user is reading. Either through explicit propagation in resolveFallbackTransition() / buildFallbackNotice() in agent-runner.runtime-BapylDFW.js, or by elevating the notice to a session-level event that the channel/transport layer is expected to surface.
A follow-up enhancement would be a small in-process circuit-breaker on ECONNRESET with exponential backoff and jitter per provider-host (e.g. 3 consecutive failures → open for 30s, then half-open trial) so that providers with strict 1-concurrent policies fail fast into the user-visible error path instead of silently swapping models.
Related issues
Same failure family, different provider or trigger
Session context silently lost between consecutive turns with z.ai provider (GLM gateway) #76665 — Session context silently lost between consecutive turns with z.ai provider (GLM gateway) (different proximate cause — session-history pruning vs ECONNRESET — but same end-user experience: an OpenClaw + Z.AI interaction silently produces incorrect behavior, and the maintainer labels include impact:data-loss and impact:message-loss. A general "Z.AI providers need explicit user-visible failure paths" theme is visible across both issues.)
Bug type
Behavior bug (incorrect output/state without crash)
Beta release blocker
No
Summary
OpenClaw 2026.6.8 with
zai/glm-5-turboas primary and a different model as fallback swaps the active model to the fallback provider whenever amodel_fetchrequest fails withECONNRESETagainst the Z.AI Coding-Plan endpoint. OpenClaw's agent runtime does emit a fallback notice (the format↪️ Model Fallback: <active> (selected <selected>; <reason>)is built in/app/dist/agent-runner.runtime-BapylDFW.js), but the notice is rendered to the agent's local output stream, not surfaced back to the user who triggered the run. In async contexts (cron jobs, sub-agents, isolated agent runs) the line is buried in the sub-agent's own context; the user only sees the final response and the underlyingzai → minimaxswap is invisible. The failures are bursty and intermittent, correlate with concurrent fetches, and the end-user symptom is the assistant switching style/voice mid-task with no warning that the configured primary model was abandoned.Steps to reproduce
openclaw.json:{ "models": { "providers": { "zai": { "api": "openai-completions", "baseUrl": "https://api.z.ai/api/coding/paas/v4", "apiKey": "***}", "models": [{ "id": "glm-5-turbo", "name": "GLM-5 Turbo" }] } } } }minimax/MiniMax-M3).model_fetchcalls within a few hundred ms of each other — e.g. concurrent sub-agent spawns, heartbeat checks, multi-model tool-call chains, or a simple load test using the same API key directly againsthttps://api.z.ai/api/coding/paas/v4/chat/completions.[model-fetch] error provider=zai ... causeCode=ECONNRESETfollowed by[model-fallback] decision=candidate_failed ... fallbackStepToModel=<fallback>./app/dist/agent-runner.runtime-BapylDFW.jsis↪️ Model Fallback: <active> (selected <selected>; <reason>). In a synchronous chat context the user sees this line interleaved with the assistant's response; in an async context (cron job, sub-agent, isolated run) the line is emitted into the sub-agent's own local context, not the user-visible output stream that produced the trigger.Expected behavior
When
model_fetchagainst the configured Z.AI primary fails and a fallback model is selected, the user who triggered the run should be reliably informed — regardless of whether the run was synchronous (chat) or async (cron, sub-agent, isolated run). The fallback notice should reach the same output stream that the user's final response is delivered on, so a model swap is never invisible.In addition, the transport should expose a per-provider connection/serialization knob so users with strict providers (Z.AI Coding-Plan, regional or self-hosted endpoints) can opt into a connection policy that respects the provider's limits, instead of relying on whatever default
undicidispatcher the gateway shares across all providers. This would reduce the frequency of fallback events in the first place.Actual behavior
Parallel
model_fetchcalls againsthttps://api.z.ai/api/coding/paas/v4fail withECONNRESETin roughly 50–70% of attempts under bursty load. The fallback model activates onECONNRESETand the chat thread continues as the new model for the rest of the turn. OpenClaw's agent runtime does emit a fallback notice (↪️ Model Fallback: <active> (selected <selected>; <reason>)— built in/app/dist/agent-runner.runtime-BapylDFW.js), but that notice is written to the agent's local output stream. In a synchronous chat context the user typically sees it; in async contexts (cron jobs, sub-agents, isolated agent runs) the line is buried in the sub-agent's own context and the user only sees the final response, with the underlyingzai → minimaxswap invisible. The user discovers the swap only by noticing a style/voice change in the response.Why this happens (root cause, with sources)
The Z.AI Coding-Plan tier enforces per-account rate (concurrency) limits that vary by subscription tier. This is explicitly documented by Z.AI, not an undocumented quirk: [Z.AI Usage Policy](https://docs.z.ai/devpack/usage-policy) states "Rate (concurrency) limits are tied to your plan tier... Max > Pro > Lite", with recommended concurrent projects of Lite = 1, Pro = 1–2, Max = 2+, and higher concurrency during off-peak hours. The same policy explicitly suggests the workaround shape: "For each project, you can use methods like Subagent to make concurrent model calls" — i.e. one main session, with concurrent work delegated to sub-agents.
Z.AI also recommends GLM-5-Turbo specifically for the OpenClaw scenario (https://docs.z.ai/guides/llm/glm-5-turbo).
On the OpenClaw side, the
model_fetchtransport goes throughsanitizeConfiguredProviderRequest→fetchWithSsrFGuard→createPinnedDispatcher→createHttp1Agent, but none of these layers currently consume a per-provider connection-policy field. All providers share the same undici dispatcher configuration, so the transport has no way to opt into a "1 concurrent request per host" behavior for Z.AI. The transport therefore hits the Coding-Plan's per-account limit, fails withECONNRESET, and triggers the fallback chain.On the user-visible side,
buildFallbackNotice()in/app/dist/agent-runner.runtime-BapylDFW.jsreturns the notice to the agent-runner, which writes it into the local output of the agent that experienced the fallback. There is no propagation to the originating user's output stream when that agent was spawned as a sub-agent, cron job, or isolated run. As a result, the configured primary modelzai/glm-5-turbois abandoned in favor of the fallback, and the user who triggered the run has no way to know.OpenClaw version
2026.6.8
Operating system
Linux 6.18.33-Unraid (x64), Docker container
Install method
Docker (
openclaw/openclawimage,openclaw/openclaw:2026.6.8)Model
zai/glm-5-turbo
(primary). Fallback chain observed:zai/glm-5-turbo → minimax/MiniMax-M3`. Reproducible with any Z.AI Coding-Plan model against the same endpoint.Provider / routing chain
openclaw (embedded agent runtime) → zai provider (openai-completions, https://api.z.ai/api/coding/paas/v4) → anthropic-messages fallback (minimax via https://api.minimax.io/anthropic). No intermediate gateway or proxy. Auth profilezai:default(api_key mode).Additional provider/model setup details
No response
Logs, screenshots, and evidence
Impact and severity
No response
Additional information
Workaround (current)
A user-space
globalThis.fetchwrapper hook athooks/init-tools/zai-connection-limiter.jsthat intercepts requests toapi.z.aiand routes them through a custom undiciAgentwithconnections: 1andConnection: closeheader achieves a partial improvement (~50% success rate). It is not production-safe: the wrap happens beforefetchWithSsrFGuardand bypasses the SSRF guard's pinned DNS dispatcher (see related issue #87763). The hook also has to be maintained per-host and is fragile across OpenClaw upgrades.Suggested fix
Two complementary changes would resolve the user-visible problem:
1. Per-provider
fetchOptionsconfig (reduce fallback frequency):Expose a per-provider
providerRequest.fetchOptions(or equivalentlyproviderRequest.dispatcher) configuration surface that flows through the existing transport pipeline:Implementation should extend
sanitizeConfiguredProviderRequest(whitelist validation, bounded numeric ranges only) and thread the options throughbuildProviderRequestDispatcherPolicy→fetchWithSsrFGuard→createPinnedDispatcher→createHttp1Agent. Keeping the option inside the existingfetchWithSsrFGuardpath preserves the SSRF-guard's DNS-pinning and policy validation (no bypass). The pattern of whitelisted per-provider scalar config is already established in the repo by #65143 (feat(models): add per-provider timeoutSeconds config).2. Propagate fallback notices to the originating user output (close the visibility gap):
When a sub-agent, cron job, or isolated run falls back from the configured primary to a fallback model, propagate the
↪️ Model Fallback: <active> (selected <selected>; <reason>)notice to the same output stream that the originating user is reading. Either through explicit propagation inresolveFallbackTransition()/buildFallbackNotice()inagent-runner.runtime-BapylDFW.js, or by elevating the notice to a session-level event that the channel/transport layer is expected to surface.A follow-up enhancement would be a small in-process circuit-breaker on
ECONNRESETwith exponential backoff and jitter per provider-host (e.g. 3 consecutive failures → open for 30s, then half-open trial) so that providers with strict 1-concurrent policies fail fast into the user-visible error path instead of silently swapping models.Related issues
Same failure family, different provider or trigger
UND_ERR_SOCKETkeep-alive failures trigger silent mid-turn fallback (same symptom: silent model swap mid-conversation, different proximate cause — keep-alive socket aging rather than LB connection limit)UND_ERR_SOCKETon Windows WSL when connecting to Google Gemini (sameSocketErrorfamily, different host and platform)genericRepeatcritical/circuit-breaker never fires when exec results vary (related to the circuit-breaker idea in the Suggested fix above)globalThis.fetchworkaround for this issue bypasses the SSRF guard discussed there)Different bug, same "silent z.ai" UX pattern
ECONNRESET— but same end-user experience: an OpenClaw + Z.AI interaction silently produces incorrect behavior, and the maintainer labels includeimpact:data-lossandimpact:message-loss. A general "Z.AI providers need explicit user-visible failure paths" theme is visible across both issues.)