Summary
When api.telegram.org is unreachable (offline, regional block, blackholed route), the gateway emits 1,500+ ENETUNREACH errors per hour. Each Telegram fetch runs the 3-attempt cascade in extensions/telegram/src/fetch.ts (DNS-resolved → IPv4-only → hardcoded fallback IP 149.154.167.220) without any global circuit-breaker state, so each long-poll cycle wastes three connection attempts and immediately re-fires.
Multiplied across getUpdates polling and concurrent media downloads, this produces sustained event-loop starvation (eventLoopDelayMaxMs > 3000ms), which in turn causes the TUI streaming watchdog (dist/tui-…js, default ~30s) to falsely report backend may have dropped this run silently — send a new message to resync on otherwise-healthy Codex/OpenAI streams.
The streaming watchdog is doing its job — it's reporting that no frames arrived for 30s. The frames didn't arrive because the gateway loop was blocked by Telegram retries, not because the upstream model failed.
Repro
- Have Telegram channel enabled in
openclaw.json with active long-poll (entries.telegram.enabled: true)
- Disconnect WiFi (or block
149.154.167.220:443 at the firewall) while leaving DNS resolution working
- Within ~5 minutes, gateway logs fill with
ENETUNREACH 149.154.167.220:443 (~25/min)
- After ~10–15 minutes, liveness warnings appear with
eventLoopDelayMaxMs > 1000ms despite no real workload
- TUI sessions running on healthy providers (e.g.
openai-codex/gpt-5.5) trip the streaming watchdog despite the upstream stream being alive
Evidence (one user's gateway, last 60 min)
- 1,535
ENETUNREACH 149.154.167.220:443 errors in 60 min (~25/min)
eventLoopDelayMaxMs = 9646.9, 3116.4, 2803.9, 1722.8 across multiple windows
- TUI watchdog fired on a session backed by
openai-codex/gpt-5.5 while OpenAI itself was healthy (verified by openclaw agent --agent main --message "respond with PONG" returning PONG in the same time window from a separate shell)
- Standalone
curl --max-time 5 https://149.154.167.220/ returns code 000 (immediate failure), while curl https://api.telegram.org/ returns 302 — DNS-based path works, the hardcoded fallback IP route does not
Source references
dist/fetch-B1BHd80D.js:135 — TELEGRAM_FALLBACK_IPS = [\"149.154.167.220\"] (single hardcoded IP, no rotation, no health awareness)
dist/fetch-B1BHd80D.js:383-426 — createTelegramTransportAttempts: cascade builds 3 attempts per request with no per-host circuit-breaker state
dist/fetch-B1BHd80D.js:149-155 — FALLBACK_RETRY_ERROR_CODES correctly identifies ENETUNREACH / EHOSTUNREACH etc. as retry triggers, but the cascade still re-runs on every getUpdates cycle
- No matches in the codebase for
circuitBreaker, markUnhealthy, or exponentialBackoff — the pattern is genuinely missing
Adjacent issues (related, different scope)
These came up in search; this issue is differentiated by being about the inbound getUpdates fetch transport cascade, not outbound actions or webhooks:
Suggested fix (first-principles)
Add a per-host circuit breaker in the Telegram fetch dispatcher path (Hystrix/Polly pattern, Google SRE Ch. 22):
- Track consecutive transient failures per
(host, error-class) key
- After N consecutive failures within window W → state=
OPEN: short-circuit further requests with host_unhealthy for cooldown T
- After T → state=
HALF_OPEN: allow one probe; success → CLOSED + reset, failure → back to OPEN with longer T (exponential, capped)
- Reset to
CLOSED on any successful response
Defaults that would match the telemetry above: N=5, W=30s, T=10s initial doubling to max 60s. All configurable under channels.telegram.transport.circuitBreaker.{ failureThreshold, window, cooldown, maxCooldown }.
Pair with exponential backoff + jitter on the getUpdates polling loop itself:
delay_ms = min(MAX_DELAY, BASE * 2^consecutive_failures)
delay_ms += random_jitter(0, delay_ms / 2)
Suggested defaults: BASE=500ms, MAX_DELAY=60s. Reset on first successful response. Jitter prevents thundering-herd on shared networks.
The same component should be reusable by Discord (#77634), Slack (#58519), and future channels — strong fit for the framework in #41899.
Out of scope (cross-reference, would file separately if useful)
Happy to test a candidate fix in a real environment with offline-Telegram + active Codex traffic.
Environment
- openclaw 2026.5.3-1 (also reproduces in 2026.5.4 per source-equivalent inspection)
- macOS 15.x (darwin 25.3.0)
- Node 22
- Channels enabled: telegram, whatsapp, discord
Summary
When
api.telegram.orgis unreachable (offline, regional block, blackholed route), the gateway emits 1,500+ENETUNREACHerrors per hour. Each Telegram fetch runs the 3-attempt cascade inextensions/telegram/src/fetch.ts(DNS-resolved → IPv4-only → hardcoded fallback IP149.154.167.220) without any global circuit-breaker state, so each long-poll cycle wastes three connection attempts and immediately re-fires.Multiplied across
getUpdatespolling and concurrent media downloads, this produces sustained event-loop starvation (eventLoopDelayMaxMs > 3000ms), which in turn causes the TUI streaming watchdog (dist/tui-…js, default ~30s) to falsely reportbackend may have dropped this run silently — send a new message to resyncon otherwise-healthy Codex/OpenAI streams.The streaming watchdog is doing its job — it's reporting that no frames arrived for 30s. The frames didn't arrive because the gateway loop was blocked by Telegram retries, not because the upstream model failed.
Repro
openclaw.jsonwith active long-poll (entries.telegram.enabled: true)149.154.167.220:443at the firewall) while leaving DNS resolution workingENETUNREACH 149.154.167.220:443(~25/min)eventLoopDelayMaxMs > 1000msdespite no real workloadopenai-codex/gpt-5.5) trip the streaming watchdog despite the upstream stream being aliveEvidence (one user's gateway, last 60 min)
ENETUNREACH 149.154.167.220:443errors in 60 min (~25/min)eventLoopDelayMaxMs = 9646.9, 3116.4, 2803.9, 1722.8across multiple windowsopenai-codex/gpt-5.5while OpenAI itself was healthy (verified byopenclaw agent --agent main --message "respond with PONG"returningPONGin the same time window from a separate shell)curl --max-time 5 https://149.154.167.220/returns code000(immediate failure), whilecurl https://api.telegram.org/returns302— DNS-based path works, the hardcoded fallback IP route does notSource references
dist/fetch-B1BHd80D.js:135—TELEGRAM_FALLBACK_IPS = [\"149.154.167.220\"](single hardcoded IP, no rotation, no health awareness)dist/fetch-B1BHd80D.js:383-426—createTelegramTransportAttempts: cascade builds 3 attempts per request with no per-host circuit-breaker statedist/fetch-B1BHd80D.js:149-155—FALLBACK_RETRY_ERROR_CODEScorrectly identifiesENETUNREACH/EHOSTUNREACHetc. as retry triggers, but the cascade still re-runs on every getUpdates cyclecircuitBreaker,markUnhealthy, orexponentialBackoff— the pattern is genuinely missingAdjacent issues (related, different scope)
These came up in search; this issue is differentiated by being about the inbound
getUpdatesfetch transport cascade, not outbound actions or webhooks:sendChatActioninfinite retry loop (outbound action; same family, different code path)sendMessagefailure (outbound)deleteWebhookENETUNREACH on startup (one-shot, not poll loop)Suggested fix (first-principles)
Add a per-host circuit breaker in the Telegram fetch dispatcher path (Hystrix/Polly pattern, Google SRE Ch. 22):
(host, error-class)keyOPEN: short-circuit further requests withhost_unhealthyfor cooldown THALF_OPEN: allow one probe; success →CLOSED+ reset, failure → back toOPENwith longer T (exponential, capped)CLOSEDon any successful responseDefaults that would match the telemetry above:
N=5,W=30s,T=10sinitial doubling to max60s. All configurable underchannels.telegram.transport.circuitBreaker.{ failureThreshold, window, cooldown, maxCooldown }.Pair with exponential backoff + jitter on the
getUpdatespolling loop itself:Suggested defaults:
BASE=500ms,MAX_DELAY=60s. Reset on first successful response. Jitter prevents thundering-herd on shared networks.The same component should be reusable by Discord (#77634), Slack (#58519), and future channels — strong fit for the framework in #41899.
Out of scope (cross-reference, would file separately if useful)
core-plugin-tools:8s+system-prompt:7s+stream-setup:7s) makes each turn vulnerable to short network blips during prep.core-plugin-toolshas no dependency onsession-resource-loader/system-promptand could parallelize. Plugin tool definitions are also invariant per gateway lifetime and would benefit from caching.Happy to test a candidate fix in a real environment with offline-Telegram + active Codex traffic.
Environment