Skip to content

[Bug] Telegram fetch transport lacks circuit breaker/backoff: 1500+ ENETUNREACH/hour during offline window starves event loop and trips TUI stream watchdog #77900

Description

@artyomx33

Summary

When api.telegram.org is unreachable (offline, regional block, blackholed route), the gateway emits 1,500+ ENETUNREACH errors per hour. Each Telegram fetch runs the 3-attempt cascade in extensions/telegram/src/fetch.ts (DNS-resolved → IPv4-only → hardcoded fallback IP 149.154.167.220) without any global circuit-breaker state, so each long-poll cycle wastes three connection attempts and immediately re-fires.

Multiplied across getUpdates polling and concurrent media downloads, this produces sustained event-loop starvation (eventLoopDelayMaxMs > 3000ms), which in turn causes the TUI streaming watchdog (dist/tui-…js, default ~30s) to falsely report backend may have dropped this run silently — send a new message to resync on otherwise-healthy Codex/OpenAI streams.

The streaming watchdog is doing its job — it's reporting that no frames arrived for 30s. The frames didn't arrive because the gateway loop was blocked by Telegram retries, not because the upstream model failed.

Repro

  1. Have Telegram channel enabled in openclaw.json with active long-poll (entries.telegram.enabled: true)
  2. Disconnect WiFi (or block 149.154.167.220:443 at the firewall) while leaving DNS resolution working
  3. Within ~5 minutes, gateway logs fill with ENETUNREACH 149.154.167.220:443 (~25/min)
  4. After ~10–15 minutes, liveness warnings appear with eventLoopDelayMaxMs > 1000ms despite no real workload
  5. TUI sessions running on healthy providers (e.g. openai-codex/gpt-5.5) trip the streaming watchdog despite the upstream stream being alive

Evidence (one user's gateway, last 60 min)

  • 1,535 ENETUNREACH 149.154.167.220:443 errors in 60 min (~25/min)
  • eventLoopDelayMaxMs = 9646.9, 3116.4, 2803.9, 1722.8 across multiple windows
  • TUI watchdog fired on a session backed by openai-codex/gpt-5.5 while OpenAI itself was healthy (verified by openclaw agent --agent main --message "respond with PONG" returning PONG in the same time window from a separate shell)
  • Standalone curl --max-time 5 https://149.154.167.220/ returns code 000 (immediate failure), while curl https://api.telegram.org/ returns 302 — DNS-based path works, the hardcoded fallback IP route does not

Source references

  • dist/fetch-B1BHd80D.js:135TELEGRAM_FALLBACK_IPS = [\"149.154.167.220\"] (single hardcoded IP, no rotation, no health awareness)
  • dist/fetch-B1BHd80D.js:383-426createTelegramTransportAttempts: cascade builds 3 attempts per request with no per-host circuit-breaker state
  • dist/fetch-B1BHd80D.js:149-155FALLBACK_RETRY_ERROR_CODES correctly identifies ENETUNREACH / EHOSTUNREACH etc. as retry triggers, but the cascade still re-runs on every getUpdates cycle
  • No matches in the codebase for circuitBreaker, markUnhealthy, or exponentialBackoff — the pattern is genuinely missing

Adjacent issues (related, different scope)

These came up in search; this issue is differentiated by being about the inbound getUpdates fetch transport cascade, not outbound actions or webhooks:

Suggested fix (first-principles)

Add a per-host circuit breaker in the Telegram fetch dispatcher path (Hystrix/Polly pattern, Google SRE Ch. 22):

  1. Track consecutive transient failures per (host, error-class) key
  2. After N consecutive failures within window W → state=OPEN: short-circuit further requests with host_unhealthy for cooldown T
  3. After T → state=HALF_OPEN: allow one probe; success → CLOSED + reset, failure → back to OPEN with longer T (exponential, capped)
  4. Reset to CLOSED on any successful response

Defaults that would match the telemetry above: N=5, W=30s, T=10s initial doubling to max 60s. All configurable under channels.telegram.transport.circuitBreaker.{ failureThreshold, window, cooldown, maxCooldown }.

Pair with exponential backoff + jitter on the getUpdates polling loop itself:

delay_ms = min(MAX_DELAY, BASE * 2^consecutive_failures)
delay_ms += random_jitter(0, delay_ms / 2)

Suggested defaults: BASE=500ms, MAX_DELAY=60s. Reset on first successful response. Jitter prevents thundering-herd on shared networks.

The same component should be reusable by Discord (#77634), Slack (#58519), and future channels — strong fit for the framework in #41899.

Out of scope (cross-reference, would file separately if useful)

Happy to test a candidate fix in a real environment with offline-Telegram + active Codex traffic.

Environment

  • openclaw 2026.5.3-1 (also reproduces in 2026.5.4 per source-equivalent inspection)
  • macOS 15.x (darwin 25.3.0)
  • Node 22
  • Channels enabled: telegram, whatsapp, discord

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions