Skip to content

fix: cool down unhealthy telegram transports#78097

Merged
obviyus merged 1 commit into
openclaw:mainfrom
bryce-d-greybeard:bryce/issue-77900-telegram-fetch-cooldown
May 10, 2026
Merged

fix: cool down unhealthy telegram transports#78097
obviyus merged 1 commit into
openclaw:mainfrom
bryce-d-greybeard:bryce/issue-77900-telegram-fetch-cooldown

Conversation

@bryce-d-greybeard

@bryce-d-greybeard bryce-d-greybeard commented May 5, 2026 •
edited by obviyus
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds a Telegram-local cooldown for repeatedly failing Bot API transport attempts.

The existing fallback cascade can become sticky on the pinned Telegram IP. If that route is blackholed, every poll cycle keeps paying another failed network attempt. This patch tracks each managed transport attempt, opens a short cooldown after repeated failures, and short-circuits that attempt until the cooldown expires.

Changes

  • Track per-attempt health inside resolveTelegramTransport().
  • After 5 fallback-eligible failures, mark the attempt unhealthy for 10s; repeated opens back off up to 60s.
  • Treat code-less TypeError: fetch failed as transport fallback-eligible, while preserving non-retry for unknown explicit codes such as ECONNRESET.
  • Do not charge caller-provided dispatchers against managed transport health.
  • Preserve fallback progression when earlier attempts are cooling down.

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: Telegram managed fetch no longer keeps hammering a sticky pinned Bot API fallback route when that route is unreachable.
  • Real environment tested: Blacksmith Testbox Linux, PR checkout b4299a6ffabbac99630c9f61a49e75866df3b5df, real HTTPS requests to api.telegram.org; only 149.154.167.220/32 was temporarily marked unreachable.
  • Exact steps or command run after this patch: pnpm crabbox:run -- --provider blacksmith-testbox --blacksmith-org openclaw --blacksmith-workflow .github/workflows/ci-check-testbox.yml --blacksmith-job check --blacksmith-ref main --idle-timeout 90m --ttl 90m --timing-json --shell -- '<temp route + resolveTelegramTransport proof script>'
  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):
Testbox: tbx_01kr82w5ayfjxd3230t1ew5x8w
local_proof_head=b4299a6ffabbac99630c9f61a49e75866df3b5df
curl_pinned_exit=7 remote_ip= http_code=000
curl_api_exit=0 remote_ip=149.154.166.110 http_code=302
proof_head=b4299a6ffabbac99630c9f61a49e75866df3b5df
attempts=3
attempt_2_pinned=149.154.167.220
force_fallback_1=true
force_fallback_2=true
[telegram] fetch fallback: DNS-resolved IP unreachable; trying alternative Telegram API IP (codes=none, reason=proof-pinned-ip)
request_1=error name=TypeError cause_code=none message=fetch failed elapsed_ms=10
request_2=error name=TypeError cause_code=none message=fetch failed elapsed_ms=1
request_3=error name=TypeError cause_code=none message=fetch failed elapsed_ms=1
request_4=error name=TypeError cause_code=none message=fetch failed elapsed_ms=0
request_5=error name=TypeError cause_code=none message=fetch failed elapsed_ms=2
[telegram] transport attempt marked temporarily unhealthy for 10000ms (codes=none)
request_6=ok status=200 elapsed_ms=66
blacksmith run summary sync=delegated command=39.231s total=41.798s exit=0
  • Observed result after fix: the pinned fallback route failed, OpenClaw marked that transport attempt temporarily unhealthy, and the next managed fetch skipped back to the normal Telegram API path and returned status=200 instead of continuing to hit 149.154.167.220.
  • What was not tested: a live long-polling bot session on the reporter's exact network; this proof isolates the same managed Telegram transport path with a controlled unreachable pinned route.

Testing

  • git diff --check origin/main...HEAD — passed.
  • bunx pnpm test extensions/telegram/src/fetch.test.ts -- --reporter=dot — passed, 36 tests.
  • Blacksmith Testbox route proof above — passed.

Fixes #77900

@openclaw-barnacle openclaw-barnacle Bot added channel: telegram Channel integration: telegram size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 5, 2026
@clawsweeper

clawsweeper Bot commented May 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
The branch adds per-attempt health/cooldown state in Telegram fetch transport, code-less fetch-failed fallback handling, focused tests, and a changelog entry.

Reproducibility: Do we have a high-confidence way to reproduce the issue? Yes at source level: current main still has a test proving the pinned fallback remains sticky after all attempts fail, and the linked report plus PR proof exercise the same blackholed Telegram route behavior.

Real behavior proof
Sufficient (terminal): Sufficient: the PR body contains after-fix Blacksmith Testbox terminal output showing a blackholed pinned Telegram route cooling down and a later request returning status=200.

Next step before merge
Human review is appropriate because the remaining blocker is exact-head CI/merge handling, not a narrow repairable code defect in this PR.

Security
Cleared: Cleared: the diff only changes Telegram runtime fallback logic, colocated tests, and the changelog; it adds no dependencies, workflows, secret handling, downloads, or package execution surface.

Review details

Best possible solution:

Land the Telegram-local cooldown after the unrelated CI failures are understood or cleared, and leave configurable cross-channel circuit-breaker/backoff work to a separate product discussion.

Do we have a high-confidence way to reproduce the issue?

Do we have a high-confidence way to reproduce the issue? Yes at source level: current main still has a test proving the pinned fallback remains sticky after all attempts fail, and the linked report plus PR proof exercise the same blackholed Telegram route behavior.

Is this the best way to solve the issue?

Is this the best way to solve the issue? Yes for this PR's scope: a Telegram-owned cooldown in resolveTelegramTransport() is the narrowest maintainable fix for the immediate sticky-route bug, while broader configurable circuit-breaker policy can remain separate.

What I checked:

  • Current main has the pinned Telegram fallback path without cooldown state: Current main defines the hardcoded pinned Bot API fallback IP and fallback retry classifier, and a search found no TELEGRAM_TRANSPORT_ATTEMPT* or unhealthy-attempt cooldown symbols on main. (extensions/telegram/src/fetch.ts:31, 4c1e6ba2f03b)
  • Current main source-level reproduction: The existing main test still asserts that after all fallback attempts fail, the next request reuses the pinned fallback dispatcher, matching the blackholed-route report. (extensions/telegram/src/fetch.test.ts:934, 4c1e6ba2f03b)
  • PR implementation adds bounded per-attempt cooldown state: The PR head adds failure threshold, initial/max cooldown constants, health records, failure recording, cooldown skipping, and primary probing while a sticky attempt cools down. (extensions/telegram/src/fetch.ts:45, 1e8ca266166b)
  • PR fallback loop preserves progression through cooling attempts: The PR records fallback-eligible failures, opens capped cooldowns, skips attempts still unhealthy, and records success to reset attempt health. (extensions/telegram/src/fetch.ts:666, 1e8ca266166b)
  • PR regression coverage targets the reported behavior: The added tests cover code-less TypeError: fetch failed fallback and repeated ENETUNREACH failures cooling down the sticky fallback while probing earlier attempts. (extensions/telegram/src/fetch.test.ts:962, 1e8ca266166b)
  • Real behavior proof is present on the current PR head: The PR body supplies Blacksmith Testbox terminal output for head 1e8ca266166beafd8ac79c22891df5a189235d56, showing the pinned route blackholed, the attempt marked unhealthy for 10000ms, and the next request returning status=200 via the normal API path. (1e8ca266166b)

Likely related people:

  • obviyus: GitHub commit history confirms recent merged work on extensions/telegram/src/fetch.ts and fetch.test.ts for unified fallback and sticky fallback recovery, and the current PR head commit is also authored by this login. (role: recent Telegram transport contributor; confidence: high; commits: e4825a0f9385, 252456e2f6f0, 175c42eacd7e; files: extensions/telegram/src/fetch.ts, extensions/telegram/src/fetch.test.ts)
  • steipete: GitHub commit history shows multiple recent changes on Telegram fallback, proxy, startup-control retry, docs, and related tests on the same runtime surface. (role: adjacent Telegram transport contributor; confidence: high; commits: e873c1e1f815, dc9f1b8525b1, 74a667f119cf; files: extensions/telegram/src/fetch.ts, extensions/telegram/src/fetch.test.ts, extensions/telegram/src/bot-core.ts)

Remaining risk / open question:

  • Exact PR head still has failing check-test-types and aggregate check jobs; the visible annotations point at extensions/xai/runtime-model-compat.test.ts and .github, so this needs normal CI triage before merge.
  • The supplied real behavior proof isolates the managed Telegram transport path with a controlled blackholed pinned route, but it is not a full live long-polling bot session on the reporter's exact network.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 4c1e6ba2f03b.

@obviyus
obviyus force-pushed the bryce/issue-77900-telegram-fetch-cooldown branch 2 times, most recently from c23f555 to ab8ef1c Compare May 10, 2026 04:18
@openclaw-barnacle openclaw-barnacle Bot added size: M proof: supplied External PR includes structured after-fix real behavior proof. and removed size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 10, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@obviyus
obviyus force-pushed the bryce/issue-77900-telegram-fetch-cooldown branch from ab8ef1c to 1e8ca26 Compare May 10, 2026 04:28
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@obviyus
obviyus force-pushed the bryce/issue-77900-telegram-fetch-cooldown branch from 1e8ca26 to b4299a6 Compare May 10, 2026 04:40
@obviyus
obviyus merged commit 1b38f80 into openclaw:main May 10, 2026
104 checks passed
@obviyus

obviyus commented May 10, 2026

Copy link
Copy Markdown
Contributor

Landed via rebase onto main.

  • Scoped tests: git diff --check origin/main...HEAD; bunx pnpm test extensions/telegram/src/fetch.test.ts -- --reporter=dot; Blacksmith Testbox route proof in PR body
  • Changelog: CHANGELOG.md updated
  • Land commit: b4299a6
  • Merge commit: 1b38f80

Thanks @bryce-d-greybeard!

@clawsweeper clawsweeper Bot mentioned this pull request Jul 8, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

channel: telegram Channel integration: telegram proof: supplied External PR includes structured after-fix real behavior proof. size: M

Projects

None yet

Milestone

No milestone

2 participants

@bryce-d-greybeard @obviyus