Skip to content

fix(telegram): cool down transient sendChatAction failures#55886

Closed
Boulea7 wants to merge 5 commits into
openclaw:mainfrom
Boulea7:fix/telegram-sendchataction-transient-cooldown
Closed

fix(telegram): cool down transient sendChatAction failures#55886
Boulea7 wants to merge 5 commits into
openclaw:mainfrom
Boulea7:fix/telegram-sendchataction-transient-cooldown

Conversation

@Boulea7

@Boulea7 Boulea7 commented Mar 27, 2026

Copy link
Copy Markdown

Summary

  • Problem: transient Telegram sendChatAction failures (network / 429 / temporary 5xx) can keep firing repeatedly, creating noisy retry loops and log spam.
  • Why it matters: typing indicators are best-effort, so repeated failures should not keep hammering Telegram or interfere with normal reply delivery.
  • What changed: kept the existing 401 suspend behavior, and added a separate transient cooldown path that temporarily suppresses repeated sendChatAction calls until a later recovery.
  • What did NOT change (scope boundary): this PR does not change typingMode config behavior, message delivery flow, or Telegram channel startup/runtime wiring outside the dedicated handler.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause / Regression History (if applicable)

  • Root cause: the dedicated Telegram sendChatAction handler only had global backoff / suspension behavior for 401s, but transient failures still retried on every later call path.
  • Missing detection / guardrail: the handler had no cooldown state for transient network / rate-limit / temporary server failures, and the dedicated test file did not currently lock in that behavior.
  • Prior context (git blame, prior PR, issue, or refactor if known): the current per-account guard already existed for 401 token failures in sendchataction-401-backoff.ts.
  • Why this regressed now: as more users hit Telegram transient failures, the handler protected the invalid-token case but not the best-effort transient case.
  • If unknown, what was ruled out: this PR does not treat typingMode config propagation as the root cause for this specific issue.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: extensions/telegram/src/sendchataction-401-backoff.test.ts
  • Scenario the test should lock in: transient failures enter cooldown, repeated calls are suppressed during cooldown, recovery clears transient state, and 401 suspension semantics still work independently.
  • Why this is the smallest reliable guardrail: the bug lives in the dedicated per-account handler state machine, so direct unit coverage is the narrowest reliable test.
  • Existing test that already covers this (if any): extensions/telegram/src/send.test.ts -t "sends typing" still covers the normal typing send seam.
  • If no new test is added, why not: N/A

User-visible / Behavior Changes

  • Repeated transient sendChatAction failures no longer keep retrying on every call path.
  • 401 invalid-token behavior is unchanged.
  • No config or default changes.

Diagram (if applicable)

Before:
transient sendChatAction failure -> next typing call retries immediately -> repeated failures/log spam

After:
transient sendChatAction failure -> cooldown starts -> repeated typing calls skipped -> later success clears cooldown

Security Impact (required)

  • New permissions/capabilities? (Yes/No) No
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) No
  • Data access scope changed? (Yes/No) No

Repro + Verification

Environment

  • OS: macOS
  • Runtime/container: local pnpm workspace
  • Model/provider: N/A
  • Integration/channel (if any): Telegram
  • Relevant config (redacted): existing Telegram channel config

Steps

  1. Simulate repeated transient sendChatAction failures.
  2. Trigger repeated handler calls during the cooldown window.
  3. Verify repeated calls are suppressed and later recovery clears the transient state.

Expected

  • transient failures do not suspend the handler permanently
  • repeated calls during cooldown do not keep calling Telegram
  • later success restores normal behavior
  • 401 behavior still suspends as before

Actual

  • matches expected in targeted local verification

Evidence

  • Failing test/log before + passing after

Human Verification (required)

  • Verified scenarios:
    • transient network failure cooldown
    • transient recovery path
    • 429 transient handling
    • 5xx transient handling
    • unexpected non-transient error still throws
    • 401 semantics remain active
    • mixed 401 -> transient -> 401 sequence preserves 401 counter
  • Edge cases checked:
    • cooldown suppression is global per handler
    • Date.now-driven cooldown expiry path
  • What you did not verify:
    • live Telegram outage against the real API
    • broader typingMode configuration behavior

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) No
  • Migration needed? (Yes/No) No

Risks and Mitigations

  • Risk: treating some Telegram-side failures as transient could suppress useful signal for a short period.
    • Mitigation: only network / 429 / temporary 5xx go through cooldown; unexpected non-transient errors still throw, and 401 still suspends.

🤖 AI-assisted (Codex), tested with focused local verification

@greptile-apps

greptile-apps Bot commented Mar 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a transient cooldown path to the sendChatAction handler, complementing the existing 401 suspension mechanism. When a network error, 429, or 5xx is caught, the handler silently resolves (typing indicators are best-effort), sets an exponentially-backed-off transientCooldownUntil timestamp, and suppresses subsequent calls until the cooldown expires. The existing 401 path is unchanged — it still accumulates, backs off with sleepWithAbort, and suspends after maxConsecutive401 failures. A 401 during an active transient cooldown correctly resets the transient state before incrementing the 401 counter.\n\nKey observations from the review:\n- State machine correctness: the ordering of guards (suspendedtransientCooldownUntil → 401 backoff sleep → try/catch) is logically sound with no unreachable or mis-ordered branches.\n- First-failure silent swallow is intentional: callers of sendChatAction receive Promise<void> and the PR explicitly scopes typing indicators as best-effort, so this is appropriate — but reviewers should be aware this is a behavior change from the pre-PR "always throw on non-401" contract.\n- Mock refactor: switching from importOriginal-spread to a full factory correctly includes the four infra-runtime helpers transitively required by network-errors.ts (collectErrorGraphCandidates, extractErrorCode, formatErrorMessage, readErrorName).\n- Minor style note: isRateLimitError uses string-matching while the rest of the error-detection layer uses typed error_code fields via hasTelegramErrorCode. Functionally equivalent for all realistic Telegram shapes but inconsistent with the existing pattern.

Confidence Score: 5/5

Safe to merge — the transient cooldown path is correctly implemented, all key scenarios are tested, and the 401 suspension contract is preserved.

No P0 or P1 issues found. The only comment is a P2 style suggestion about using structured error_code detection for 429 instead of string-matching, which is functionally equivalent for all realistic Telegram error shapes.

No files require special attention.

Important Files Changed

Filename Overview
extensions/telegram/src/sendchataction-401-backoff.ts Adds transient cooldown state machine alongside existing 401 suspension path; logic and guard ordering are correct.
extensions/telegram/src/sendchataction-401-backoff.test.ts Mock refactored to full factory covering all transitively-used infra-runtime exports; seven new test cases cover all key cooldown paths.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/telegram/src/sendchataction-401-backoff.ts
Line: 74-81

Comment:
**`isRateLimitError` duplicates structured detection already available**

`isRateLimitError` relies on string-matching (`JSON.stringify`) while `isTelegramServerError` (used right below it) uses the typed `error_code` field via `hasTelegramErrorCode`. A 429 response from Telegram carries `error_code: 429`, so it would be detected consistently if `isTransientSendChatActionError` used `hasTelegramErrorCode(error, (code) => code === 429)` (or a new exported `isTelegramRateLimitError`) instead of string-matching. The current approach works for every realistic Telegram error shape (since `JSON.stringify` will always include the number `429`), but introduces a style inconsistency that could drift if error shapes change.

```suggestion
function isRateLimitError(error: unknown): boolean {
  return hasTelegramErrorCode(error, (code) => code === 429);
}
```

> Note: `hasTelegramErrorCode` is a file-private helper in `network-errors.ts`. If you prefer to keep the string-fallback for non-structured errors (e.g. raw `Error("429 Too Many Requests")`), consider exporting a dedicated helper from `network-errors.ts` that handles both paths.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(telegram): cool down transient sendC..." | Re-trigger Greptile

Comment thread extensions/telegram/src/sendchataction-401-backoff.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78297b3386

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread extensions/telegram/src/sendchataction-401-backoff.ts Outdated
@Boulea7

Boulea7 commented Mar 27, 2026

Copy link
Copy Markdown
Author

Updated this PR again.

Latest follow-ups I addressed:

  • narrowed the test-only runtime overrides so the sendChatAction tests no longer replace the whole runtime module surface
  • switched transient 429 handling to structured error_code detection via isTelegramRateLimitError
  • changed transient sendChatAction failures to keep rejecting during cooldown, so the existing typing start guard can still count failures and trip during rate limits / outages instead of treating those starts as successful

Fresh local verification:

  • pnpm test -- extensions/telegram/src/sendchataction-401-backoff.test.ts
  • pnpm test -- extensions/telegram/src/network-errors.test.ts
  • pnpm test -- extensions/telegram/src/send.test.ts -t "sends typing"
  • pnpm lint:plugins:no-monolithic-plugin-sdk-entry-imports
  • pnpm test -- src/plugins/contracts/web-search-provider.contract.test.ts

I also cross-checked the earlier failing PR run against a failing main run (23655258874). A large part of the red CI surface already overlaps with main, including build-smoke, checks-fast-contracts-protocol, checks-fast-extensions, extension-fast-telegram, checks-node-channels-*, checks-node-test-*, and the Windows shards.

I’m keeping this PR scoped to the Telegram sendChatAction fix and the review follow-ups, and I’ll keep watching the new CI run.

@vincentkoc
vincentkoc force-pushed the fix/telegram-sendchataction-transient-cooldown branch from e902920 to c4fe168 Compare April 28, 2026 07:43
@vincentkoc

Copy link
Copy Markdown
Member

ProjectClownfish pushed a narrow repair to this branch so the original contributor path can stay canonical.

Source PR: #55886
Validation: pnpm test -- extensions/telegram/src/sendchataction-401-backoff.test.ts; pnpm test -- extensions/telegram/src/network-errors.test.ts; pnpm test -- extensions/telegram/src/send.test.ts -t "sends typing"; pnpm lint:plugins:no-monolithic-plugin-sdk-entry-imports; pnpm check:changed
Contributor credit is preserved in the branch history and PR context.

Copilot AI review requested due to automatic review settings April 28, 2026 07:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

@clawsweeper

clawsweeper Bot commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Keep open: current main still lacks the central transient Telegram sendChatAction cooldown, and this branch contains useful bug-fix work. It is not merge-ready because it conflicts with current main, narrows current rate-limit fallback behavior, edits release-owned CHANGELOG.md, and lacks real Telegram behavior proof.

Canonical path: Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review.

So I’m closing this here because the remaining work is already tracked in the canonical issue.

Review details

Best possible solution:

Close this stale PR. The latest review rated it F, the branch still lacks merge-ready proof, and there has been no human follow-up after the durable review.

Do we have a high-confidence way to reproduce the issue?

Yes from source inspection: current main rethrows non-401 transient sendChatAction failures and has no transient cooldown. I did not run a live Telegram outage reproduction in this read-only review.

Is this the best way to solve the issue?

No, not as submitted. The handler is the right layer for the cooldown, but the branch must preserve current rate-limit/coalescing behavior and provide real Telegram proof before it is the best mergeable fix.

Security review:

Security review cleared: No concrete security or supply-chain concern found; the diff changes Telegram error handling/tests and a changelog entry without new dependencies, permissions, secrets handling, or code execution surfaces.

AGENTS.md: found and applied where relevant.

What I checked:

  • stale F-rated PR: PR was opened 2026-03-27T16:01:13Z, is older than 60 days, and the latest review rated it F.
  • proof blocker: real behavior proof is mock_only and proof tier is F, so this branch is not merge-ready without contributor follow-up.
  • no human follow-up: live comments and timeline hydrated by apply contain no non-automation activity after the ClawSweeper review.

Likely related people:

  • vincentkoc: Recent current-main history touches the Telegram sendChatAction handler and network-error helpers, and the PR timeline shows this person force-pushed a narrow repair to the branch. (role: recent area contributor and branch repair owner; confidence: high; commits: 8c802aa68351, c4fe168624fc; files: extensions/telegram/src/sendchataction-401-backoff.ts, extensions/telegram/src/network-errors.ts)
  • chinar-amrutkar: Git history for the rate-limit retry_after helper points to retry-safe wrapped Telegram send failure work that this PR now narrows. (role: adjacent rate-limit contract contributor; confidence: medium; commits: 3f67581e50a3; files: extensions/telegram/src/network-errors.ts, extensions/telegram/src/network-errors.test.ts, extensions/telegram/src/send.ts)
  • scoootscooob: The sendChatAction handler history includes the commit that moved Telegram channel implementation into extensions, making this person a low-confidence adjacent routing candidate. (role: Telegram extension migration contributor; confidence: low; commits: e5bca0832fbd; files: extensions/telegram/src/sendchataction-401-backoff.ts)

Codex review notes: model internal, reasoning high; reviewed against db5e415888ae.

@vincentkoc vincentkoc added clawsweeper Tracked by ClawSweeper automation and removed clownfish:merge-ready labels Apr 28, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels May 19, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. label May 19, 2026
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 19, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. label May 19, 2026
@clawsweeper

clawsweeper Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 9, 2026
@openclaw-clownfish

Copy link
Copy Markdown
Contributor

Clownfish 🐠 reef update

Thanks for the work on this. Clownfish could not push to this branch with the permissions available, so it opened a narrow replacement PR to keep the fix swimming forward without losing the contributor trail. not your fault, just GitHub branch-permission tides.

Replacement PR: #93020
Source PR: #55886
This source PR remains open; the replacement is just the writable fix lane.
The original contribution stays credited in the replacement PR context.

fish notes: model gpt-5.5, reasoning xhigh; reviewed against 5b4b09f.

@vincentkoc vincentkoc added the clownfish Tracked by Clownfish automation label Jun 14, 2026
@vincentkoc

Copy link
Copy Markdown
Member

Clownfish 🐠 landed

Thanks for the report and context here. This is superseded by #93020, which has landed as the canonical Clownfish fix path for this cluster.

Closing this now that the validated fix has landed. If current main still shows a different path, reply here and we can reopen.

fish notes: model gpt-5.5, reasoning medium; reviewed against 6134c00.

@vincentkoc vincentkoc closed this Jun 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: telegram Channel integration: telegram clawsweeper Tracked by ClawSweeper automation clownfish Tracked by Clownfish automation merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M stale Marked as stale due to inactivity status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegram sendChatAction retry spam during transient failures

3 participants