fix(auto-reply): auto-retry transient edge-block exhaustion + honest busy message#28
Conversation
…busy message
When the model fallback chain exhausts on a pure-transient condition (every
hop rate_limit/overloaded) with no cooldown expiry or usage-limit detail, the
cause is almost always an upstream gateway/edge block — e.g. a Cloudflare/Render
HTML 429 that 429s every hop because they share one upstream host. Model-level
fallback can't help, and the old terminal message ("All models are temporarily
rate-limited") was both inaccurate (the models were never rate-limited) and
premature (a brief edge blip became a failed conversation).
- Retry the full primary→fallback chain up to 3x with exponential backoff
(2s/4s/8s) on an *unexplained* pure-transient exhaustion before surfacing
anything to the user. Extends the existing single transient-HTTP retry.
- Gate excludes a known cooldown expiry and Codex/usage-limit messages so their
actionable detail still surfaces immediately rather than being retried away.
- Replace the terminal copy with honest, non-revealing wording that signals the
automatic retry: "The AI service is briefly busy — retrying automatically…".
Deliberately names no provider/edge/cause.
Classification is unchanged (HTML-429 still classifies as rate_limit, per
upstream openclaw#67517) — this only changes retry behavior and user-facing copy.
Tests: message copy (both paths), backoff helper, retry-then-recover,
exhaust-then-friendly-message (asserts no cause leakage), and no-retry-when-
cooldown-known. Full file 107/107; format/lint/tsgo core+test all green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CI statusAll checks this change can influence are green (67 passing): every Two red checks are pre-existing/environmental, not caused by this 2-file diff:
Local verification on the diff: |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
![gandalfboon[bot]](https://avatars.githubusercontent.com/in/3087119?s=60&v=4){kind=small}
There was a problem hiding this comment.
Approved — clean, well-scoped fix.
The core change is correct: isUnexplainedTransientEdgeBlock narrows precisely to pure-transient chain exhaustion with no cooldown expiry and no Codex/usage-limit detail — i.e. the upstream gateway/edge HTML-429 shape (every same-host hop 429s) where a blind chain retry can genuinely ride out the throttle window. Known cooldowns and usage-limits are deliberately excluded so their actionable countdown still surfaces immediately instead of being retried away. Placement at the tail of the error loop (last guard before terminal classification) with a bounded 3× 2s/4s/8s backoff via continue is right.
Honest copy fix is a real improvement — the old 'All models are temporarily rate-limited' was inaccurate (the models were never rate-limited) and the new wording leaks no provider/edge/cause.
Tests are thorough: retry-then-recover, exhaust-then-friendly with explicit no-leak assertions (rate-limit/429/cloudflare/render/gateway/waf/boon-llm-gateway all negated), no-retry-when-cooldown-known, and the backoff/cap math. No secrets, no migrations, no breaking callers.
Non-blocking note for follow-up: 2+4+8s = up to ~14s added latency before the user sees anything on a sustained outage — acceptable tradeoff that the PR description owns, and the auto-retry copy signals it. Ship it.
…ries
The initial 2s/4s/8s schedule was wrong for an edge rate-limiter / WAF
(Cloudflare in front of the gateway), the dominant trigger for pure-transient
chain exhaustion:
- Retrying ~2s later almost always lands inside the same rate-limit window,
and each retry re-feeds the very counter that is blocking us — fast retries
can *prolong* the block rather than ride it out.
- A fleet-wide fixed schedule synchronizes retries against one shared gateway
host (thundering herd), which itself keeps the limiter tripped.
Redesign:
- MAX_TRANSIENT_BUSY_RETRIES 3 → 5.
- Base 5s, double per attempt, cap 60s/step.
- Equal jitter (AWS "equal jitter": half fixed + half random). The fixed half
guarantees a retry never fires too soon (no early re-trip); the random half
de-synchronizes concurrent sessions sharing the edge.
- rng is injectable for deterministic tests.
Recovery window grows from ~14s to ~67–135s — long enough to outlast a short
WAF/edge block instead of failing fast.
Tests: backoff bounds (un-jittered cap, min-jitter floor, [half,full] envelope),
retry count = 5. Full file 109/109; format/lint/tsgo core+test green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
…st) (#29) pnpm-audit-prod (CI "security-fast") flagged 17 high/critical advisories in production deps. This clears 16, including the lone CRITICAL. Transitive bumps via pnpm-workspace overrides (conservative, in-range, no major): @grpc/grpc-js 1.14.4 · linkify-it 5.0.1 · undici 7.28.0 · vite 8.0.16 · ws 8.21.0 · @vitest/browser 4.1.9 · hono 4.12.18→4.12.27 · form-data 2.5.4→2.5.6 · protobufjs 7.5.6→7.6.4 baileys 7.0.0-rc11 → rc13 (CRITICAL GHSA-qvv5-jq5g-4cgg): - bumped in extensions/whatsapp + release-age exclude - DROPPED patches/[email protected] — rc13 ships all three fixes natively (sequential-await encrypted-stream finish + dispatcher guard) - taught scripts/postinstall-bundled-plugins.mjs to recognize rc13's native dispatcher guard (hoisted `const dispatcher = … ? agent : undefined`) so the runtime hotfix no-ops instead of failing with "unexpected_content" Deferred (1 remaining HIGH): @earendil-works/pi-coding-agent (GHSA-jfgx-wxx8-mp94, LPE on shared Linux). Fix needs pi-* 0.75→0.78+, a BREAKING pi-ai API change (streamSimple/completeSimple/getApiProvider/complete removed) — a separate migration, not a security bump. Trial hosts are single-tenant, so the shared-host LPE is low-risk in our deployment. Verify: pnpm-audit-prod 17→1 · tsgo:core clean · postinstall hotfix no-ops · oxlint clean. Co-authored-by: Claude Opus 4.8 <[email protected]>
Dependency Changes DetectedThis PR changes dependency-related files. Maintainers should confirm these changes are intentional. Changed files:
Maintainer follow-up:
|
2 participants
Summary
Fixes the user-facing symptom behind the arguijo incident (2026-06-26): the agent replied "⚠️ All models are temporarily rate-limited. Please try again in a few minutes." when the real cause was an upstream Cloudflare/Render edge 429 (HTML "Blocked" page) in front of
boon-llm-gateway, not any LLM rate-limit.Because arguijo routes every model through one gateway host, the edge 429 hit every fallback hop →
chain_exhausted→ the misleading terminal message. The models were never rate-limited, and a brief edge blip became a failed conversation.What changed
rate_limit/overloaded, no cooldown expiry, no usage-limit detail), retry the full primary→fallback chain up to 3× with 2s/4s/8s backoff before surfacing anything. Extends the existing single transient-HTTP retry pattern inrunAgentTurnWithFallback.Classification is intentionally unchanged — an HTML-body 429 still classifies as
rate_limit(per upstream openclaw#67517). This PR changes only retry behavior and user-facing copy, so the fork stays aligned with upstream classification.Why not "fix the gateway / raise capacity"
Investigation ruled both out: gateway Go error paths are JSON-only (the HTML came from the edge in front of the process), Render app CPU/mem were flat ~0% during the window (not a capacity event), and arguijo sent only 67 requests all day from one fixed IP (can't have tripped a rate rule). The block originated at Cloudflare's view of Render's shared edge / AWS-EIP reputation — external to both the box and gateway compute. The durable in-our-control fix is this client-side resilience.
Tests
New tests in
agent-runner-execution.test.ts:resolveTransientRetryBackoffMs(2s/4s/8s) +MAX_TRANSIENT_BUSY_RETRIES === 3kind: success, 3 chain invocations, user sees no errorVerification
agent-runner-execution.test.ts: 107/107 passoxfmt --check,oxlint(core): cleantsgo:core+tsgo:test:src: clean (rc=0)RCA: https://getboon.slite.com/app/docs/iki4aP3u37Xvot
🤖 Generated with Claude Code
Summary by cubic
Adds automatic retries for transient edge‑block (HTML 429) chain exhaustions and replaces the misleading rate‑limit copy with a truthful “briefly busy” notice to prevent failed turns during short gateway edge blocks.
Bug Fixes
Dependencies
baileysto7.0.0-rc13and remove the rc11 patch; teachscripts/postinstall-bundled-plugins.mjsto detect rc13’s native dispatcher guard so the hotfix no‑ops.hono4.12.27,form-data2.5.6,protobufjs7.6.4, plus@grpc/grpc-js1.14.4,linkify-it5.0.1,undici7.28.0,vite8.0.16,ws8.21.0,@vitest/browser4.1.9.Written for commit 1e12efd. Summary will update on new commits.