Summary
When the openai-codex OAuth provider has an expired or reused refresh token, OpenClaw enters a persistent retry loop that injects ~120 failed HTTP cycles/hour directly into the gateway event loop. This caused multi-minute response latency, Telegram polling stalls (90–114s), and gateway heap growth to 5–6GB before OOM/restart.
Environment
- OpenClaw: v2026.3.28
- Platform: Linux/systemd (Debian 13)
- Gateway flag:
--max-old-space-size=8192
- Provider: openai-codex OAuth (2 profiles configured)
Observed behavior
Gateway logs flood with:
[openai-codex] Token refresh failed: 401 refresh_token_reused
[xai-auth] bootstrap config fallback: no config-backed key found
- Frequency: every 30–90 seconds (~120+ combined failure events/hour)
- Each cycle generates: HTTP request → 401 parse → error handling → WS reconnect attempt
- With a single-threaded Node.js event loop, this volume causes measurable stalls
Downstream effects confirmed:
lane wait exceeded up to 286 seconds with queueAhead=0 (nothing queued — pure event loop blockage)
- Telegram
getUpdates stalls of 90–114 seconds (gateway too busy to dispatch outbound HTTP)
- Gateway RSS growing from ~500MB to 5–6GB over 30–90 minutes before timeout/restart
- Local CLI (
openclaw status) taking 10+ seconds to respond during bad periods
Additional issue: auto-detection of ~/.codex/ credentials
Even after removing both OAuth auth profiles from OpenClaw config, the xai-auth bootstrap noise continues because the gateway auto-detects and loads credentials from ~/.codex/ (Codex CLI auth store). There is no clean way to stop the bootstrap probing without removing the CLI auth entirely — which breaks ACP harness use.
Expected behavior
- Failed OAuth refresh should back off exponentially and stop retrying after N failures with the same error
refresh_token_reused (401) is a permanent error — it should not be retried at all
- Provider bootstrap should not repeatedly probe when credentials are confirmed invalid
- Auto-detection of
~/.codex/ credentials should not trigger periodic bootstrap attempts when no key is configured in OpenClaw config
Workaround applied
- Removed
openai-codex from fallback chains
- Removed both OAuth auth profiles from config
- Moved all 10 background cron jobs using
openai-codex/gpt-5.4 to Haiku/Sonnet
- Switched to ACP harness (Codex CLI) for Codex access
Note: workaround stops the refresh_token_reused loop but does not fully silence xai-auth bootstrap noise (still fires every ~90s from CLI credential detection).
Impact
This caused a sustained multi-hour degradation of an entire Ray_OS agent stack — every agent session was affected. The root cause was only identified through a full forensic audit. A simple exponential backoff + permanent-error detection on refresh_token_reused would have prevented this entirely.
Summary
When the
openai-codexOAuth provider has an expired or reused refresh token, OpenClaw enters a persistent retry loop that injects ~120 failed HTTP cycles/hour directly into the gateway event loop. This caused multi-minute response latency, Telegram polling stalls (90–114s), and gateway heap growth to 5–6GB before OOM/restart.Environment
--max-old-space-size=8192Observed behavior
Gateway logs flood with:
Downstream effects confirmed:
lane wait exceededup to 286 seconds withqueueAhead=0(nothing queued — pure event loop blockage)getUpdatesstalls of 90–114 seconds (gateway too busy to dispatch outbound HTTP)openclaw status) taking 10+ seconds to respond during bad periodsAdditional issue: auto-detection of ~/.codex/ credentials
Even after removing both OAuth auth profiles from OpenClaw config, the
xai-authbootstrap noise continues because the gateway auto-detects and loads credentials from~/.codex/(Codex CLI auth store). There is no clean way to stop the bootstrap probing without removing the CLI auth entirely — which breaks ACP harness use.Expected behavior
refresh_token_reused(401) is a permanent error — it should not be retried at all~/.codex/credentials should not trigger periodic bootstrap attempts when no key is configured in OpenClaw configWorkaround applied
openai-codexfrom fallback chainsopenai-codex/gpt-5.4to Haiku/SonnetNote: workaround stops the
refresh_token_reusedloop but does not fully silencexai-authbootstrap noise (still fires every ~90s from CLI credential detection).Impact
This caused a sustained multi-hour degradation of an entire Ray_OS agent stack — every agent session was affected. The root cause was only identified through a full forensic audit. A simple exponential backoff + permanent-error detection on
refresh_token_reusedwould have prevented this entirely.