Skip to content

openai-codex provider: refresh_token_reused loop causes severe gateway event-loop degradation #56960

Description

@rayrage213

Summary

When the openai-codex OAuth provider has an expired or reused refresh token, OpenClaw enters a persistent retry loop that injects ~120 failed HTTP cycles/hour directly into the gateway event loop. This caused multi-minute response latency, Telegram polling stalls (90–114s), and gateway heap growth to 5–6GB before OOM/restart.

Environment

  • OpenClaw: v2026.3.28
  • Platform: Linux/systemd (Debian 13)
  • Gateway flag: --max-old-space-size=8192
  • Provider: openai-codex OAuth (2 profiles configured)

Observed behavior

Gateway logs flood with:

[openai-codex] Token refresh failed: 401 refresh_token_reused
[xai-auth] bootstrap config fallback: no config-backed key found
  • Frequency: every 30–90 seconds (~120+ combined failure events/hour)
  • Each cycle generates: HTTP request → 401 parse → error handling → WS reconnect attempt
  • With a single-threaded Node.js event loop, this volume causes measurable stalls

Downstream effects confirmed:

  • lane wait exceeded up to 286 seconds with queueAhead=0 (nothing queued — pure event loop blockage)
  • Telegram getUpdates stalls of 90–114 seconds (gateway too busy to dispatch outbound HTTP)
  • Gateway RSS growing from ~500MB to 5–6GB over 30–90 minutes before timeout/restart
  • Local CLI (openclaw status) taking 10+ seconds to respond during bad periods

Additional issue: auto-detection of ~/.codex/ credentials

Even after removing both OAuth auth profiles from OpenClaw config, the xai-auth bootstrap noise continues because the gateway auto-detects and loads credentials from ~/.codex/ (Codex CLI auth store). There is no clean way to stop the bootstrap probing without removing the CLI auth entirely — which breaks ACP harness use.

Expected behavior

  1. Failed OAuth refresh should back off exponentially and stop retrying after N failures with the same error
  2. refresh_token_reused (401) is a permanent error — it should not be retried at all
  3. Provider bootstrap should not repeatedly probe when credentials are confirmed invalid
  4. Auto-detection of ~/.codex/ credentials should not trigger periodic bootstrap attempts when no key is configured in OpenClaw config

Workaround applied

  • Removed openai-codex from fallback chains
  • Removed both OAuth auth profiles from config
  • Moved all 10 background cron jobs using openai-codex/gpt-5.4 to Haiku/Sonnet
  • Switched to ACP harness (Codex CLI) for Codex access

Note: workaround stops the refresh_token_reused loop but does not fully silence xai-auth bootstrap noise (still fires every ~90s from CLI credential detection).

Impact

This caused a sustained multi-hour degradation of an entire Ray_OS agent stack — every agent session was affected. The root cause was only identified through a full forensic audit. A simple exponential backoff + permanent-error detection on refresh_token_reused would have prevented this entirely.

Metadata

Metadata

Assignees

Labels

close:duplicateClosed as duplicatededupe:childDuplicate issue/PR child in dedupe cluster

Type

No type

Fields

Priority

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions