Repository navigation
Bug: Codex event-stale watchdog (60s) kills GPT-5.6 reasoning streams before first content token — scaled TTFB fix (#64594) insufficient for extended pre-thinking #90449
Description
Bug Description
GPT-5.6 reasoning models (terra, sol) on the openai-codex provider are consistently killed by the stream-idle (event-gap) watchdog before they produce their first content SSE token. The reasoning/pre-thinking phase legitimately exceeds the 60s default HERMES_CODEX_EVENT_STALE_TIMEOUT_SECONDS for contexts in the 10k–50k token range.
The scaled TTFB watchdog from PR #64594 caught the no-byte case, but the event-stale watchdog still fires at 60s because the backend emits initial lifecycle events (response.created, response.in_progress) which arm the watchdog timer, then goes silent for >60s while the model reasons before emitting the first response.output_text.delta.
Observable Error
Codex stream produced no SSE events for 60s after first byte (threshold 60s, model=gpt-5.6-terra, context=~32,149 tokens). Killing connection so the retry loop can reconnect.
Log chain from a real session (3 retries, all killed by the same watchdog):
Codex stream produced no bytes within TTFB cutoff (120s > 120s)— TTFB: first byte never arrivedCodex stream produced no SSE events for 60s after first byte (threshold 60s, ...)— event-stale: backend emitted initial events, then >60s gap- Same as Support passing morph snapshot id #2 on retry Architecture planning #3
API call failed after 3 retries.
Root Cause
In agent/chat_completion_helpers.py:
- TTFB watchdog (lines 1645-1649): checks
_codex_stream_last_event_ts is None— this now scales with context size (60/120/180s) thanks to fix(codex): scale, don't disable, the no-byte TTFB watchdog for large requests (#61778) #64594. Catches sockets that never deliver a byte. - Event-stale watchdog (lines 1706-1709): checks
_codex_stream_last_event_ts is not None AND gap > _codex_idle_timeout— this fires when the backend DID send initial events (e.g.response.in_progress) but then goes quiet during LLM reasoning.
_codex_idle_timeout_default scales: 12s (<10k), 60s (10k–50k), 120s (50k–100k), 180s (>100k). For gateway/subagent sessions shipping ~15–25k tokens of tools + instructions, the default is 60s — but GPT-5.6 reasoning prefill can take 120–300s before emitting the first content delta.
In agent/codex_runtime.py line 1395, _codex_stream_last_event_ts is set on every SSE event, including early lifecycle frames. So the event-stale watchdog arms immediately when the backend sends response.created (within seconds), then fires at 60s while the model is still thinking.
Environment
- Hermes Agent: main branch at
8408edcfb5(post-v0.20.4, includes PR fix(codex): scale, don't disable, the no-byte TTFB watchdog for large requests (#61778) #64594 fix) - Provider:
openai-codex(ChatGPT OAuth,https://chatgpt.com/backend-api/codex) - Model:
gpt-5.6-terra/gpt-5.6-sol - Context: ~32,000 tokens (typical gateway session with tool schema)
- Platform: macOS (Apple Silicon)
Steps to Reproduce
- Configure
openai-codexprovider with any GPT-5.6 model (terra, sol). - Start a session with enough context to push estimated tokens above 10k (any gateway session with tools enabled qualifies).
- Send a query that triggers reasoning (non-trivial payload).
- Observe: connection accepted, initial SSE events arrive, then 60s of silence → watchdog kill → retry loop × 3 → total failure.
Workaround
export HERMES_CODEX_EVENT_STALE_TIMEOUT_SECONDS=300Raising the event-stale timeout to 300s (5 minutes) accommodates extended reasoning prefill. This should be set by the user until the default is raised or the watchdog is made reasoning-aware.
Expected Behavior
The event-stale watchdog should either:
- Scale with context like the TTFB watchdog — raise the 10k–50k idle timeout from 60s to 300–600s, matching what subscription-backed reasoning models actually need.
- Or use reasoning-effort metadata — if the model is a reasoning variant and
reasoning_effortis set, defer the event-stale watchdog for a longer period.
Relationship to Existing Issues
- [Bug]: Codex 0-event hang is undetectable for >10k-token requests — TTFB exemption + event-idle precondition leave only the wall-clock stale timeout, which loses the race to delegation child_timeout #61778 (closed by fix(codex): scale, don't disable, the no-byte TTFB watchdog for large requests (#61778) #64594): The TTFB watchdog was correctly scaled instead of disabled. This report is about the adjacent event-stale watchdog, which was not addressed by that PR.
- fix(codex): scale, don't disable, the no-byte TTFB watchdog for large requests (#61778) #64594: Scaled the no-byte TTFB watchdog; the event-stale watchdog retained its original 60/120/180 second ladder, which is insufficient for GPT-5.6 reasoning prefill.
Log Excerpts
2026-08-19 18:50:24,282 WARNING agent.chat_completion_helpers: Codex stream produced no SSE events for 60s after first byte (threshold 60s, model=gpt-5.6-sol, context=~32,149 tokens). Killing connection so the retry loop can reconnect.
2026-08-19 18:50:24,283 WARNING run_agent: OpenAI client aborted (codex_stream_idle_kill, ...)
2026-08-19 18:50:26,288 WARNING agent.conversation_loop: API call failed (attempt 2/3) error_type=TimeoutError ...
Activity
alt-glitch commentedon Aug 20, 2026
Related: #74054, #73645, and #78818 cover adjacent Codex watchdog policies. This report isolates the post-lifecycle-event pre-first-output gap, so it is a related mechanism rather than a duplicate.
tadeogutierrez commentedon Aug 29, 2026
Additional controlled and real-world evidence
I reproduced the reported Codex stalls both in a controlled A/B test and in a separate interactive session.
Environment
4209d371openai-codexgpt-5.6-solThe local carried commit is unrelated to Codex streaming.
Controlled A/B test
I ran six fresh sequential sessions with the exact same self-contained prompt, 16,522 input tokens, and no tool calls:
highmediumThe first
highrequest produced no SSE events for 60 seconds after the first byte at only approximately 20,335 context tokens. Hermes killed the connection, retried, and completed successfully; total provider latency was 89.5 seconds.All six runs completed successfully and produced comparable outputs. This small sample does not establish that high reasoning effort causes the stalls, but it shows that the event-idle condition is reproducible in a fresh, relatively small-context session and is not limited to oversized histories or tool payloads.
Separate interactive-session reproduction
A separate live session reproduced both watchdog modes within four minutes:
The same session therefore exhibited both a pre-first-byte stall and a post-first-byte SSE stall. Retries recovered both requests, but only after waiting for the complete watchdog interval. High prompt-cache coverage makes request upload or uncached prompt processing a less likely common explanation.
Broader log window
Across the available recent log window:
This evidence suggests two related failure modes along the same Codex streaming path. A reasoning-aware event-idle floor, treating raw stream bytes or SSE keepalive comments as activity, or another progress-aware distinction between long provider-side reasoning and a dead transport could reduce unnecessary reconnects and multi-minute waits.
PeaceMaker-best commentedon Sep 5, 2026
I am taking this. I reproduced the lifecycle-only pre-output gap on current main with a focused regression: response.created / response.in_progress arms the scaled event-stale watchdog before the first substantive delta, while the same watchdog still correctly kills a post-progress stall. I will keep the fix narrow: track substantive Codex progress separately from raw SSE activity, defer event-stale until progress begins, preserve TTFB and wall-clock hard bounds, and add behavior-contract coverage for both phases.
refactor(agent): trim the watchdog phase split to what the tests pin