Problem
When a session accumulates a very large transcript (e.g. 7,069 messages, 205 image references, 20MB on disk, 170k tokens), the gateway enters a death spiral:
- Incoming message triggers an embedded agent run
sanitizeSessionMessagesImages() iterates ALL messages, resizing images via Sharp — takes significant time on a large session
- The serialized transcript is sent to the LLM API
- The 120s timeout fires before the request completes (often during prep, not even network)
- Failover tries the next model — same timeout
- All 3 fallback models time out →
FailoverError: "All models failed (3): ... LLM request timed out"
- The error message is returned to the user
- The next incoming message triggers another run → goto step 1
This repeats indefinitely. In production, 217 consecutive timeout retries were observed over several hours before manual intervention (clearing the session store and restarting the gateway).
Root Cause
The error "LLM request timed out" does not match any of the existing auto-recovery checks:
isLikelyContextOverflowError() — no, it's a timeout not a context overflow API error
isCompactionFailureError() — no
isTransientHttpError() — no
- Session corruption / role ordering patterns — no
So the timeout falls through to the generic error path, which just returns a user-facing error message. Since each incoming message starts a fresh run against the same bloated session, the cycle never breaks.
Proposed Fix
In src/auto-reply/reply/agent-runner-execution.ts, detect when ALL models have timed out and auto-reset the session, similar to the existing compaction failure recovery.
The negative lookahead excludes rate-limit timeouts (which should be handled by cooldown logic, not session reset).
Impact
- Without fix: A single bloated session can render the bot completely unresponsive for hours, consuming resources on every retry
- With fix: The session auto-resets on the first "all timed out" failure, immediately recovering functionality
Problem
When a session accumulates a very large transcript (e.g. 7,069 messages, 205 image references, 20MB on disk, 170k tokens), the gateway enters a death spiral:
sanitizeSessionMessagesImages()iterates ALL messages, resizing images via Sharp — takes significant time on a large sessionFailoverError: "All models failed (3): ... LLM request timed out"This repeats indefinitely. In production, 217 consecutive timeout retries were observed over several hours before manual intervention (clearing the session store and restarting the gateway).
Root Cause
The error
"LLM request timed out"does not match any of the existing auto-recovery checks:isLikelyContextOverflowError()— no, it's a timeout not a context overflow API errorisCompactionFailureError()— noisTransientHttpError()— noSo the timeout falls through to the generic error path, which just returns a user-facing error message. Since each incoming message starts a fresh run against the same bloated session, the cycle never breaks.
Proposed Fix
In
src/auto-reply/reply/agent-runner-execution.ts, detect when ALL models have timed out and auto-reset the session, similar to the existing compaction failure recovery.The negative lookahead excludes rate-limit timeouts (which should be handled by cooldown logic, not session reset).
Impact