Repository navigation
Description
Summary
openai-codex / gpt-5.5 sometimes leaves the interactive TUI stuck on Working... with no streamed text, no tool call, and no visible error. The only way to recover is pressing Escape, which records an aborted assistant turn.
This has happened repeatedly over the last couple of days in normal interactive use.
Environment
- pi:
0.75.5 - Node:
v22.22.1 - Provider/model:
openai-codex/gpt-5.5 - Thinking level:
xhigh - No explicit
transportsetting in user settings - Default tools/extensions enabled
Observed behavior
When the issue occurs:
- TUI keeps showing
Working...for minutes. - No assistant text is streamed.
- No tool call is emitted.
- Pressing Escape aborts the turn.
- The saved session entry for that assistant turn has:
{
"role": "assistant",
"stopReason": "aborted",
"content": [],
"usage": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0,
"totalTokens": 0
}
}I saw this pattern multiple times, including cases where the previous turn had completed normally and the next user message then hung before any provider usage was recorded.
This looks different from a long reasoning turn: there is no usage, no partial reasoning/text, and no tool call.
Expected behavior
If the provider/transport stalls before the first event, pi should eventually surface a timeout/transport error or retry in a bounded way, instead of keeping the TUI in Working... indefinitely until the user manually aborts.
Suspected area
From the installed 0.75.5 package:
SettingsManager.getTransport()returns"auto"when no setting is present.- The docs/settings table says the default is
"sse", so there may also be a docs/runtime default mismatch. - For
openai-codex-responses,transport=autoattempts WebSocket first. retry.provider.timeoutMsappears to be passed intostreamSimple(), but theopenai-codex-responsesimplementation does not seem to apply it to the Codex fetch/WebSocket wait path in the same way as SDK-based providers.- The WebSocket event loop can wait for the first message/completion without an obvious idle timer.
So the likely failure mode is: Codex WebSocket/transport waits before the first event; no assistant event is produced; interactive UI keeps showing Working...; Escape finally records an aborted zero-usage turn.
Suggested fix / mitigation
Possible fixes:
- Add a hard idle timeout for
openai-codex-responsesWebSocket and SSE stream waits, especially before the first event. - Ensure
retry.provider.timeoutMsorhttpIdleTimeoutMsapplies consistently to this provider path. - If
autois intended as the runtime default, update docs; ifsseis intended, adjustSettingsManager.getTransport(). - Optionally show a clearer status/error when a provider turn has produced zero events for a long time.
Local workaround I am considering, but have not applied yet:
{
"transport": "sse",
"httpIdleTimeoutMs": 120000
}I can provide more sanitized session metadata if helpful, but I avoided attaching raw session logs because they contain private conversation/tool context.
Activity
github-actions commentedon May 24, 2026
This issue was auto-closed. All issues from new contributors are auto-closed by default.
Maintainers review auto-closed issues daily and reopen worthwhile ones. Issues that do not meet the quality bar in CONTRIBUTING.md will not be reopened or receive a reply.
If a maintainer replies
lgtmion one of your issues, your future issues will stay open. If a maintainer replieslgtm, your future issues and PRs will stay open.See CONTRIBUTING.md.
liushuaiiu commentedon May 24, 2026
Additional repro from the same environment/session.
It happened again after this issue was filed.
Environment is unchanged:
0.75.5v22.22.1openai-codex/gpt-5.5xhightransportsetting, so runtime default isautoTimeline:
Working...for about 2 minutes.Sanitized session evidence:
{ "role": "assistant", "content": [], "stopReason": "aborted", "usage": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0, "totalTokens": 0 } }The immediately following retry of essentially the same user prompt succeeded and emitted tool calls, so this was not caused by the prompt itself or a long-running tool.
This supports the narrower failure mode described above: a pre-stream / zero-event hang where no assistant event, tool call, or usage has been recorded yet. This seems different from the previously discussed
after_message_stream_startWebSocket close case, where retry may be unsafe because partial assistant state may already exist.For this zero-event case, a bounded idle timeout or visible transport error should be safe because no assistant content/tool call has entered the session state yet.
mitsuhiko commentedon May 24, 2026
Can you
/sharean effected session with us?liushuaiiu commentedon May 24, 2026
I cannot share the raw affected session because it contains private local/project context, but I created a sanitized excerpt from a real affected session:
https://gist.github.com/liushuaiiu/48f8b4b7a3298932e90f71ceb986cd83
What it includes:
README.md: timeline, environment, and sanitization notespi-issue-4945-affected-session.redacted.jsonl: redacted JSONL excerptpi-issue-4945-affected-session.redacted.html: redacted HTML exportThis excerpt preserves the relevant failure shape:
stopReason: "aborted", emptycontent, zero usageRedactions applied before upload:
<HOME>,<PI_AGENT_DIR>,<TMP_REPO>, etc.)mitsuhiko commentedon May 24, 2026
Did you validate that setting the mode to
sseresolves the issue for you? Would be good to understand better what is going on here.liushuaiiu commentedon May 24, 2026
Not yet, sorry.
I have not validated
sseyet. In the current setup/session state I cannot safely switch the transport/mode without disturbing the affected environment I am using to collect evidence, and I had intentionally kept the local config unchanged while waiting for upstream feedback.So the current repro data is still from the default/current transport path only.
I can run a separate clean test with
sseand report back whether it suppresses the zero-event / zero-usageWorking...hang, if that is the preferred next data point.mitsuhiko commentedon May 24, 2026
The issue is that I cannot reproduce this at all, so I'm entirely in the dark. Anything that can narrow this down would help.
liushuaiiu commentedon May 24, 2026
Additional narrowing data. I now have two related but distinct observations.
1. Long-context interactive sessions: the tools are usually not the slow part
In another affected interactive session, the config was again:
openai-codex/gpt-5.5xhighThe repeated ~5 minute stalls happened after tool results had already returned, before the next assistant action.
A representative example:
That is ~309s where no tool was running.
Similar gaps also occurred after other tool results, not only
recap_project:Tool durations in that session were much smaller:
So this part looks less like a specific Pi tool/local extension hanging, and more like model/provider waiting after tool completion. The pattern is amplified by
gpt-5.5+xhigh+ long context.2. Transport test: forcing
ssedid not eliminate pre-output stallsI also ran a separate clean print-mode batch test with a tiny prompt:
Environment:
0.75.5v22.22.1openai-codex/gpt-5.5xhighpi -ptimeout 150sDefault / auto transport
Command shape:
timeout 150s pi -p --model openai-codex/gpt-5.5 --thinking xhigh "请只回答 OK,不要使用工具。"Result:
Excerpt:
Explicit
sseI used a separate working directory with project settings:
{ "transport": "sse" }I also verified locally that Pi's settings manager sees
transport = ssefor that directory.Same command shape:
timeout 150s pi -p --model openai-codex/gpt-5.5 --thinking xhigh "请只回答 OK,不要使用工具。"Result:
Excerpt:
Caveat: this print-mode test is not exactly the same as the interactive
/sharefailure shape. The timed-out print-mode attempts did not create a completed session file, whereas the original interactive repro records an aborted assistant turn with empty content and zero usage after pressing Escape.Still, this suggests that simply forcing
transport: "sse"does not fully avoid the pre-output / pre-first-event stall in my environment.Current hypothesis
There may be two related symptoms:
xhighthinking can produce multi-minute model-side waits after tool results.sse.This makes a specific local Pi tool/extension hang less likely. It seems more like provider/model-side waiting or pre-first-event streaming behavior with
openai-codex/gpt-5.5, with Pi currently only showing genericWorking....Potentially useful Pi-side diagnostics would be to distinguish these states in logs/UI/session data:
mitsuhiko commentedon May 24, 2026
I'm wondering if you are connecting through some proxy that is misbehaving. Could you share more details how you connect?
liushuaiiu commentedon May 25, 2026
Thanks. Here are more details about my connection path:
This issue has started happening frequently in the last 3–4 days. There have been no major changes to my network environment during that period.
51616 commentedon May 25, 2026
I'm also facing the same problem running pi on my vps in Tokyo (non-aws local provider). Wonder if it has something to do with the region.
To add some anecdotes, the same problem occurs even gpt-5.5. with low reasoning or 5.3-codex-spark model. Using codex has been smooth in the same period that I've been having this problem in pi. Spawning a new pi instance while another pi instance is hanging does somewhat consistently avoid this problem on the new pi process.
Plus, the "hang" doesn't only happen during the zero-even phase. Many times pi hangs after the first few tool calls.
Edit2: another anecdote to add is that this also occurred with EU ip as well. By the time i'm posting this I no longer encounter the problem on new sessions. Old sessions with this "slow" behavior is still slow. I suspect that this has something to do with recent cache validation bug https://x.com/thsottiaux/status/2058280452851638313
Edit: the problem seems to go away now?
70 remaining items
Blue-B commentedon Jul 15, 2026
I am seeing a separate failure mode on the current release.
openai-codex/gpt-5.6-terraauto2026-07-15 05:11:19Zand2026-07-15 05:13:02Zchatgpt.com/backend-api/codex/responsesreturned the Cloudflare HTML 520 pagea1b63e92080180a1,a1b64591d9f188f8This was not a persistent local session failure. I copied the persisted session, kept its session ID, and sent a follow-up with a fresh npm-installed 0.80.7 using no extensions or settings. It completed in 8 seconds after the incident had cleared.
There is an independent report 41 minutes later of
Unexpected status 520on the same endpoint:https://x.com/JacobRothfield/status/2077268771547574412
Pi's Codex low-level retry list currently includes 429, 500, 502, 503, and 504, but not 520. Could 520 be handled as a bounded transient failure, perhaps one or two retries? That would not cure the origin problem, but it would stop a brief Cloudflare failure from ending the turn.
hcxcomeon commentedon Sep 7, 2026
I encountered the same problem
fix(ai): use HTTP timeout for Codex SSE headers
chigkim commentedon Sep 12, 2026
It seems it's broken again? Now it stalls pretty often.
jparradog commentedon Sep 17, 2026
Additional Pi 0.85.1 data point from WSL2 with Node.js 24.19.0, using
openai-codexwithgpt-6-astraandgpt-5.6-sol.A local session-log scan found 82
provider_transport_failurediagnostics across 64 sessions and 18 working directories:message_stream_start, 13 before it.configuredTransport: "auto",eventsEmitted: true, andphase: "after_message_stream_start".One long session at about 85% context displayed repeated
WebSocket errormessages, followed byAuto-compaction failed: Turn prefix summarization failed: This operation was aborted, thenOperation aborted. The terminal was hosted in Herdr, but the persisted diagnostics and stack originate in Pi'sopenai-codex-responsestransport. No proxy environment variables were configured.I also ran three small core-only checks with extensions, tools, skills, context files, and session storage disabled. All three returned
OKin 6 to 7 seconds, so I could not reproduce this with a small context. This is recurrence evidence, not a deterministic minimal reproduction. Large requests or long streams appear correlated, but the physical source of the disconnect remains unknown.One observability gap: the retry and compaction failures visible in the TUI were not persisted with a WebSocket close code or reason, which makes provider, network, and runtime failures difficult to distinguish.
AI-assisted diagnosis; evidence and wording reviewed before posting.
j57n-3co-x-5735 commentedon Sep 26, 2026
I was seeing repeated Codex connection failures in Pi, particularly during long conversations containing images. The recurring error was “Codex SSE response headers timed out after 20000ms.” This investigation and fix targeted my local v0.80.2 checkout, not necessarily current upstream; newer versions may already differ.
Investigation of the saved session diagnostics identified this sequence in two affected sessions:
Retries could then repeat the same failure because the request remained large and the session continued using the fallback connection.
A local fix removed that independent 20-second cutoff and made connection attempts and retries share the configured time budget. It also stopped repeatedly sending the same rejected WebSocket request, allowed WebSocket recovery after temporary failures, and prevented automatic replay after a response had already started. Images and conversation history were preserved rather than silently removed.
For anyone tracing this in the codebase, the useful starting points in that checkout are:
packages/ai/src/api/openai-codex-responses.ts: the oldDEFAULT_SSE_HEADER_TIMEOUT_MS = 20_000andcreateSSEHeaderTimeout()imposed a deadline independently of the suppliedtimeoutMs. The timer ran from immediately beforefetch()until response headers arrived—not during the streamed response body. The same adapter owns WebSocket-to-SSE fallback; look for close code 1009,recordWebSocketFailure, and the oldwebsocketSseFallbackSessionsset.packages/coding-agent/src/core/sdk.ts: the stream wrapper suppliestimeoutMsfrom caller options, provider retry settings, orhttpIdleTimeoutMs. This is why increasing the configured timeout alone did not fix the old adapter's separate timer.packages/coding-agent/src/core/agent-session.ts, particularly_isRetryableError: stopping fallback inside the provider was not sufficient; agent-level retries also needed to reject replay after output. The fix checks whether anyprovider_transport_failurediagnostic haseventsEmitted: true, since an earlier WebSocket failure can say false while a later SSE failure says true.A useful regression case is to delay mocked SSE headers beyond 20 seconds while supplying a longer timeout: the old implementation still times out at 20 seconds. Also test a pre-output WebSocket 1009 followed by an SSE failure after output, ensuring the agent does not replay the turn. The existing
packages/ai/test/openai-codex-stream.test.tsis a starting point for transport tests.After rebuilding the compiled
distfiles used by my launcher and using the patched version, I haven’t experienced any further issues.The premature timeout was reproduced locally, but the reason the HTTP response needed more than 20 seconds wasn’t established. Slow upload, network conditions, or backend latency remain possibilities. This appears to have resolved my case, but it doesn’t establish that every Codex connection failure has the same cause.
NachoVazquez commentedon Oct 6, 2026
Adding size data from Pi 1.0.4 (macOS arm64,
openai-codex/gpt-6.1-sol, ChatGPT OAuth, no proxy or gateway).Two failure shapes, both ending in a zero-usage error turn:
WebSocket errorbefore the stream starts, which surfaces asfetch failed. The diagnostics readconfiguredTransport: "auto",fallbackTransport: "sse",eventsEmitted: false,phase: "before_message_stream_start",requestBytes: 3514964. A second run failed the same way withrequestBytes: 146917, so small bodies fail on this transport too.upstream connect error or disconnect/reset before headers, withreset reason: remote connection failureand onceconnection timeout. CF-Ray location EZE.Under
transport: "auto"the Codex path uses WebSocket, and that frame is uncompressed JSON. The SSE path compresses the same body with zstd. I probed both with one small request. Withtransport: "sse"it posts tochatgpt.com/backend-api/codex/responseswithcontent-encoding: zstdand returns HTTP 200. Withtransport: "auto"thefetchfunction is never called at all, so nothing is compressed.On the plain HTTP provider (
openai,api.openai.com/v1/responses, same model, OAuth) I can reproduce a size boundary with a change that means nothing to the model: leading whitespace inside the JSON body, which parsers ignore.upstream connect errortextA 1.61 MB conversation failed. Compressing it losslessly recovered it: 1,611,984 bytes became 993,986 bytes with zstd and 861,554 bytes with gzip, both accepted with images and history intact.
This matches @j57n-3co-x-5735, who measured 19-24 MB requests with accumulated images and saw the WebSocket reject them as too large, and @jparradog, whose diagnostics ranged from 97,632 to 2,767,402 bytes with a median of 607,917.
Is the uncompressed WebSocket frame intentional, to match the official Codex client? If so, would you consider compressing the WS frame, or preferring SSE above a size threshold? I set
transport: "sse"locally, though this thread suggests SSE has its own regional problems.AI-assisted: an agent pulled these diagnostics from my sessions and ran the two transport probes. I reviewed the numbers and the conclusion before posting.
fix(ai): use HTTP timeout for Codex SSE headers
dpetrou-continua commentedon Oct 11, 2026
hey guys -- i've also been seeing this on Pi 1.1.0 with Codex/Sol on macOS. (tweeted about it a few days ago.) failures sometimes persist across turns, but restarting Pi with the same saved session restores operation without removing history. since it's 1.1.0, the SSE timeout fix is already present. i basically can't work anymore (maybe a good thing? :│. blocks unattended agents. happy to help debug.
petrroll commentedon Oct 19, 2026
@dpetrou-continua could you try to instrument your pi to add logging proxy and check what exactly those failures are for you? Is it the tcp connection just dying, explicit http status code, or something else?
Does it seem to be heavily correlated with request size (always fails bellow / always succeeds bellow)?
Does forcing sse vs wss help? Does it change anything? Is there a session that always fails? Or is it (high probability in some cases) random?
Happy to help with setting up the proxy if needed.
petrroll commentedon Oct 19, 2026
@NachoVazquez coudl you instrument your pi with http / tcp proxy and ask it to check few things:
How does the wss connect fails? Does it fail before upgrade? is it just tcp connection dies, is there http status code? (I know js standard wss api doesn't let you see that, but proxy should).
I'm a bit puzzled by this one. You mention http 503 (so likely status code was replied) but then the error message mentions error before headers but I'd expect that if you get http code back you'd also get response headers (or is this about sending headers)? Could you ask your pi to instrument itself to disambiguate what exactly fails here and what exactly the server returns?
If needed I can help with instrumented fork, but just asking it to instrument itself (unless of course it always fails, then it'd not work 😅) should be enough and might work better.
Do I get that you're effectively reporting that in addition to the two codex issues, there's also additional "standard openai / chatgpt http endpoint" where anything above 1mb uncompressed fails?
NachoVazquez commentedon Oct 20, 2026
@petrroll Here is my agent's assessment and test results.
I ran fresh checks with Pi 1.3.0 / Node 24.15.0 on macOS arm64, using direct TCP/TLS and Undici hooks. The prompt was synthetic, with no tools or saved conversation.
I could not reproduce the original pre-stream WebSocket failure. Both 296-byte and 1,049,600-byte application messages completed. TCP/TLS succeeded; Undici handled unsupported HTTP/2 extended CONNECT by opening the WebSocket over HTTP/1.1. Both negotiated
permessage-deflate. My earlier claim about uncompressed WebSocket traffic was too broad: application JSON size does not establish compressed wire size.The earlier 503 was a received HTTP response, with headers and a CF-Ray value. “Upstream connect error or disconnect/reset before headers” was text in its error body. That appears to describe a gateway's upstream connection; the failed hop remains unknown. Today's HTTP checks all returned 200, so I have no fresh failing response to trace.
Yes, the earlier measurements included
api.openai.com/v1/responseswith ChatGPT OAuth. A paired October 8 test also observed size-dependent 503s on the Codex HTTP endpoint. Today, Codex accepted uncompressed bodies of 1,047,552 and 1,049,600 bytes over both HTTP/1.1 and HTTP/2. The larger body also passed with zstd. Padding changed only whitespace outside the JSON value. I did not retest the separate API endpoint because its saved OAuth login expired. These results do not establish a current universal 1 MiB limit.I excluded one initial control because my instrumentation prevented HTTP/1 fallback. I corrected that before collecting the results above.
I also have a small local recovery patch working through Pi 1.3.0.
recordWebSocketFailure()puts the session into persistent SSE fallback even after streaming starts. My change is:The helper adds the session to the fallback set only when that flag is true, while retaining failure counts and accurate diagnostics. The interrupted response still fails; the next request can reconnect over WebSocket. Pre-stream fallback stays unchanged. The patch adds no replay of the interrupted response.
The 1.3.0 port passed 37 offline tests and actual bundled-CLI fixtures for disconnect, reconnect, cancellation, reuse, and cleanup, with zero HTTP attempts in those CLI fixtures. An earlier bounded live SDK test recovered after forced in-stream TCP termination. A longer Pi 1.1 pilot completed 242 model steps, recovered from two natural interruptions, and crossed compaction; its real traffic was not instrumented. The original network failure cause remains unresolved.
Would this narrower recovery change be useful upstream?
petrroll commentedon Oct 23, 2026
@NachoVazquez If I read that correctly it seems that it's fine now, no?
We can think about adding recovery, but the tl;dr is that the issue was likely temporal and we can't no longer repro it to know what to fix :).
As for the narrover recovery, feel free to link patch I'll think :)
NachoVazquez commentedon Oct 24, 2026
@petrroll Patch and regression test, based on
mainat136b87e8a7f5.The test closes the socket after text starts streaming. Stock Pi sends the next turn to HTTP. With the patch, that turn reconnects over WebSocket with the full history. The interrupted response still fails, without replay. Pre-stream fallback stays unchanged.
npm run checkand 41 focused tests pass. The new test fails on stock and when the recovery condition is inverted.The original network failures remain unexplained. This patch covers the persistent fallback after a disconnect.
This comment and patch were prepared by my coding agent.
marandaneto commentedon Oct 24, 2026
openai-codex with gpt-6.1-sol is giving some many
Error: no healthy upstream
Error: terminated
Error: upstream connect error or disconnect/reset before headers. reset reason: connection termination
and other variants