fix(tui_app_server): preserve transcript events under backpressure#15759
Merged
etraut-openai merged 5 commits intoopenai:mainfrom Mar 25, 2026
Merged
fix(tui_app_server): preserve transcript events under backpressure#15759etraut-openai merged 5 commits intoopenai:mainfrom
etraut-openai merged 5 commits intoopenai:mainfrom
Conversation
Widen the set of must-deliver notifications to include all transcript events (AgentMessageDelta, ItemCompleted, PlanDelta, reasoning deltas) so the TUI receives lossless assistant text even when the event queue is saturated. Extract backpressure forwarding into a standalone function for testability.
Add test helpers, expand the event_requires_delivery unit test to cover transcript variants, and add integration tests proving lossless delivery under backpressure for both in-process and remote transports.
Add rustdoc to server_notification_requires_delivery, ForwardEventResult, and forward_in_process_event explaining the lossless/best-effort event classification and blocking semantics.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the app-server-client event forwarding logic to ensure transcript-critical server notifications are preserved under consumer backpressure (instead of being dropped), preventing corrupted/incomplete incremental markdown rendering in the TUI.
Changes:
- Centralizes the “must-deliver” classification for
ServerNotificationintoserver_notification_requires_delivery, shared by both in-process and remote transports. - Refactors the in-process forwarding decision tree into
forward_in_process_event, ensuring lossless transcript/terminal events block onsend().awaitwhile best-effort events still usetry_send. - Adds/updates tests to validate ordering and preservation of transcript events under bounded-channel backpressure for both in-process and websocket (remote) paths.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
codex-rs/app-server-client/src/lib.rs |
Adds shared notification classification + extracts in-process forwarding logic to preserve transcript/terminal events under backpressure; expands/updates tests. |
codex-rs/app-server-client/src/remote.rs |
Switches remote delivery classification to the shared helper and adds a unit test for transcript + disconnect delivery requirements. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
etraut-openai
approved these changes
Mar 25, 2026
Collaborator
etraut-openai
left a comment
There was a problem hiding this comment.
Good analysis and fix.
Relax the remote websocket backpressure test so it asserts the lossless contract instead of a brittle fixed slot for `Lagged`. The test now accepts either a lag marker or the surviving best-effort `stdout-2` event while still requiring transcript and completion notifications to arrive in order. This matches the remote transport behavior more closely under CI scheduling variance, where websocket task timing can change whether a best-effort event is dropped before the consumer starts draining the channel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
When running codex with
-c features.tui_app_server=truewe see corruption when streaming large amounts of data. This PR marks other event types as critical by making them must-deliver.Problem
When the TUI consumer falls behind the app-server event stream, the bounded
mpscchannel fills up and the forwarding layer drops events viatry_send. Previously onlyTurnCompletedwas marked as must-deliver. Streamed assistant text (AgentMessageDelta) and the authoritative final item (ItemCompleted) were treated as droppable — the same as ephemeral command output deltas. Because the TUI renders markdown incrementally from these deltas, dropping any of them produces permanently corrupted or incomplete paragraphs that persist for the rest of the session.Mental model
The app-server event stream has two tiers of importance:
Lossless (transcript + terminal): Events that form the authoritative record of what the assistant said or that signal turn lifecycle transitions. Losing any of these corrupts the visible output or leaves surfaces waiting forever. These are:
AgentMessageDelta,PlanDelta,ReasoningSummaryTextDelta,ReasoningTextDelta,ItemCompleted, andTurnCompleted.Best-effort (everything else): Ephemeral status events like
CommandExecutionOutputDeltaand progress notifications. Dropping these under load causes cosmetic gaps but no permanent corruption.The forwarding layer uses
try_sendfor best-effort events (dropping on backpressure) and blockingsend().awaitfor lossless events (applying back-pressure to the producer until the consumer catches up).Non-goals
Tradeoffs
Blocking on transcript events means a slow consumer can now stall the producer for the duration of those events. This is acceptable because: (a) the alternative is permanently broken output, which is worse; (b) the consumer already had to keep up with
TurnCompletedblocking sends; and (c) transcript events arrive at model-output speed, not burst speed, so sustained saturation is unlikely in practice.Architecture
Two parallel changes, one per transport:
In-process path (
lib.rs): The inline forwarding logic was extracted intoforward_in_process_event, a standalone async function that encapsulates the lag-marker / must-deliver / try-send decision tree. The worker loop now delegates to it. A newserver_notification_requires_deliveryfunction (sharedpub(crate)) centralizes the notification classification.Remote path (
remote.rs): The localevent_requires_deliverynow delegates to the same sharedserver_notification_requires_delivery, keeping both transports in sync.Observability
No new metrics or log lines. The existing
warn!on event drops continues to fire for best-effort events. Lossless events that block will not produce a log line (they simply wait).Tests
event_requires_delivery_marks_transcript_and_terminal_events: unit test confirming the expanded classification coversAgentMessageDelta,ItemCompleted,TurnCompleted, and excludesCommandExecutionOutputDeltaandLagged.forward_in_process_event_preserves_transcript_notifications_under_backpressure: integration-style test that fills a capacity-1 channel, verifies a best-effort event is dropped (skipped count increments), then sends lossless transcript events and confirms they all arrive in order with the correct lag marker preceding them.remote_backpressure_preserves_transcript_notifications: end-to-end test over a real websocket that verifies the remote transport preserves transcript events under the same backpressure scenario.event_requires_delivery_marks_transcript_and_disconnect_events(remote): unit test confirming the remote-side classification covers transcript events andDisconnected.