Skip to content

Commit 16bdad1

Browse files
feat(responses): add WebSocket lane overflow isolation (#4225)
A slow WebSocket lane currently closes the managed session when its event queue fills. Add `isolate_lane_overflow=True` to the sync and async session constructors. With this option, healthy lanes keep using the same connection. The default remains `False`. Failed lanes discard queued events and collected response state. Reads keep raising the first buffer error. Their IDs stay reserved until reconnect. The guide explains how the session frees space when its queues fill. It also explains which events become unavailable when the default lane fails. Physical connection failures still end the session. This uses the existing router and reader, without new limits or dependencies. Validation: all 200 focused session cases pass on Python 3.10/Pydantic 1 and Python 3.13/Pydantic 2. They cover both policies, all queue budgets, eviction, cached finals, default routing, transport failure and reconnect. Pinned Ruff, format and ty checks pass. Co-authored-by: markstuart-oai <[email protected]>
1 parent 48cdad3 commit 16bdad1

4 files changed

Lines changed: 419 additions & 15 deletions

File tree

‎src/openai/lib/responses_websocket/README.md‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -207,14 +207,32 @@ it. After reconnect releases a closed default lane, `session.lane()` (or
207207
`session.lane(None)`) registers a fresh default and updates `session.default`.
208208

209209
Limits apply only to this opt-in session. Event queues are bounded per lane and
210-
across the session; overflow reports `ResponsesWebSocketBufferError` and closes
210+
across the session. By default, overflow reports `ResponsesWebSocketBufferError` and closes
211211
the owned connection so ordered events are not silently dropped. Queue byte
212212
counts measure received message bytes before parsing, not Python object overhead or
213213
transport decoding memory. Response accumulation is uncapped by default:
214214
omit `max_response_bytes` or set it to `None`. A positive integer opts into a
215215
budget for the cumulative compatible event bytes passed to an accumulator;
216216
exceeding it disables that response's collection, while raw events and other
217217
lanes remain usable. Queue and lane limits still require positive integers.
218+
219+
To keep healthy lanes usable when another lane falls behind, pass
220+
`isolate_lane_overflow=True` to either session constructor. The default is `False`.
221+
An overflowing lane loses its queued events and response collection; its reads
222+
and final-response collection raise the same first `ResponsesWebSocketBufferError`.
223+
Later events for that failed ID are dropped, and its reservation lasts until
224+
reconnect. Manual lane detachment and unregistered-stream inspection keep their
225+
existing default-lane behavior.
226+
227+
For aggregate queue pressure, the session fails the largest existing backlog
228+
in the pressured unit (bytes first, otherwise events), excluding the incoming
229+
event. Registration order breaks ties. An individually oversized event fails
230+
its target lane. The default lane can also fail under this policy: subsequent
231+
default events, including API error/control events, are then unavailable, while
232+
registered named lanes remain usable. Physical connection failures and explicit
233+
session close still end the session. After reconnect, register a new default
234+
explicitly with `session.lane()` if the old default failed.
235+
218236
Choose budgets for large legitimate outputs. These are application budgets,
219237
not protocol limits.
220238
The examples disable the transport's separate 1 MiB message limit with

‎src/openai/lib/responses_websocket/_session.py‎

Lines changed: 71 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,7 @@ def __init__(self, stream_id: str | None, limit: int | None) -> None:
157157
self.queue: deque[tuple[ResponsesServerEvent, int]] = deque()
158158
self.size = 0
159159
self.closed = False
160+
self.failure: ResponsesWebSocketBufferError | None = None
160161
self.in_flight = False
161162
self.closed_connection: object | None = None
162163
self.received_generation = 0
@@ -167,10 +168,15 @@ def __init__(self, stream_id: str | None, limit: int | None) -> None:
167168

168169
class _Router:
169170
def __init__(
170-
self, limits: ResponsesWebSocketLimits, connection: ResponsesConnection | AsyncResponsesConnection
171+
self,
172+
limits: ResponsesWebSocketLimits,
173+
connection: ResponsesConnection | AsyncResponsesConnection,
174+
*,
175+
isolate_lane_overflow: bool = False,
171176
) -> None:
172177
self.connection = connection
173178
self.limits = limits
179+
self.isolate_lane_overflow = isolate_lane_overflow
174180
self.condition = threading.Condition()
175181
self.lanes: dict[str | None, _LaneState] = {}
176182
self.events = 0
@@ -213,6 +219,11 @@ def detach(self, lane: _LaneState) -> None:
213219
lane.wake.set()
214220
self.condition.notify_all()
215221

222+
def _fail(self, lane: _LaneState) -> None:
223+
if not lane.closed:
224+
lane.failure = ResponsesWebSocketBufferError("The lane exceeded its buffered event limits")
225+
self.detach(lane)
226+
216227
def reconnected(self, connection: object) -> None:
217228
with self.condition:
218229
for stream_id, lane in list(self.lanes.items()):
@@ -227,11 +238,21 @@ def dispatch(self, event: ResponsesServerEvent, size: int) -> None:
227238
return
228239
stream_id = _field(event, "stream_id")
229240
lane = self.lanes.get(stream_id, self.default)
241+
if lane.failure is not None:
242+
return
230243
if lane.closed:
231244
lane = self.default
232245
if lane.closed:
233246
return
234-
if (
247+
if self.isolate_lane_overflow:
248+
if (
249+
len(lane.queue) >= self.limits.max_events_per_lane
250+
or lane.size + size > self.limits.max_bytes_per_lane
251+
or size > self.limits.max_bytes
252+
):
253+
self._fail(lane)
254+
return
255+
elif (
235256
len(lane.queue) >= self.limits.max_events_per_lane
236257
or self.events >= self.limits.max_events
237258
or lane.size + size > self.limits.max_bytes_per_lane
@@ -245,6 +266,18 @@ def dispatch(self, event: ResponsesServerEvent, size: int) -> None:
245266
lane.size += size
246267
self.size += size
247268
self.events += 1
269+
if self.isolate_lane_overflow:
270+
while self.events > self.limits.max_events or self.size > self.limits.max_bytes:
271+
byte_pressure = self.size > self.limits.max_bytes
272+
largest, largest_size = lane, 0
273+
for candidate in self.lanes.values():
274+
# Exclude this incoming event once; ties follow registration order.
275+
backlog = candidate.size if byte_pressure else len(candidate.queue)
276+
if candidate is lane:
277+
backlog -= size if byte_pressure else 1
278+
if backlog > largest_size:
279+
largest, largest_size = candidate, backlog
280+
self._fail(largest)
248281
lane.wake.set()
249282
self.condition.notify_all()
250283

@@ -259,6 +292,8 @@ def finish(self, error: BaseException | None = None) -> None:
259292

260293
def pop(self, lane: _LaneState) -> ResponsesServerEvent | None:
261294
# The caller holds the condition. No await occurs after dequeueing.
295+
if lane.failure is not None:
296+
raise lane.failure
262297
if lane.closed:
263298
raise EOFError("The WebSocket lane is detached")
264299
if lane.queue:
@@ -353,11 +388,17 @@ def get_final_response(self, *, timeout: float | None = None) -> Response:
353388
remaining = None if deadline is None else max(0.0, deadline - time.monotonic())
354389
self.recv(timeout=remaining)
355390
while self._state.accumulator.final is None:
356-
if self._state.accumulator.error is not None:
357-
raise self._state.accumulator.error
391+
with self._session._router.condition:
392+
if self._state.failure is not None:
393+
raise self._state.failure
394+
if self._state.accumulator.error is not None:
395+
raise self._state.accumulator.error
358396
remaining = None if deadline is None else max(0.0, deadline - time.monotonic())
359397
self.recv(timeout=remaining)
360-
return self._state.accumulator.final
398+
with self._session._router.condition:
399+
if self._state.failure is not None:
400+
raise self._state.failure
401+
return self._state.accumulator.final
361402

362403
def close(self) -> None:
363404
self._session._router.detach(self._state)
@@ -375,13 +416,19 @@ class ResponsesWebSocketSession:
375416
Keep a lane open for sequential responses. Reconnect releases closed lanes;
376417
open lanes remain registered on the new socket.
377418
419+
By default queue overflow closes the connection. ``isolate_lane_overflow=True``
420+
instead fails the affected lane and drops its later events. A failed default
421+
lane also loses later default/API controls; physical failures remain global.
422+
378423
Recovery remains the existing connection's opt-in policy. Reconnecting a
379424
transport does not restore server-side response state or replay lane sends.
380425
"""
381426

382-
def __init__(self, connection: ResponsesConnection, *, limits: ResponsesWebSocketLimits) -> None:
427+
def __init__(
428+
self, connection: ResponsesConnection, *, limits: ResponsesWebSocketLimits, isolate_lane_overflow: bool = False
429+
) -> None:
383430
self.connection = connection
384-
self._router = _Router(limits, connection)
431+
self._router = _Router(limits, connection, isolate_lane_overflow=isolate_lane_overflow)
385432
self._thread: threading.Thread | None = None
386433
self._closed = False
387434
self._owns_connection = False
@@ -515,10 +562,16 @@ async def get_final_response(self) -> Response:
515562
):
516563
await self.recv()
517564
while self._state.accumulator.final is None:
518-
if self._state.accumulator.error is not None:
519-
raise self._state.accumulator.error
565+
with self._session._router.condition:
566+
if self._state.failure is not None:
567+
raise self._state.failure
568+
if self._state.accumulator.error is not None:
569+
raise self._state.accumulator.error
520570
await self.recv()
521-
return self._state.accumulator.final
571+
with self._session._router.condition:
572+
if self._state.failure is not None:
573+
raise self._state.failure
574+
return self._state.accumulator.final
522575

523576
def close(self) -> None:
524577
self._session._router.detach(self._state)
@@ -527,9 +580,15 @@ def close(self) -> None:
527580
class AsyncResponsesWebSocketSession:
528581
"""Async counterpart of ResponsesWebSocketSession, owning the connection lifetime."""
529582

530-
def __init__(self, connection: AsyncResponsesConnection, *, limits: ResponsesWebSocketLimits) -> None:
583+
def __init__(
584+
self,
585+
connection: AsyncResponsesConnection,
586+
*,
587+
limits: ResponsesWebSocketLimits,
588+
isolate_lane_overflow: bool = False,
589+
) -> None:
531590
self.connection = connection
532-
self._router = _Router(limits, connection)
591+
self._router = _Router(limits, connection, isolate_lane_overflow=isolate_lane_overflow)
533592
self._task: asyncio.Task[None] | None = None
534593
self._close_task: asyncio.Task[None] | None = None
535594
self._closed = False

‎tests/lib/responses/test_websocket_session.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ async def session_for(
9090
connection_options: dict[str, Any] | None = None,
9191
client_headers: dict[str, str] | None = None,
9292
connection_headers: Headers | None = None,
93+
session_options: dict[str, Any] | None = None,
9394
) -> AsyncGenerator[Driver]:
9495
options = connection_options or {}
9596
headers = {"X-Contract-Test": "synthetic", **(connection_headers or {})}
@@ -102,7 +103,7 @@ async def session_for(
102103
http_client=httpx2.Client(trust_env=False),
103104
) as client,
104105
client.responses.connect(extra_query={"contract": "1"}, extra_headers=headers, **options) as connection,
105-
ResponsesWebSocketSession(connection, limits=limits) as session,
106+
ResponsesWebSocketSession(connection, limits=limits, **(session_options or {})) as session,
106107
):
107108
yield Driver(session, mode)
108109
else:
@@ -116,7 +117,7 @@ async def session_for(
116117
async_client.responses.connect(
117118
extra_query={"contract": "1"}, extra_headers=headers, **options
118119
) as async_connection,
119-
AsyncResponsesWebSocketSession(async_connection, limits=limits) as async_session,
120+
AsyncResponsesWebSocketSession(async_connection, limits=limits, **(session_options or {})) as async_session,
120121
):
121122
yield Driver(async_session, mode)
122123

0 commit comments

Comments
 (0)