Skip to content

Commit 775f017

Browse files
feat(python-sdk): use pyqwest native HTTP/2 connection balancing (#1931)
## Summary [curioswitch/pyqwest#246](curioswitch/pyqwest#246), released in pyqwest 0.12.0, adds client-side load balancing to pyqwest's connection pool. It spreads an origin's HTTP/2 streams across connections by free stream capacity (taken from the server's `SETTINGS_MAX_CONCURRENT_STREAMS`) and dials another connection only once every existing one is full. `max_connections_per_address` caps the number of connections per resolved address. Our Python-side `ConnectionBalancer` / `BalancingTransport` did the same job by fanning out over several pyqwest transports, so this PR removes them and uses one native transport. ```python # before: ConnectionRetryTransport(BalancingTransport(ConnectionBalancer(build))) # after ConnectionRetryTransport( SyncHTTPTransport(..., max_connections_per_address=max_connections), # HTTPTransport for async max_retries=connection_retries, ) ``` - The dependency moves to `pyqwest>=0.12.0,<0.13`. pyqwest 0.12.0 requires Python 3.11 or newer, so this PR is based on #1974, whose commit `chore(python): require Python 3.11` is included here. Merge #1974 first, then rebase this branch onto main. The `code-interpreter-python` and `desktop-python` lockfiles are relocked because they resolve `e2b` from the workspace path. - `E2B_MAX_CONNECTIONS` (default `200`) now maps to `max_connections_per_address`. Once the cap is reached, a request waits for a free stream on an existing connection. Previously it piled onto the least-loaded connection, which had the same effect. - `E2B_STREAMS_PER_CONNECTION` is removed. The per-connection stream limit now comes from the server (the envd edge advertises 100). - Removed `ConnectionBalancer`, `PooledConnection`, `request_origin`, `BalancingTransport` and `_TrackedContent`. Slot accounting for a response's lifetime is now done by pyqwest: it keeps the hyper-util connection lease in the response extensions until the body completes or is closed. - `test_envd_stream_capacity.py` is rewritten to check the native pool against the local HTTP/2 frame server, for both sync and async: - With a 100-stream limit, waves of 80 streams use 1, then 2, then 3 connections, and freed streams are reused. - With a cap of 2 connections × 10 streams, the 21st request waits until a stream closes. - The balancer unit tests are removed. Link to Devin session: https://app.devin.ai/sessions/81af1a093fda4f22a89e3c1e4d78047c Open in Devin Desktop: https://app.devin.ai/desktop/session/81af1a093fda4f22a89e3c1e4d78047c?variant=devin Requested by: @mishushakov --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: [email protected] <[email protected]> Co-authored-by: Mish Ushakov <[email protected]>
1 parent 547decc commit 775f017

13 files changed

Lines changed: 341 additions & 840 deletions

File tree

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@e2b/python-sdk": minor
3+
---
4+
5+
Balance HTTP/2 streams with pyqwest's native connection pool instead of a Python-side balancer over multiple transports. The pool dials a new connection once every existing one is at the server-advertised concurrent-stream limit, capped at `E2B_MAX_CONNECTIONS` (default `200`) connections per resolved address. `E2B_STREAMS_PER_CONNECTION` is removed — the limit now comes from the server's `SETTINGS_MAX_CONCURRENT_STREAMS`.

‎packages/code-interpreter-python/uv.lock‎

Lines changed: 56 additions & 46 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/desktop-python/uv.lock‎

Lines changed: 56 additions & 46 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/python-sdk/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,12 +68,12 @@ Per-call params still take precedence over the client's params, and clients are
6868

6969
### High-concurrency sandbox streams
7070

71-
Sandbox `commands` and `files` traffic from all sandboxes in a process shares one set of HTTP/2 connections. Each request goes to the connection with the fewest streams in flight to its host, and a new connection is dialed only when every existing one already carries `E2B_STREAMS_PER_CONNECTION` streams to that host (default `100`, the usual server limit), up to `E2B_MAX_CONNECTIONS` (default `200`). Long-running streams therefore never queue behind one connection's concurrent-stream limit, and idle processes keep a single connection.
71+
Sandbox `commands` and `files` traffic from all sandboxes in a process shares one HTTP/2 connection pool. Requests are spread over the pooled connections by free stream capacity, and a new connection is dialed only once every existing one is at the server's concurrent-stream limit (`100` for `sandbox.e2b.app`), up to `E2B_MAX_CONNECTIONS` connections per resolved address (default `200`). Past that cap, a request waits for a stream to free up. Long-running streams therefore never queue behind one connection's concurrent-stream limit, and idle processes keep a single connection.
7272

73-
Both variables are read when `e2b` is imported:
73+
`E2B_MAX_CONNECTIONS` is read when `e2b` is imported:
7474

7575
```sh
76-
E2B_STREAMS_PER_CONNECTION=50 E2B_MAX_CONNECTIONS=400 python eval.py
76+
E2B_MAX_CONNECTIONS=400 python eval.py
7777
```
7878

7979
### 5. Code execution with Code Interpreter

‎packages/python-sdk/e2b/api/__init__.py‎

Lines changed: 7 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,16 @@
11
import json
22
import logging
33
import os
4-
import threading
54
from dataclasses import dataclass
65
from types import TracebackType
76
from typing import (
8-
Callable,
9-
Dict,
10-
Generic,
11-
List,
127
NamedTuple,
138
Optional,
149
Protocol,
1510
Tuple,
16-
TypeVar,
1711
Union,
1812
)
19-
from urllib.parse import quote, urlsplit
13+
from urllib.parse import quote
2014

2115
import httpx
2216
from httpx import AsyncBaseTransport, BaseTransport, Timeout
@@ -95,78 +89,14 @@ async def on_response(response: Response) -> None:
9589
pool_idle_timeout = float(os.getenv("E2B_KEEPALIVE_EXPIRY") or "300")
9690
pool_max_idle_per_host = int(os.getenv("E2B_MAX_KEEPALIVE_CONNECTIONS") or "20")
9791

98-
# A reqwest pool multiplexes every request to an origin over one HTTP/2
99-
# connection and queues past the server's concurrent-stream limit (envd's
100-
# edge, which all sandboxes share, advertises 100) rather than dialing another
101-
# — long-running commands hold those streams. So each cached transport is a
102-
# set of reqwest pools that grows on demand, like undici's
103-
# `Agent({ connections })` in the JS SDK: a request goes to the pool with the
104-
# fewest in flight to its origin, another pool is dialed once every one
105-
# carries `streams_per_connection` to that origin, up to `max_connections`
106-
# pools.
107-
streams_per_connection = max(1, int(os.getenv("E2B_STREAMS_PER_CONNECTION") or "100"))
92+
# pyqwest spreads an origin's HTTP/2 streams over as many connections as the
93+
# server's concurrent-stream limit requires (envd's edge, which all sandboxes
94+
# share, advertises 100), dialing another once every connection is full, so
95+
# long-running commands never queue behind one connection. `max_connections`
96+
# caps the connections to each address an origin resolves to; past it, a
97+
# request waits on the least loaded connection for a stream to free up.
10898
max_connections = max(1, int(os.getenv("E2B_MAX_CONNECTIONS") or "200"))
10999

110-
T = TypeVar("T")
111-
112-
113-
def request_origin(url: str) -> str:
114-
"""The origin a request URL is served from — what a reqwest pool keeps
115-
one HTTP/2 connection, and so one concurrent-stream budget, for."""
116-
parts = urlsplit(url)
117-
return f"{parts.scheme}://{parts.netloc}".lower()
118-
119-
120-
class PooledConnection(Generic[T]):
121-
"""One reqwest pool in a :class:`ConnectionBalancer` and the number of
122-
requests in flight on it, per origin: the pool keeps a connection per
123-
origin, so the streams of one never crowd out another's."""
124-
125-
__slots__ = ("transport", "active")
126-
127-
def __init__(self, transport: T) -> None:
128-
self.transport = transport
129-
self.active: Dict[str, int] = {}
130-
131-
def in_flight(self, origin: str) -> int:
132-
return self.active.get(origin, 0)
133-
134-
135-
class ConnectionBalancer(Generic[T]):
136-
"""Least-loaded selection over a growable, bounded set of reqwest pools.
137-
138-
The policy shared by the sync and async balancing transports, which do the
139-
I/O: :meth:`acquire` picks (or dials) a pool and counts the request on it,
140-
:meth:`release` uncounts it once the response is consumed or closed.
141-
Per-request state is a lock away from every thread and loop that shares
142-
the transport; the lock is reentrant because a release can run from a
143-
``__del__`` at any point, including inside :meth:`acquire`."""
144-
145-
def __init__(self, build: Callable[[], T]) -> None:
146-
self._build = build
147-
self._lock = threading.RLock()
148-
self.connections: List[PooledConnection[T]] = [PooledConnection(build())]
149-
150-
def acquire(self, origin: str) -> PooledConnection[T]:
151-
with self._lock:
152-
connection = min(self.connections, key=lambda c: c.in_flight(origin))
153-
if (
154-
connection.in_flight(origin) >= streams_per_connection
155-
and len(self.connections) < max_connections
156-
):
157-
connection = PooledConnection(self._build())
158-
self.connections.append(connection)
159-
connection.active[origin] = connection.in_flight(origin) + 1
160-
return connection
161-
162-
def release(self, connection: PooledConnection[T], origin: str) -> None:
163-
with self._lock:
164-
remaining = connection.in_flight(origin) - 1
165-
if remaining:
166-
connection.active[origin] = remaining
167-
else:
168-
del connection.active[origin]
169-
170100

171101
class ProxyConfig(NamedTuple):
172102
"""The ``proxy`` connection option in the shape pyqwest transports take.

‎packages/python-sdk/e2b/api/client_async/__init__.py‎

Lines changed: 31 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
import threading
2-
from functools import partial
3-
from typing import AsyncIterator, Callable, Dict, Optional, Tuple, Union
2+
from typing import Dict, Optional, Tuple, Union
43

54
import httpx
65

@@ -11,14 +10,13 @@
1110
from e2b.retry import AsyncRetryableTransport
1211
from e2b.api import (
1312
AsyncApiClient,
14-
ConnectionBalancer,
1513
ProxyConfig,
1614
connection_retries,
15+
max_connections,
1716
make_async_logging_event_hooks,
1817
pool_idle_timeout,
1918
pool_max_idle_per_host,
2019
proxy_to_config,
21-
request_origin,
2220
)
2321
from e2b.connection_config import (
2422
DEFAULT_HTTP_VERSION,
@@ -61,89 +59,13 @@ def should_retry_response(
6159
return isinstance(response, ConnectionError)
6260

6361

64-
_Chunk = Union[bytes, bytearray, memoryview]
65-
66-
67-
class _TrackedContent:
68-
"""The content iterator of a balanced response: hands back the original
69-
response's chunks and releases its connection slot exactly once, after
70-
the stream is over — when the content is exhausted, when the response is
71-
closed (``Response`` forwards ``aclose`` to its content), when reading
72-
fails, or, failing all of those, when it is garbage collected, which
73-
drops the original response and so resets its stream."""
74-
75-
__slots__ = ("_response", "_content", "_release")
76-
77-
def __init__(self, response: Response, release: Callable[[], None]) -> None:
78-
self._response = response
79-
self._content = response.content.__aiter__()
80-
self._release: Optional[Callable[[], None]] = release
81-
82-
def __aiter__(self) -> AsyncIterator[_Chunk]:
83-
return self
84-
85-
async def __anext__(self) -> _Chunk:
86-
try:
87-
return await self._content.__anext__()
88-
except BaseException:
89-
await self.aclose()
90-
raise
91-
92-
async def aclose(self) -> None:
93-
release, self._release = self._release, None
94-
if release is None:
95-
return
96-
try:
97-
await self._response.aclose()
98-
finally:
99-
release()
100-
101-
def __del__(self) -> None:
102-
release, self._release = self._release, None
103-
if release is None:
104-
return
105-
del self._content, self._response
106-
release()
107-
108-
109-
class BalancingTransport(Transport):
110-
"""A pyqwest transport over a growable, bounded set of reqwest pools
111-
(see :class:`e2b.api.ConnectionBalancer`): each request runs on the pool
112-
with the fewest in flight to its origin and counts against it until its
113-
response is consumed or closed. Pools are built with ``build`` — the first
114-
eagerly, the rest as load requires — so they share every construction
115-
knob."""
116-
117-
def __init__(self, build: Callable[[], HTTPTransport]) -> None:
118-
self.balancer: ConnectionBalancer[HTTPTransport] = ConnectionBalancer(build)
119-
120-
async def execute(self, request: Request) -> Response:
121-
origin = request_origin(request.url)
122-
connection = self.balancer.acquire(origin)
123-
try:
124-
response = await connection.transport.execute(request)
125-
except BaseException:
126-
self.balancer.release(connection, origin)
127-
raise
128-
return Response(
129-
status=response.status,
130-
http_version=response.http_version,
131-
headers=response.headers,
132-
content=_TrackedContent(
133-
response, partial(self.balancer.release, connection, origin)
134-
),
135-
# Populated in place once the original content is consumed.
136-
trailers=response.trailers,
137-
)
138-
139-
14062
_TransportKey = Tuple[Optional[ProxyConfig], Optional[float], HttpVersion]
14163
"""Cache key: proxy, idle read bound, HTTP version — fixed when a pyqwest
14264
transport is constructed, so each distinct combination is necessarily its own
14365
transport."""
14466

14567
_transport_lock = threading.Lock()
146-
# One balancing transport — a set of reqwest connection pools — per key; a
68+
# One transport — a reqwest connection pool — per key; a
14769
# `None` proxy is the direct transport. The REST API, envd RPC, envd HTTP API
14870
# and volume stacks all draw from the same one for their proxy, so a sandbox's
14971
# RPC and HTTP traffic multiplex over the same connections instead of opening
@@ -153,9 +75,9 @@ async def execute(self, request: Request) -> Response:
15375
# they replaced, the transports are not bound to an event loop and the caches
15476
# are process-global rather than per-loop.
15577
_transports: Dict[_TransportKey, ConnectionRetryTransport] = {}
156-
# The balancing transport under each retrying one, for callers that must not
78+
# The pyqwest transport under each retrying one, for callers that must not
15779
# retry connecting (the health probe run after a request failed).
158-
_pools: Dict[_TransportKey, BalancingTransport] = {}
80+
_pools: Dict[_TransportKey, HTTPTransport] = {}
15981
# The httpx adapter over each transport, shared by every httpx client on it.
16082
_httpx_transports: Dict[Tuple[_TransportKey, bool], AsyncPyqwestTransport] = {}
16183

@@ -169,18 +91,19 @@ def get_pyqwest_transport(
16991
) -> Transport:
17092
"""The shared pyqwest transport with the SDK's tuning — system CA certs
17193
(without which TLS through an intercepting proxy fails) and the
172-
httpx-equivalent pool limits — behind connect-only retries. It balances
173-
requests over as many reqwest connection pools as the load needs (see
174-
:class:`BalancingTransport`), so long-running streams do not all queue
175-
behind one HTTP/2 connection's concurrent-stream limit.
94+
httpx-equivalent pool limits — behind connect-only retries. pyqwest
95+
spreads each origin's HTTP/2 streams over as many connections as the
96+
server's concurrent-stream limit requires, up to ``max_connections`` per
97+
address, so long-running streams do not all queue behind one
98+
connection.
17699
177100
Consumers speaking pyqwest natively (the envd RPC clients) take this;
178101
consumers speaking httpx take :func:`get_httpx_transport`, the adapter over
179102
the very same transport. Layer concerns above it rather than into it — the
180103
RPC stack's plain-HTTP-error normalization wraps it, headers and codecs are
181-
per-request — so that the pools stay shareable.
104+
per-request — so that the pool stays shareable.
182105
183-
``read_timeout`` bounds every read on the pools' connections and
106+
``read_timeout`` bounds every read on the pool's connections and
184107
``http_version`` fixes the HTTP version; both are part of the cache key
185108
because they are transport-construction knobs, so one pool cannot serve two
186109
values of either. reqwest's read timer keeps running while a request body is sent and
@@ -193,30 +116,28 @@ def get_pyqwest_transport(
193116
diagnostics httpcore used to provide. The SDK's own ``logger`` option is
194117
separate and sits above this, on the httpx client.
195118
196-
``retry_connect=False`` returns the balancing transport under the retrying
197-
one — the same pools, without the connection retries."""
119+
``retry_connect=False`` returns the pyqwest transport under the retrying
120+
one — the same pool, without the connection retries."""
198121
key = (proxy, read_timeout, http_version)
199122

200-
def build() -> HTTPTransport:
201-
return HTTPTransport(
202-
tls_include_system_certs=True,
203-
proxy=proxy.to_pyqwest() if proxy is not None else None,
204-
pool_idle_timeout=pool_idle_timeout,
205-
pool_max_idle_per_host=pool_max_idle_per_host,
206-
read_timeout=read_timeout,
207-
# `None` leaves the version to ALPN on TLS connections (HTTP/2
208-
# against the E2B API and envd) and uses HTTP/1 for plaintext,
209-
# like the http2-enabled httpx transport this replaced.
210-
http_version=HTTPVersion.HTTP1 if http_version == "1.1" else None,
211-
# Redirects belong to the httpx client above (which the generated
212-
# clients leave off), not to reqwest.
213-
follow_redirects=False,
214-
)
215-
216123
with _transport_lock:
217124
transport = _transports.get(key)
218125
if transport is None:
219-
pool = BalancingTransport(build)
126+
pool = HTTPTransport(
127+
tls_include_system_certs=True,
128+
proxy=proxy.to_pyqwest() if proxy is not None else None,
129+
pool_idle_timeout=pool_idle_timeout,
130+
pool_max_idle_per_host=pool_max_idle_per_host,
131+
max_connections_per_address=max_connections,
132+
read_timeout=read_timeout,
133+
# `None` leaves the version to ALPN on TLS connections (HTTP/2
134+
# against the E2B API and envd) and uses HTTP/1 for plaintext,
135+
# like the http2-enabled httpx transport this replaced.
136+
http_version=HTTPVersion.HTTP1 if http_version == "1.1" else None,
137+
# Redirects belong to the httpx client above (which the generated
138+
# clients leave off), not to reqwest.
139+
follow_redirects=False,
140+
)
220141
transport = ConnectionRetryTransport(pool, max_retries=connection_retries)
221142
_pools[key] = pool
222143
_transports[key] = transport
@@ -233,9 +154,9 @@ def get_httpx_transport(
233154
"""The httpx adapter over the shared transport of
234155
:func:`get_pyqwest_transport`, for the generated httpx clients (control
235156
plane, envd HTTP API, volume content). The adapter holds no state of its
236-
own and does not close the pools, so closing an httpx client leaves them
157+
own and does not close the pool, so closing an httpx client leaves it
237158
intact for the other clients on it. ``retry_connect=False`` adapts the
238-
same pools without the connection retries."""
159+
same pool without the connection retries."""
239160
key = ((proxy, read_timeout, http_version), retry_connect)
240161
# Resolve the pool before taking the lock: it takes the same one.
241162
pool = get_pyqwest_transport(

0 commit comments

Comments
 (0)