Skip to content

Commit 781e72a

Browse files
cliffhallclaude
andcommitted
fix(fetch): bound decoding of a compressed body by the byte cap
httpx's gzip and deflate decoders inflate each network chunk whole, so one small chunk of a compressed body could allocate far past the 5 MiB cap before the size check saw it: a 199 KiB gzip body made one decode step allocate 200 MiB. The server now reads raw bytes and inflates them itself with zlib's max_length set to the room left under the cap, so no step produces more than one byte past it. It asks for and decodes only gzip and deflate, and refuses any other content coding (br, zstd, a chain of codings) rather than decode it without a bound. The test harness now serves each response unread over its stream, as a network transport does: a response built with content= is read and decoded when constructed, which left nothing for a raw read. Refs #5370 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent 3f140ed commit 781e72a

4 files changed

Lines changed: 216 additions & 16 deletions

File tree

‎src/fetch/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ The server is built on the MCP Python SDK's high-level `MCPServer`: the tool and
4444

4545
### Response size
4646

47-
The server reads at most 5 MiB (5,242,880 bytes) of a response body, so a very large or endless response cannot exhaust its memory. The limit is fixed and counts the body after any `gzip`, `deflate` or `br` content encoding is decoded. A page whose `Content-Length` is over it is refused before any of the body is read; a page that streams past it is refused once it does. Either way the tool returns an error saying so (and the `fetch` prompt returns that message), rather than part of the page. A robots.txt over the limit is not refused: its rules are read from the first 5 MiB. Each redirect is followed without reading the redirect response's own body. `max_length` and `start_index` page through what was read, so they are not affected by this limit and do not raise it.
47+
The server reads at most 5 MiB (5,242,880 bytes) of a response body, so a very large or endless response cannot exhaust its memory. The limit is fixed and counts the body after its content encoding is decoded, and decoding itself stops at the limit, so a small compressed body that expands enormously is stopped without being expanded. The server asks for, and decodes, only `gzip` and `deflate` (`Accept-Encoding: gzip, deflate`); a response in any other content encoding (`br`, `zstd`, or more than one encoding applied in turn) is refused with an error. A page whose `Content-Length` is over it is refused before any of the body is read; a page that streams past it is refused once it does. Either way the tool returns an error saying so (and the `fetch` prompt returns that message), rather than part of the page. A robots.txt over the limit is not refused: its rules are read from the first 5 MiB. Each redirect is followed without reading the redirect response's own body. `max_length` and `start_index` page through what was read, so they are not affected by this limit and do not raise it.
4848

4949
### Content negotiation
5050

‎src/fetch/src/mcp_server_fetch/server.py‎

Lines changed: 94 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import ipaddress
2727
import os
2828
import socket
29+
import zlib
2930
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping, Sequence
3031
from contextlib import asynccontextmanager
3132
from typing import Annotated, Any, Tuple
@@ -403,10 +404,76 @@ def make_client(proxy_url: str | None) -> httpx.AsyncClient:
403404
# streamed and the read stops at this cap, so a very large or endless response
404405
# cannot make the server buffer all of it. `max_length` cannot do this: it
405406
# limits what is returned, after the whole page has been read and converted.
406-
# The cap counts decoded bytes (after gzip, deflate or brotli), so a small
407-
# compressed body that expands enormously is stopped too.
407+
# The cap counts decoded bytes, so a small compressed body that expands
408+
# enormously is stopped too, and decoding itself is bounded by it (see
409+
# `read_capped`).
408410
MAX_RESPONSE_BYTES = 5 * 1024 * 1024
409411

412+
# The content codings the server asks for, and the only ones it decodes. They
413+
# are named explicitly because httpx would also offer br and zstd whenever
414+
# their packages happen to be installed, and decode them with no output limit.
415+
ACCEPT_ENCODING = "gzip, deflate"
416+
417+
418+
class BoundedInflater:
419+
"""Incremental gzip or deflate decoding whose output per call is bounded.
420+
421+
httpx's decoders inflate each network chunk whole, so one small chunk of a
422+
compression bomb could allocate far past the cap before any check saw it
423+
(#5370). zlib's ``max_length`` stops each call at the room left instead,
424+
and keeps the input it did not get to in ``unconsumed_tail``.
425+
"""
426+
427+
def __init__(self, coding: str) -> None:
428+
self._gzip = coding in ("gzip", "x-gzip")
429+
self._first = True
430+
self._z = zlib.decompressobj(
431+
zlib.MAX_WBITS | 16 if self._gzip else zlib.MAX_WBITS
432+
)
433+
434+
@property
435+
def unconsumed_tail(self) -> bytes:
436+
return self._z.unconsumed_tail
437+
438+
def decompress(self, data: bytes, max_length: int) -> bytes:
439+
try:
440+
out = self._z.decompress(data, max_length)
441+
except zlib.error as e:
442+
if self._first and not self._gzip:
443+
# Servers often send deflate raw, without zlib's wrapper; httpx
444+
# retries the same way.
445+
self._first = False
446+
self._z = zlib.decompressobj(-zlib.MAX_WBITS)
447+
return self.decompress(data, max_length)
448+
raise httpx.DecodingError(f"Error decoding the response body: {e}")
449+
self._first = False
450+
return out
451+
452+
def flush(self) -> bytes:
453+
return self._z.flush()
454+
455+
456+
def inflater_for(response: httpx.Response) -> BoundedInflater | None:
457+
"""The decoder a response's Content-Encoding needs, or None for none.
458+
459+
Only a single gzip or deflate coding is decoded. Any other (br, zstd, a
460+
chain of codings) is refused rather than handed over undecoded or decoded
461+
without a bound.
462+
"""
463+
codings = [
464+
coding.strip().lower()
465+
for coding in response.headers.get("content-encoding", "").split(",")
466+
]
467+
codings = [coding for coding in codings if coding not in ("", "identity")]
468+
if not codings:
469+
return None
470+
if len(codings) == 1 and codings[0] in ("gzip", "x-gzip", "deflate"):
471+
return BoundedInflater(codings[0])
472+
raise httpx.DecodingError(
473+
f"Unsupported content encoding {', '.join(codings)}: this server "
474+
f"decodes only {ACCEPT_ENCODING}"
475+
)
476+
410477

411478
@asynccontextmanager
412479
async def stream_get(
@@ -421,7 +488,10 @@ async def stream_get(
421488
``build_request`` (the timeout).
422489
"""
423490
request = client.build_request(
424-
"GET", url, headers={"User-Agent": user_agent}, **kwargs
491+
"GET",
492+
url,
493+
headers={"User-Agent": user_agent, "Accept-Encoding": ACCEPT_ENCODING},
494+
**kwargs,
425495
)
426496
response = await client.send(request, stream=True, follow_redirects=False)
427497
try:
@@ -448,15 +518,29 @@ async def read_capped(response: httpx.Response, limit: int) -> tuple[bytes, bool
448518
449519
Returns the body, at most ``limit`` bytes of it, and whether it was cut
450520
short. Whatever the server has not sent yet is never read: closing the
451-
response (leaving `stream_get`) closes the connection.
521+
response (leaving `stream_get`) closes the connection. The raw bytes are
522+
decoded here rather than by httpx, so no single step produces more than
523+
one byte past the room left under ``limit`` (see `BoundedInflater`).
452524
"""
525+
inflater = inflater_for(response)
453526
body = bytearray()
454-
async for chunk in response.aiter_bytes():
455-
room = limit - len(body)
456-
if len(chunk) > room:
457-
body += chunk[:room]
458-
return bytes(body), True
459-
body += chunk
527+
async for raw in response.aiter_raw():
528+
pending = raw
529+
while pending:
530+
room = limit - len(body)
531+
if inflater is None:
532+
out, pending = pending[: room + 1], b""
533+
else:
534+
out = inflater.decompress(pending, room + 1)
535+
pending = inflater.unconsumed_tail
536+
body += out
537+
if len(body) > limit:
538+
return bytes(body[:limit]), True
539+
if inflater is not None:
540+
# All input is consumed by now, so zlib has at most a few bytes left.
541+
body += inflater.flush()
542+
if len(body) > limit:
543+
return bytes(body[:limit]), True
460544
return bytes(body), False
461545

462546

‎src/fetch/tests/harness.py‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,21 @@ def handle(self, request: httpx.Request) -> httpx.Response:
6666
self.requests.append(request)
6767
responder = self.routes.get(str(request.url))
6868
if responder is None:
69-
return httpx.Response(404, content=b"no route in FakeWeb")
70-
if isinstance(responder, httpx.Response):
71-
return responder
72-
return responder(request)
69+
response = httpx.Response(404, content=b"no route in FakeWeb")
70+
elif isinstance(responder, httpx.Response):
71+
response = responder
72+
else:
73+
response = responder(request)
74+
# Served unread, as from the network. `httpx.Response(content=...)`
75+
# reads (and decodes) its body when it is built, which leaves nothing
76+
# for the server's raw, capped read (#5370); its stream still holds the
77+
# bytes as given, so a fresh response over that stream is the wire form.
78+
return httpx.Response(
79+
response.status_code,
80+
headers=response.headers,
81+
stream=response.stream,
82+
extensions=response.extensions,
83+
)
7384

7485
def urls(self) -> list[str]:
7586
return [str(r.url) for r in self.requests]

‎src/fetch/tests/test_response_size_cap.py‎

Lines changed: 106 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,10 @@
77
read, a page that streams past the cap is refused once it does, and a
88
robots.txt is parsed from its first ``MAX_RESPONSE_BYTES``. Redirects are
99
followed without reading the redirect's body, which httpx would otherwise read
10-
whole. These tests drive the real server through an MCP client
10+
whole. A gzip or deflate body is decoded by the server itself with each
11+
step's output bounded by the room left under the cap, since httpx inflates a
12+
whole network chunk at once (a compression bomb); any other content coding is
13+
refused. These tests drive the real server through an MCP client
1114
(``harness.connect``) with HTTP served by ``harness.FakeWeb``, under a small
1215
cap so a body over it stays cheap; endless bodies count the chunks they hand
1316
out, to show the read stopped.
@@ -16,6 +19,7 @@
1619
from __future__ import annotations
1720

1821
import gzip
22+
import zlib
1923
from collections.abc import AsyncIterator
2024
from typing import Any
2125

@@ -242,3 +246,104 @@ async def test_the_prompt_reports_an_oversized_page(web: FakeWeb) -> None:
242246
result = await session.get_prompt("fetch", {"url": PAGE})
243247
message = result.model_dump(mode="json")["messages"][0]["content"]["text"]
244248
assert message == too_large().removeprefix(TOOL_ERROR)
249+
250+
251+
# --- content codings ----------------------------------------------------------
252+
253+
254+
def encoded(body: bytes, coding: str) -> httpx.Response:
255+
# Over a stream, not `content=`, which httpx would decode while building it.
256+
return httpx.Response(
257+
200,
258+
stream=httpx.ByteStream(body),
259+
headers={"content-type": "text/plain", "content-encoding": coding},
260+
)
261+
262+
263+
def raw_deflate(data: bytes) -> bytes:
264+
compressor = zlib.compressobj(wbits=-zlib.MAX_WBITS)
265+
return compressor.compress(data) + compressor.flush()
266+
267+
268+
async def test_a_compression_bomb_never_decodes_past_the_cap(
269+
web: FakeWeb, monkeypatch: pytest.MonkeyPatch
270+
) -> None:
271+
# One small network chunk that inflates to 10,000 times the cap. httpx
272+
# would inflate it whole before any size check; the server's own decoder
273+
# must never produce more than the room left plus one byte in a step.
274+
outputs: list[int] = []
275+
real = zlib.decompressobj
276+
277+
class Spy:
278+
def __init__(self, *args: Any) -> None:
279+
self._z = real(*args)
280+
281+
@property
282+
def unconsumed_tail(self) -> bytes:
283+
return self._z.unconsumed_tail
284+
285+
def decompress(self, data: bytes, max_length: int = 0) -> bytes:
286+
out = self._z.decompress(data, max_length)
287+
outputs.append(len(out))
288+
return out
289+
290+
monkeypatch.setattr(zlib, "decompressobj", Spy)
291+
bomb = gzip.compress(b"a" * (CAP * 10_000))
292+
assert len(bomb) < CAP * 50
293+
web.add(PAGE, encoded(bomb, "gzip"))
294+
result = await call()
295+
assert text_of(result) == too_large()
296+
assert outputs and max(outputs) <= CAP + 1
297+
assert sum(outputs) <= CAP + 1
298+
299+
300+
@pytest.mark.parametrize(
301+
("coding", "encode"),
302+
[
303+
("gzip", gzip.compress),
304+
("x-gzip", gzip.compress),
305+
("GZIP", gzip.compress),
306+
("identity, gzip", gzip.compress),
307+
("deflate", zlib.compress),
308+
("deflate", raw_deflate), # sent without zlib's wrapper
309+
],
310+
)
311+
async def test_a_gzip_or_deflate_body_is_decoded(
312+
web: FakeWeb, coding: str, encode: Any
313+
) -> None:
314+
web.add(PAGE, encoded(encode(b"d" * CAP), coding))
315+
result = await call()
316+
assert wire(result)["isError"] is False
317+
assert text_of(result).endswith("d" * CAP)
318+
319+
320+
async def test_an_identity_coding_is_read_as_is(web: FakeWeb) -> None:
321+
web.add(PAGE, encoded(b"plain", "identity"))
322+
result = await call()
323+
assert text_of(result).endswith("plain")
324+
325+
326+
@pytest.mark.parametrize("coding", ["br", "zstd", "gzip, gzip", "compress"])
327+
async def test_a_coding_the_server_cannot_bound_is_refused(
328+
web: FakeWeb, coding: str
329+
) -> None:
330+
web.add(PAGE, encoded(b"\x00\x01\x02", coding))
331+
result = await call()
332+
assert wire(result)["isError"] is True
333+
assert "Unsupported content encoding" in text_of(result)
334+
assert "this server decodes only gzip, deflate" in text_of(result)
335+
336+
337+
@pytest.mark.parametrize("coding", ["gzip", "deflate"])
338+
async def test_a_corrupt_body_is_a_decoding_error(web: FakeWeb, coding: str) -> None:
339+
# Deflate is retried as raw deflate once, then fails like gzip.
340+
web.add(PAGE, encoded(b"\xff" * 64, coding))
341+
result = await call()
342+
assert wire(result)["isError"] is True
343+
assert "Error decoding the response body" in text_of(result)
344+
345+
346+
async def test_only_the_codings_it_decodes_are_asked_for(web: FakeWeb) -> None:
347+
web.add(PAGE, httpx.Response(200, content=b"ok"))
348+
await call()
349+
assert web.requests[0].headers["accept-encoding"] == "gzip, deflate"

0 commit comments

Comments
 (0)