Skip to content

Commit 3f140ed

Browse files
cliffhallclaude
andcommitted
feat(fetch): cap the response body size while reading it
The server read a whole response body into memory before converting or truncating it, so a URL serving a very large or endless body made it buffer all of it. Bodies are now streamed and read up to a fixed 5 MiB cap, counted after content decoding: a page whose Content-Length is over the cap is refused unread, and one that streams past it is refused once it does. robots.txt is parsed from its first 5 MiB. Redirects are followed by hand so a redirect's own body is closed unread, since httpx reads it whole before following. Based on the prototype in modelcontextprotocol#4185 by @neuralmanacle. Closes modelcontextprotocol#5370 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent 281b167 commit 3f140ed

3 files changed

Lines changed: 376 additions & 35 deletions

File tree

‎src/fetch/README.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,12 +36,16 @@ The markdown of an HTML page opens with the page's title as a `#` heading (its `
3636

3737
### Errors
3838

39-
- A failed fetch (a robots.txt refusal, an unreachable page, an HTTP error status, a refused private address, an unusable proxy setting) and arguments that do not match the tool's schema are returned as a tool result with `isError: true`, whose text starts `Error executing tool fetch:` and says what went wrong, so the model can read it and decide what to do.
39+
- A failed fetch (a robots.txt refusal, an unreachable page, an HTTP error status, a refused private address, an unusable proxy setting, a response body over the [size limit](#response-size)) and arguments that do not match the tool's schema are returned as a tool result with `isError: true`, whose text starts `Error executing tool fetch:` and says what went wrong, so the model can read it and decide what to do.
4040
- Calling a tool the server does not have is a JSON-RPC protocol error, `-32602` (`Unknown tool: <name>`), as the specification requires. So is getting a prompt it does not have (`Unknown prompt: <name>`), or the `fetch` prompt without its `url` (`URL is required`) or with a `url` that is not a valid URL (`Invalid URL '<url>': <reason>`).
4141
- When the `fetch` prompt's fetch fails, the prompt still returns: its message holds the reason the fetch failed.
4242

4343
The server is built on the MCP Python SDK's high-level `MCPServer`: the tool and the prompt are decorated functions, and their type hints and field descriptions are the input schema and the prompt's arguments.
4444

45+
### Response size
46+
47+
The server reads at most 5 MiB (5,242,880 bytes) of a response body, so a very large or endless response cannot exhaust its memory. The limit is fixed and counts the body after any `gzip`, `deflate` or `br` content encoding is decoded. A page whose `Content-Length` is over it is refused before any of the body is read; a page that streams past it is refused once it does. Either way the tool returns an error saying so (and the `fetch` prompt returns that message), rather than part of the page. A robots.txt over the limit is not refused: its rules are read from the first 5 MiB. Each redirect is followed without reading the redirect response's own body. `max_length` and `start_index` page through what was read, so they are not affected by this limit and do not raise it.
48+
4549
### Content negotiation
4650

4751
The tool and the prompt ask for markdown first, with the request header `Accept: text/markdown, text/html;q=0.9, */*;q=0.8`. A site that serves markdown through content negotiation sends it directly, and a `text/markdown` response is returned as-is: it is not simplified and carries no raw-content notice. A site that ignores the header sends what it always did, and HTML is simplified to markdown as before. A `raw` fetch keeps the default `Accept: */*`, so it gets whatever the site serves by default (for a web page, its HTML), unsimplified; a markdown response it receives anyway is returned as-is too.

‎src/fetch/src/mcp_server_fetch/server.py‎

Lines changed: 127 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
# URL and convert HTML to Markdown for a model. The network policy lives here
33
# beside the handlers because every request path (tool, prompt and robots.txt
44
# check) must go through it: the private-address guard on each request and
5-
# redirect hop, robots.txt for autonomous fetches, and proxy normalization.
5+
# redirect hop, robots.txt for autonomous fetches, the cap on how much of a
6+
# response body is read (#5370), and proxy normalization.
67
#
78
# Built on the MCP Python SDK's high-level `MCPServer` (#5174), as the example
89
# to follow: the tool and the prompt are plain decorated functions, whose type
@@ -25,7 +26,8 @@
2526
import ipaddress
2627
import os
2728
import socket
28-
from collections.abc import Awaitable, Callable, Mapping, Sequence
29+
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping, Sequence
30+
from contextlib import asynccontextmanager
2931
from typing import Annotated, Any, Tuple
3032
from importlib.metadata import version
3133
from urllib.parse import urlparse, urlunparse
@@ -397,7 +399,86 @@ def make_client(proxy_url: str | None) -> httpx.AsyncClient:
397399
)
398400

399401

400-
def decode_page(response: httpx.Response) -> str:
402+
# The most body bytes the server reads from one response (#5370). A body is
403+
# streamed and the read stops at this cap, so a very large or endless response
404+
# cannot make the server buffer all of it. `max_length` cannot do this: it
405+
# limits what is returned, after the whole page has been read and converted.
406+
# The cap counts decoded bytes (after gzip, deflate or brotli), so a small
407+
# compressed body that expands enormously is stopped too.
408+
MAX_RESPONSE_BYTES = 5 * 1024 * 1024
409+
410+
411+
@asynccontextmanager
412+
async def stream_get(
413+
client: httpx.AsyncClient, url: str, user_agent: str, **kwargs: Any
414+
) -> AsyncIterator[httpx.Response]:
415+
"""GET ``url`` with redirects followed, yielding the final response unread.
416+
417+
httpx's own redirect handling reads each redirect's body whole before it
418+
follows it, so a redirect with an endless body would get round the cap.
419+
Here each hop is closed unread instead. The request hooks still run before
420+
every hop, since they run on each ``send``. ``kwargs`` go to
421+
``build_request`` (the timeout).
422+
"""
423+
request = client.build_request(
424+
"GET", url, headers={"User-Agent": user_agent}, **kwargs
425+
)
426+
response = await client.send(request, stream=True, follow_redirects=False)
427+
try:
428+
hops = 0
429+
while response.next_request is not None:
430+
hops += 1
431+
if hops > client.max_redirects:
432+
raise httpx.TooManyRedirects(
433+
"Exceeded maximum allowed redirects.",
434+
request=response.next_request,
435+
)
436+
next_request = response.next_request
437+
await response.aclose()
438+
response = await client.send(
439+
next_request, stream=True, follow_redirects=False
440+
)
441+
yield response
442+
finally:
443+
await response.aclose()
444+
445+
446+
async def read_capped(response: httpx.Response, limit: int) -> tuple[bytes, bool]:
447+
"""Read a streamed response's body, stopping once it passes ``limit`` bytes.
448+
449+
Returns the body, at most ``limit`` bytes of it, and whether it was cut
450+
short. Whatever the server has not sent yet is never read: closing the
451+
response (leaving `stream_get`) closes the connection.
452+
"""
453+
body = bytearray()
454+
async for chunk in response.aiter_bytes():
455+
room = limit - len(body)
456+
if len(chunk) > room:
457+
body += chunk[:room]
458+
return bytes(body), True
459+
body += chunk
460+
return bytes(body), False
461+
462+
463+
def declared_length(response: httpx.Response) -> int | None:
464+
"""The body size a response's Content-Length declares, or None without one."""
465+
try:
466+
return int(response.headers["content-length"])
467+
except (KeyError, ValueError):
468+
return None
469+
470+
471+
def decode_text(response: httpx.Response, content: bytes) -> str:
472+
"""Decode ``content`` as httpx's ``response.text`` would.
473+
474+
The header's charset when Python knows it, UTF-8 otherwise, with undecodable
475+
bytes replaced. A streamed response has no ``.text`` until its whole body is
476+
read through httpx, which is the read the cap exists to avoid.
477+
"""
478+
return content.decode(response.encoding or "utf-8", errors="replace")
479+
480+
481+
def decode_page(response: httpx.Response, content: bytes) -> str:
401482
"""Decode a fetched body, honoring a charset the HTML declares (#5199).
402483
403484
httpx's ``response.text`` uses the Content-Type header's charset, and
@@ -413,13 +494,15 @@ def decode_page(response: httpx.Response) -> str:
413494
unrelated attribute (``data-charset``) is not mistaken for a declaration.
414495
A declaration Python does not know, or one that names a codec that is not
415496
a web text encoding, is ignored, and the body is decoded as before.
497+
498+
``content`` is the body as read by `read_capped`; the response supplies
499+
the headers.
416500
"""
417501
# Imported here so the helper stays one self-contained block of this file.
418502
import codecs
419503
import re
420504
from html.parser import HTMLParser
421505

422-
content = response.content
423506
media_type = (
424507
response.headers.get("content-type", "").split(";", 1)[0].strip().lower()
425508
)
@@ -430,7 +513,7 @@ def decode_page(response: httpx.Response) -> str:
430513
(codecs.BOM_UTF8, codecs.BOM_UTF16_LE, codecs.BOM_UTF16_BE)
431514
)
432515
):
433-
return response.text
516+
return decode_text(response, content)
434517

435518
class MetaCharset(HTMLParser):
436519
declared: str | None = None
@@ -458,7 +541,7 @@ def handle_starttag(
458541
scan = MetaCharset()
459542
scan.feed(content[:1024].decode("latin-1"))
460543
if scan.declared is None:
461-
return response.text
544+
return decode_text(response, content)
462545
try:
463546
name = codecs.lookup(scan.declared).name
464547
# A UTF-16/32 declaration in an ASCII-compatible document cannot be
@@ -468,20 +551,20 @@ def handle_starttag(
468551
# Python codecs that are not web text encodings: decoding a page with
469552
# them would mangle it rather than fix it.
470553
if name in ("idna", "punycode", "unicode-escape", "raw-unicode-escape"):
471-
return response.text
554+
return decode_text(response, content)
472555
# A declaration found in ASCII bytes is only believable for an
473556
# ASCII-compatible encoding. UTF-7, HZ and the EBCDIC code pages
474557
# (cp037, cp500, …) would rewrite the markup itself, so they fail this
475558
# probe (by a different result or a decode error) and are ignored.
476559
probe = b"\t\n\r" + bytes(range(0x20, 0x7F))
477560
if probe.decode(name) != probe.decode("ascii"):
478-
return response.text
561+
return decode_text(response, content)
479562
return content.decode(name, errors="replace")
480563
except (LookupError, ValueError): # UnicodeError is a ValueError
481564
# An unknown charset, a name Python refuses outright (one holding a
482565
# NUL), a non-text codec (base64, rot13, …), one that refuses to
483566
# decode at all (``undefined``), or one the probe cannot decode.
484-
return response.text
567+
return decode_text(response, content)
485568

486569

487570
async def check_may_autonomously_fetch_url(
@@ -507,24 +590,25 @@ async def check_may_autonomously_fetch_url(
507590
hooks.append(_refuse_private_destination)
508591
client.event_hooks = {"request": hooks}
509592
try:
510-
response = await client.get(
511-
robot_txt_url,
512-
follow_redirects=True,
513-
headers={"User-Agent": user_agent},
514-
)
593+
async with stream_get(client, robot_txt_url, user_agent) as response:
594+
# A 5xx robots.txt is "unreachable" (RFC 9309 section 2.3.1.4),
595+
# which means complete disallow: the error page is never parsed
596+
# as rules.
597+
if response.status_code in (401, 403) or response.status_code >= 500:
598+
raise ToolError(
599+
f"When fetching robots.txt ({robot_txt_url}), received status {response.status_code} so assuming that autonomous fetching is not allowed, the user can try manually fetching by using the fetch prompt"
600+
)
601+
elif 400 <= response.status_code < 500:
602+
return
603+
# Read up to the same cap as a page (#5370). RFC 9309 section
604+
# 2.5 lets a crawler stop parsing past a limit of at least
605+
# 500 KiB, so the rules in the first part are used as they are.
606+
content, _ = await read_capped(response, MAX_RESPONSE_BYTES)
515607
except HTTPError:
516608
raise ToolError(
517609
f"Failed to fetch robots.txt {robot_txt_url} due to a connection issue"
518610
)
519-
# A 5xx robots.txt is "unreachable" (RFC 9309 section 2.3.1.4), which
520-
# means complete disallow: the error page is never parsed as rules.
521-
if response.status_code in (401, 403) or response.status_code >= 500:
522-
raise ToolError(
523-
f"When fetching robots.txt ({robot_txt_url}), received status {response.status_code} so assuming that autonomous fetching is not allowed, the user can try manually fetching by using the fetch prompt"
524-
)
525-
elif 400 <= response.status_code < 500:
526-
return
527-
robot_txt = response.text
611+
robot_txt = decode_text(response, content)
528612
processed_robot_txt = "\n".join(
529613
line for line in robot_txt.splitlines() if not line.strip().startswith("#")
530614
)
@@ -609,25 +693,34 @@ async def _check_redirect_robots_txt(request: httpx.Request) -> None:
609693
# default representation (a page's HTML), as before this negotiation.
610694
if not force_raw:
611695
client.headers["Accept"] = MARKDOWN_ACCEPT
696+
too_large = (
697+
f"Failed to fetch {url}: the response body is larger than the "
698+
f"{MAX_RESPONSE_BYTES} byte limit this server reads"
699+
)
612700
try:
613-
response = await client.get(
614-
url,
615-
follow_redirects=True,
616-
headers={"User-Agent": user_agent},
617-
timeout=timeout_ms / 1000,
618-
)
701+
async with stream_get(
702+
client, url, user_agent, timeout=timeout_ms / 1000
703+
) as response:
704+
if response.status_code >= 400:
705+
raise ToolError(
706+
f"Failed to fetch {url} - status code {response.status_code}"
707+
)
708+
# Refused before reading anything when the size is declared;
709+
# otherwise the streamed read stops at the cap (#5370).
710+
length = declared_length(response)
711+
if length is not None and length > MAX_RESPONSE_BYTES:
712+
raise ToolError(f"{too_large} (it declares {length} bytes)")
713+
content, truncated = await read_capped(response, MAX_RESPONSE_BYTES)
714+
if truncated:
715+
raise ToolError(too_large)
619716
except httpx.TimeoutException as e:
620717
raise ToolError(
621718
f"Failed to fetch {url}: timed out after {timeout_ms} ms: {e!r}"
622719
)
623720
except HTTPError as e:
624721
raise ToolError(f"Failed to fetch {url}: {e!r}")
625-
if response.status_code >= 400:
626-
raise ToolError(
627-
f"Failed to fetch {url} - status code {response.status_code}"
628-
)
629722

630-
page_raw = decode_page(response)
723+
page_raw = decode_page(response, content)
631724

632725
content_type = response.headers.get("content-type", "")
633726
# Markdown is already what this tool returns, so it is not simplified, and

0 commit comments

Comments
 (0)