Skip to content

Commit 6ca876f

Browse files
cliffhallclaude
andcommitted
docs(fetch): say a raw fetch gets the site's default representation
Accept: */* lets the site choose, so a raw fetch is not guaranteed HTML. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent 2eb63af commit 6ca876f

3 files changed

Lines changed: 6 additions & 4 deletions

File tree

‎src/fetch/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ The server is built on the MCP Python SDK's high-level `MCPServer`: the tool and
4141

4242
### Content negotiation
4343

44-
The tool and the prompt ask for markdown first, with the request header `Accept: text/markdown, text/html;q=0.9, */*;q=0.8`. A site that serves markdown through content negotiation sends it directly, and a `text/markdown` response is returned as-is: it is not simplified and carries no raw-content notice. A site that ignores the header sends what it always did, and HTML is simplified to markdown as before. A `raw` fetch does not ask for markdown, so it still gets the page's own HTML (a markdown response it receives anyway is returned as-is too).
44+
The tool and the prompt ask for markdown first, with the request header `Accept: text/markdown, text/html;q=0.9, */*;q=0.8`. A site that serves markdown through content negotiation sends it directly, and a `text/markdown` response is returned as-is: it is not simplified and carries no raw-content notice. A site that ignores the header sends what it always did, and HTML is simplified to markdown as before. A `raw` fetch keeps the default `Accept: */*`, so it gets whatever the site serves by default (for a web page, its HTML), unsimplified; a markdown response it receives anyway is returned as-is too.
4545

4646
### Protocol versions
4747

‎src/fetch/src/mcp_server_fetch/server.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -568,7 +568,8 @@ async def _check_redirect_robots_txt(request: httpx.Request) -> None:
568568
hooks.append(_check_redirect_robots_txt)
569569
if hooks:
570570
client.event_hooks = {"request": hooks}
571-
# A raw fetch keeps httpx's default Accept, so it still gets the HTML.
571+
# A raw fetch keeps httpx's default Accept: */*, so it gets the site's
572+
# default representation (a page's HTML), as before this negotiation.
572573
if not force_raw:
573574
client.headers["Accept"] = MARKDOWN_ACCEPT
574575
try:

‎src/fetch/tests/test_markdown_negotiation.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,9 @@
66
to markdown" behind the raw-content prefix. Now a non-raw fetch prefers
77
``text/markdown`` (HTML next, anything else last), and a markdown response is
88
returned as-is with no prefix. A ``raw`` fetch keeps the default ``Accept``,
9-
so it still gets the page's own HTML. These tests drive ``serve()`` through an
10-
MCP client with HTTP served by ``harness.FakeWeb``.
9+
so it gets the site's default representation (a page's HTML) as before. These
10+
tests drive ``serve()`` through an MCP client with HTTP served by
11+
``harness.FakeWeb``.
1112
"""
1213

1314
from __future__ import annotations

0 commit comments

Comments
 (0)