Skip to content

Commit 2eb63af

Browse files
cliffhallclaude
andcommitted
feat(fetch): ask for native markdown through Accept content negotiation
A non-raw fetch now sends Accept: text/markdown, text/html;q=0.9, */*;q=0.8, so a site that negotiates content can serve markdown directly. A text/markdown response is returned as-is, without the raw-content prefix that labeled it as content that "cannot be simplified to markdown". A raw fetch keeps httpx's default Accept, so it still gets the page's own HTML. Closes modelcontextprotocol#5198 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent 446be48 commit 2eb63af

3 files changed

Lines changed: 158 additions & 0 deletions

File tree

‎src/fetch/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ The fetch tool will truncate the response, but by using the `start_index` argume
3939

4040
The server is built on the MCP Python SDK's high-level `MCPServer`: the tool and the prompt are decorated functions, and their type hints and field descriptions are the input schema and the prompt's arguments.
4141

42+
### Content negotiation
43+
44+
The tool and the prompt ask for markdown first, with the request header `Accept: text/markdown, text/html;q=0.9, */*;q=0.8`. A site that serves markdown through content negotiation sends it directly, and a `text/markdown` response is returned as-is: it is not simplified and carries no raw-content notice. A site that ignores the header sends what it always did, and HTML is simplified to markdown as before. A `raw` fetch does not ask for markdown, so it still gets the page's own HTML (a markdown response it receives anyway is returned as-is too).
45+
4246
### Protocol versions
4347

4448
The server speaks both eras of the Model Context Protocol over stdio, with nothing to configure: a **2026-07-28** client (discovered with `server/discover`, no session) and a **2025-11-25** client (the `initialize` handshake). The client's first request decides which. Tools and prompts behave the same in both. Methods the 2026-07-28 specification removes, such as `ping`, are answered only on a 2025-11-25 connection.

‎src/fetch/src/mcp_server_fetch/server.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -507,6 +507,17 @@ async def check_may_autonomously_fetch_url(
507507
)
508508

509509

510+
# The Accept header of a non-raw fetch (#5198). A site that negotiates content
511+
# can then serve native markdown, which is returned as-is, and a site that does
512+
# not still serves HTML (preferred over everything else) to be simplified.
513+
MARKDOWN_ACCEPT = "text/markdown, text/html;q=0.9, */*;q=0.8"
514+
515+
516+
def _is_markdown(content_type: str) -> bool:
517+
"""Whether a Content-Type header declares markdown (``text/markdown``, RFC 7763)."""
518+
return content_type.split(";", 1)[0].strip().lower() == "text/markdown"
519+
520+
510521
def _body_looks_like_html(page_raw: str) -> bool:
511522
"""Whether a body served with no Content-Type is an HTML document.
512523
@@ -557,6 +568,9 @@ async def _check_redirect_robots_txt(request: httpx.Request) -> None:
557568
hooks.append(_check_redirect_robots_txt)
558569
if hooks:
559570
client.event_hooks = {"request": hooks}
571+
# A raw fetch keeps httpx's default Accept, so it still gets the HTML.
572+
if not force_raw:
573+
client.headers["Accept"] = MARKDOWN_ACCEPT
560574
try:
561575
response = await client.get(
562576
url,
@@ -574,6 +588,10 @@ async def _check_redirect_robots_txt(request: httpx.Request) -> None:
574588
page_raw = decode_page(response)
575589

576590
content_type = response.headers.get("content-type", "")
591+
# Markdown is already what this tool returns, so it is not simplified, and
592+
# it carries no raw-content prefix even under force_raw: it is unchanged.
593+
if _is_markdown(content_type):
594+
return page_raw, ""
577595
is_page_html = (
578596
"<html" in page_raw[:100]
579597
or "text/html" in content_type.lower()
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
"""Protocol-level tests for asking for native markdown (#5198).
2+
3+
A fetch used to send httpx's default ``Accept: */*``, so a site that can serve
4+
markdown through content negotiation sent HTML to be simplified, and a
5+
``text/markdown`` response was labeled as content that "cannot be simplified
6+
to markdown" behind the raw-content prefix. Now a non-raw fetch prefers
7+
``text/markdown`` (HTML next, anything else last), and a markdown response is
8+
returned as-is with no prefix. A ``raw`` fetch keeps the default ``Accept``,
9+
so it still gets the page's own HTML. These tests drive ``serve()`` through an
10+
MCP client with HTTP served by ``harness.FakeWeb``.
11+
"""
12+
13+
from __future__ import annotations
14+
15+
import httpx
16+
import pytest
17+
18+
from mcp_server_fetch.server import MARKDOWN_ACCEPT, _is_markdown
19+
20+
from .harness import FakeWeb, connect
21+
22+
PAGE = "https://example.com/doc"
23+
ROBOTS = "https://example.com/robots.txt"
24+
25+
# Opens with an "<html" mention inside the first 100 characters, which the
26+
# HTML sniff would otherwise take for a page to simplify.
27+
MARKDOWN = "# Title\n\nUse an `<html>` tag at the top.\n\n- one\n- two\n"
28+
29+
ARTICLE = """<html><head><title>Test Page</title></head><body>
30+
<article>
31+
<h1>Hello World</h1>
32+
<p>This is a <a href="https://example.com/x">link</a> in a test paragraph
33+
with enough words to count as the main content of the page.</p>
34+
</article>
35+
</body></html>"""
36+
37+
38+
def markdown(content_type: str = "text/markdown; charset=utf-8") -> httpx.Response:
39+
return httpx.Response(
40+
200, content=MARKDOWN.encode(), headers={"content-type": content_type}
41+
)
42+
43+
44+
async def call_fetch(web: FakeWeb, arguments: dict[str, object]) -> str:
45+
async with connect() as (session, _):
46+
result = await session.call_tool("fetch", {"url": PAGE, **arguments})
47+
data = result.model_dump(by_alias=True, mode="json", exclude_none=True)
48+
assert data["isError"] is False
49+
return data["content"][0]["text"]
50+
51+
52+
def page_requests(web: FakeWeb) -> list[httpx.Request]:
53+
return [r for r in web.requests if str(r.url) == PAGE]
54+
55+
56+
async def test_a_fetch_prefers_markdown_then_html(web: FakeWeb) -> None:
57+
web.add(ROBOTS, httpx.Response(404))
58+
web.add(PAGE, markdown())
59+
await call_fetch(web, {})
60+
[request] = page_requests(web)
61+
assert request.headers["accept"] == "text/markdown, text/html;q=0.9, */*;q=0.8"
62+
assert MARKDOWN_ACCEPT == request.headers["accept"]
63+
# The robots.txt lookup is not a page fetch and asks for nothing new.
64+
[robots] = [r for r in web.requests if str(r.url) == ROBOTS]
65+
assert robots.headers["accept"] == "*/*"
66+
67+
68+
@pytest.mark.parametrize(
69+
"content_type", ["text/markdown", "text/markdown; charset=utf-8", "Text/Markdown"]
70+
)
71+
async def test_markdown_is_returned_as_is_without_the_raw_prefix(
72+
web: FakeWeb, content_type: str
73+
) -> None:
74+
web.add(ROBOTS, httpx.Response(404))
75+
web.add(PAGE, markdown(content_type))
76+
text = await call_fetch(web, {})
77+
assert text == f"Contents of {PAGE}:\n{MARKDOWN}"
78+
79+
80+
async def test_a_raw_fetch_keeps_the_default_accept(web: FakeWeb) -> None:
81+
web.add(ROBOTS, httpx.Response(404))
82+
web.add(
83+
PAGE, httpx.Response(200, text=ARTICLE, headers={"content-type": "text/html"})
84+
)
85+
text = await call_fetch(web, {"raw": True})
86+
[request] = page_requests(web)
87+
assert request.headers["accept"] == "*/*"
88+
assert text.endswith(f"Contents of {PAGE}:\n{ARTICLE}")
89+
90+
91+
async def test_markdown_under_raw_is_unchanged_and_unprefixed(web: FakeWeb) -> None:
92+
# Markdown served unasked (a .md file, say) is the same either way.
93+
web.add(ROBOTS, httpx.Response(404))
94+
web.add(PAGE, markdown())
95+
text = await call_fetch(web, {"raw": True})
96+
assert text == f"Contents of {PAGE}:\n{MARKDOWN}"
97+
98+
99+
@pytest.mark.usefixtures("python_readability")
100+
async def test_html_from_a_site_that_ignores_accept_is_still_simplified(
101+
web: FakeWeb,
102+
) -> None:
103+
web.add(ROBOTS, httpx.Response(404))
104+
web.add(
105+
PAGE, httpx.Response(200, text=ARTICLE, headers={"content-type": "text/html"})
106+
)
107+
text = await call_fetch(web, {})
108+
assert "cannot be simplified" not in text
109+
assert "<article>" not in text
110+
assert "Hello World" in text
111+
112+
113+
async def test_the_prompt_returns_markdown_as_is(web: FakeWeb) -> None:
114+
web.add(PAGE, markdown())
115+
async with connect() as (session, _):
116+
result = await session.get_prompt("fetch", {"url": PAGE})
117+
message = result.model_dump(by_alias=True, mode="json")["messages"][0]
118+
assert message["content"]["text"].endswith(MARKDOWN)
119+
assert "cannot be simplified" not in message["content"]["text"]
120+
[request] = page_requests(web)
121+
assert request.headers["accept"] == MARKDOWN_ACCEPT
122+
123+
124+
@pytest.mark.parametrize(
125+
("content_type", "expected"),
126+
[
127+
("text/markdown", True),
128+
(" text/markdown ; variant=GFM", True),
129+
("text/markdownish", False),
130+
("text/x-markdown", False),
131+
("text/plain", False),
132+
("", False),
133+
],
134+
)
135+
def test_is_markdown(content_type: str, expected: bool) -> None:
136+
assert _is_markdown(content_type) is expected

0 commit comments

Comments
 (0)