Skip to content

Commit 315ad50

Browse files
cliffhallMustafaKemal0146claude
committed
fix(fetch): refuse private, loopback and metadata addresses by default (modelcontextprotocol#4838)
fetch followed redirects with no egress check, so a model-supplied URL, or any redirect hop from a public one, could reach loopback, RFC 1918, link-local and cloud metadata addresses from the host running the server. An httpx request hook now resolves each request's host and refuses it when any address is not globally routable. httpx runs request hooks before every request it sends, so the robots.txt fetch, the page fetch and every redirect hop are all checked. --allow-private-ips turns the guard off for local development; the README documents it next to the existing CAUTION. The KNOWN BUG modelcontextprotocol#4838 pinning test now asserts the redirect is refused, and the test harness gains a fake DNS so no test resolves a real name. Ported from modelcontextprotocol#4890 by Mustafa Kemal Çıngıl, adapted: a CLI flag in place of the FETCH_ALLOW_PRIVATE_IPS environment variable, is_global in place of an explicit range list (which also covers 100.64.0.0/10), a non-blocking resolver, and one hook in place of a separate pre-check. Co-authored-by: Mustafa Kemal Çıngıl <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent 571e2b3 commit 315ad50

6 files changed

Lines changed: 269 additions & 16 deletions

File tree

‎src/fetch/README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ Requires MCP Python SDK 1.x (`mcp>=1.29.0,<2`). SDK 2.0 renamed APIs this server
1010

1111
> [!CAUTION]
1212
> This server can access local/internal IP addresses and may represent a security risk. Exercise caution when using this MCP server to ensure this does not expose any sensitive data.
13+
>
14+
> By default the server refuses to fetch private, loopback, link-local and other non-public addresses, including cloud metadata endpoints such as `169.254.169.254`, and checks every redirect hop the same way. The `--allow-private-ips` argument turns this guard off; see [Customization - Private addresses](#customization---private-addresses). The guard checks the addresses a hostname resolves to before each request, so it does not protect against DNS rebinding between that check and the connection, and a hostname the server cannot resolve itself (one only a proxy can resolve) is not checked.
1315
1416
The fetch tool will truncate the response, but by using the `start_index` argument, you can specify where to start the content extraction. This lets models read a webpage in chunks, until they find the information they need.
1517

@@ -156,6 +158,17 @@ By default, the server will obey a websites robots.txt file if the request came
156158
the request was user initiated (via a prompt). This can be disabled by adding the argument `--ignore-robots-txt` to the
157159
`args` list in the configuration.
158160

161+
### Customization - Private addresses
162+
163+
By default, the server refuses any request, including a redirect hop or a robots.txt fetch, whose host resolves to an
164+
address that is not globally routable: loopback (`127.0.0.0/8`, `::1`), private (`10.0.0.0/8`, `172.16.0.0/12`,
165+
`192.168.0.0/16`, `fc00::/7`), link-local (`169.254.0.0/16`, `fe80::/10`), shared address space (`100.64.0.0/10`),
166+
multicast, unspecified and reserved ranges. A hostname is refused if any one of its addresses is in those ranges.
167+
The client receives an error naming the refused address.
168+
169+
To fetch from local or internal hosts (a development server, an intranet page), add the argument
170+
`--allow-private-ips` to the `args` list in the configuration.
171+
159172
### Customization - User-agent
160173

161174
By default, depending on if the request came from the model (via a tool), or was user initiated (via a prompt), the

‎src/fetch/src/mcp_server_fetch/__init__.py‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,21 @@ def main():
1616
help="Ignore robots.txt restrictions",
1717
)
1818
parser.add_argument("--proxy-url", type=str, help="Proxy URL to use for requests")
19+
parser.add_argument(
20+
"--allow-private-ips",
21+
action="store_true",
22+
help="Allow fetching private, loopback and link-local addresses (refused by default)",
23+
)
1924

2025
args = parser.parse_args()
21-
asyncio.run(serve(args.user_agent, args.ignore_robots_txt, args.proxy_url))
26+
asyncio.run(
27+
serve(
28+
args.user_agent,
29+
args.ignore_robots_txt,
30+
args.proxy_url,
31+
args.allow_private_ips,
32+
)
33+
)
2234

2335

2436
if __name__ == "__main__":

‎src/fetch/src/mcp_server_fetch/server.py‎

Lines changed: 81 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
1+
import asyncio
2+
import ipaddress
3+
import socket
14
from typing import Annotated, Tuple
25
from urllib.parse import urlparse, urlunparse
36

7+
import httpx
48
import markdownify
59
import readabilipy.simple_json
610
from mcp.shared.exceptions import McpError
@@ -24,6 +28,57 @@
2428
DEFAULT_USER_AGENT_MANUAL = "ModelContextProtocol/1.0 (User-Specified; +https://github.com/modelcontextprotocol/servers)"
2529

2630

31+
async def _resolve_host(host: str) -> list[str]:
32+
"""Resolve ``host`` to every address the OS would hand the connector.
33+
34+
getaddrinfo also parses IP literals, including the shorthand forms such as
35+
``2130706433`` and ``127.1`` that a connector accepts as loopback.
36+
"""
37+
infos = await asyncio.get_running_loop().getaddrinfo(host, None)
38+
return [str(info[4][0]) for info in infos]
39+
40+
41+
def _is_public_address(address: str) -> bool:
42+
"""True when ``address`` is a globally routable unicast address.
43+
44+
``is_global`` is False for loopback, RFC 1918, link-local (169.254.0.0/16,
45+
which holds most cloud metadata endpoints), shared address space
46+
(100.64.0.0/10, which holds Alibaba Cloud's 100.100.100.200), unique local
47+
IPv6 (AWS's fd00:ec2::254), unspecified and reserved ranges.
48+
"""
49+
ip = ipaddress.ip_address(address)
50+
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
51+
ip = ip.ipv4_mapped
52+
return ip.is_global and not ip.is_multicast
53+
54+
55+
async def _refuse_private_destination(request: httpx.Request) -> None:
56+
"""httpx request hook: refuse a request whose host is not a public address.
57+
58+
httpx runs request hooks before every request it sends, redirect hops
59+
included, so a public URL cannot bounce the fetch to an internal one.
60+
"""
61+
host = request.url.host
62+
if not host:
63+
return # httpx rejects a URL with no host itself.
64+
try:
65+
addresses = await _resolve_host(host)
66+
except socket.gaierror:
67+
# Unresolvable here: a direct connection fails on its own, and a proxy
68+
# resolves the name on its side.
69+
return
70+
for address in addresses:
71+
if not _is_public_address(address):
72+
raise McpError(
73+
ErrorData(
74+
code=INVALID_PARAMS,
75+
message=f"Refused to fetch {request.url}: {host} resolves to {address}, "
76+
"which is not a public address. Start the server with "
77+
"--allow-private-ips to allow private, loopback and link-local addresses.",
78+
)
79+
)
80+
81+
2782
def extract_content_from_html(html: str) -> str:
2883
"""Extract and convert HTML content to Markdown format.
2984
@@ -64,7 +119,10 @@ def get_robots_txt_url(url: str) -> str:
64119

65120

66121
async def check_may_autonomously_fetch_url(
67-
url: str, user_agent: str, proxy_url: str | None = None
122+
url: str,
123+
user_agent: str,
124+
proxy_url: str | None = None,
125+
allow_private_ips: bool = False,
68126
) -> None:
69127
"""
70128
Check if the URL can be fetched by the user agent according to the robots.txt file.
@@ -75,6 +133,8 @@ async def check_may_autonomously_fetch_url(
75133
robot_txt_url = get_robots_txt_url(url)
76134

77135
async with AsyncClient(proxy=proxy_url) as client:
136+
if not allow_private_ips:
137+
client.event_hooks = {"request": [_refuse_private_destination]}
78138
try:
79139
response = await client.get(
80140
robot_txt_url,
@@ -117,14 +177,20 @@ async def check_may_autonomously_fetch_url(
117177

118178

119179
async def fetch_url(
120-
url: str, user_agent: str, force_raw: bool = False, proxy_url: str | None = None
180+
url: str,
181+
user_agent: str,
182+
force_raw: bool = False,
183+
proxy_url: str | None = None,
184+
allow_private_ips: bool = False,
121185
) -> Tuple[str, str]:
122186
"""
123187
Fetch the URL and return the content in a form ready for the LLM, as well as a prefix string with status information.
124188
"""
125189
from httpx import AsyncClient, HTTPError
126190

127191
async with AsyncClient(proxy=proxy_url) as client:
192+
if not allow_private_ips:
193+
client.event_hooks = {"request": [_refuse_private_destination]}
128194
try:
129195
response = await client.get(
130196
url,
@@ -194,13 +260,16 @@ async def serve(
194260
custom_user_agent: str | None = None,
195261
ignore_robots_txt: bool = False,
196262
proxy_url: str | None = None,
263+
allow_private_ips: bool = False,
197264
) -> None:
198265
"""Run the fetch MCP server.
199266
200267
Args:
201268
custom_user_agent: Optional custom User-Agent string to use for requests
202269
ignore_robots_txt: Whether to ignore robots.txt restrictions
203270
proxy_url: Optional proxy URL to use for requests
271+
allow_private_ips: Allow fetching private, loopback and link-local
272+
addresses, which are refused by default
204273
"""
205274
server = Server("mcp-fetch")
206275
user_agent_autonomous = custom_user_agent or DEFAULT_USER_AGENT_AUTONOMOUS
@@ -243,11 +312,15 @@ async def call_tool(name, arguments: dict) -> list[TextContent]:
243312

244313
if not ignore_robots_txt:
245314
await check_may_autonomously_fetch_url(
246-
url, user_agent_autonomous, proxy_url
315+
url, user_agent_autonomous, proxy_url, allow_private_ips
247316
)
248317

249318
content, prefix = await fetch_url(
250-
url, user_agent_autonomous, force_raw=args.raw, proxy_url=proxy_url
319+
url,
320+
user_agent_autonomous,
321+
force_raw=args.raw,
322+
proxy_url=proxy_url,
323+
allow_private_ips=allow_private_ips,
251324
)
252325
original_length = len(content)
253326
if args.start_index >= original_length:
@@ -277,7 +350,10 @@ async def get_prompt(name: str, arguments: dict | None) -> GetPromptResult:
277350

278351
try:
279352
content, prefix = await fetch_url(
280-
url, user_agent_manual, proxy_url=proxy_url
353+
url,
354+
user_agent_manual,
355+
proxy_url=proxy_url,
356+
allow_private_ips=allow_private_ips,
281357
)
282358
# TODO: after SDK bug is addressed, don't catch the exception
283359
except McpError as e:

‎src/fetch/tests/harness.py‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,16 @@
77
go through the SDK's validation and error mapping exactly as a client would.
88
- ``web`` replaces ``httpx.AsyncClient`` with one backed by an
99
``httpx.MockTransport``, so no test touches the network. It records every
10-
request and the keyword arguments each client was constructed with.
10+
request and the keyword arguments each client was constructed with. It also
11+
replaces the server's DNS lookup (``_resolve_host``) with ``FakeWeb.dns``:
12+
a hostname listed there resolves to its addresses, an IP literal resolves to
13+
itself, and any other name resolves to ``PUBLIC_IP``.
1114
"""
1215

1316
from __future__ import annotations
1417

18+
import ipaddress
19+
import socket
1520
import tempfile
1621
from contextlib import asynccontextmanager
1722
from dataclasses import dataclass, field
@@ -30,6 +35,10 @@
3035

3136
Responder = Union[httpx.Response, Callable[[httpx.Request], httpx.Response]]
3237

38+
# A globally routable address (documentation ranges are not global, so they
39+
# would be refused by the private-address guard).
40+
PUBLIC_IP = "93.184.215.14"
41+
3342

3443
@dataclass
3544
class FakeWeb:
@@ -38,6 +47,9 @@ class FakeWeb:
3847
routes: dict[str, Responder] = field(default_factory=dict)
3948
requests: list[httpx.Request] = field(default_factory=list)
4049
client_kwargs: list[dict[str, Any]] = field(default_factory=list)
50+
# hostname -> addresses; an empty list makes the name unresolvable.
51+
dns: dict[str, list[str]] = field(default_factory=dict)
52+
lookups: list[str] = field(default_factory=list)
4153

4254
def add(self, url: str, response: Responder) -> None:
4355
self.routes[url] = response
@@ -54,6 +66,17 @@ def handle(self, request: httpx.Request) -> httpx.Response:
5466
def urls(self) -> list[str]:
5567
return [str(r.url) for r in self.requests]
5668

69+
async def resolve(self, host: str) -> list[str]:
70+
self.lookups.append(host)
71+
try:
72+
return [str(ipaddress.ip_address(host))]
73+
except ValueError:
74+
pass
75+
addresses = self.dns.get(host, [PUBLIC_IP])
76+
if not addresses:
77+
raise socket.gaierror(socket.EAI_NONAME, "Name or service not known")
78+
return addresses
79+
5780

5881
@pytest.fixture
5982
def web(monkeypatch: pytest.MonkeyPatch) -> FakeWeb:
@@ -71,6 +94,7 @@ def factory(**kwargs: Any) -> httpx.AsyncClient:
7194
# server.py imports AsyncClient from httpx inside each function, so
7295
# patching the attribute on the httpx module is what it picks up.
7396
monkeypatch.setattr(httpx, "AsyncClient", factory)
97+
monkeypatch.setattr(server_module, "_resolve_host", fake.resolve)
7498
return fake
7599

76100

‎src/fetch/tests/test_entrypoints.py‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ def run_main(monkeypatch: pytest.MonkeyPatch, *argv: str) -> AsyncMock:
3131

3232
def test_main_defaults(monkeypatch: pytest.MonkeyPatch) -> None:
3333
serve = run_main(monkeypatch)
34-
serve.assert_awaited_once_with(None, False, None)
34+
serve.assert_awaited_once_with(None, False, None, False)
3535

3636

3737
def test_main_passes_every_flag(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -42,8 +42,11 @@ def test_main_passes_every_flag(monkeypatch: pytest.MonkeyPatch) -> None:
4242
"--ignore-robots-txt",
4343
"--proxy-url",
4444
"http://proxy.example.com:8080",
45+
"--allow-private-ips",
46+
)
47+
serve.assert_awaited_once_with(
48+
"MyBot/1.0", True, "http://proxy.example.com:8080", True
4549
)
46-
serve.assert_awaited_once_with("MyBot/1.0", True, "http://proxy.example.com:8080")
4750

4851

4952
def test_main_help_exits_zero(
@@ -54,7 +57,12 @@ def test_main_help_exits_zero(
5457
assert excinfo.value.code == 0
5558
out = capsys.readouterr().out
5659
assert "give a model the ability to make web requests" in out
57-
for flag in ("--user-agent", "--ignore-robots-txt", "--proxy-url"):
60+
for flag in (
61+
"--user-agent",
62+
"--ignore-robots-txt",
63+
"--proxy-url",
64+
"--allow-private-ips",
65+
):
5866
assert flag in out
5967

6068

@@ -82,7 +90,7 @@ def test_package_init_run_as_main_calls_main(monkeypatch: pytest.MonkeyPatch) ->
8290
monkeypatch.setattr(mcp_server_fetch.server, "serve", serve)
8391
monkeypatch.setattr(sys, "argv", ["mcp-server-fetch"])
8492
runpy.run_module("mcp_server_fetch.__init__", run_name="__main__")
85-
serve.assert_awaited_once_with(None, False, None)
93+
serve.assert_awaited_once_with(None, False, None, False)
8694

8795

8896
def console_script() -> Path:

0 commit comments

Comments
 (0)