Summary
mcp-server-fetch matches robots.txt rules against the request path exactly as the client spelled it, while a typical origin resolves the path (collapse duplicate slashes, decode separator/dot escapes, remove ./.. segments) before serving. With Disallow: /private, an autonomous fetch of https://example.com//private (and similar alternate spellings) therefore passes the robots check and returns the body of the page the site asked not to be crawled.
This is a policy-enforcement bypass of the control the server documents for model-initiated fetches (src/fetch/README.md, "Customization - robots.txt"). It is not an authentication bypass: the origin was already serving the content over plain HTTP.
Severity: Low. No auth bypass, no SSRF / private-address bypass, no RCE, no crash. robots.txt is advisory crawler policy; the impact is defeating the advertised autonomous-fetch guarantee.
GHSA-3jqg-r5pp-rmp2 was a redirect-hop robots re-check gap (fixed so each hop is checked). This report is a different root cause: even with per-hop checks present, the path comparison itself does not canonicalize. Tested on commit 36380287ed3b15d115b55f546617d723072b1296 (mcp-server-fetch 2026.11.4), which already includes the hop fix.
Affected
- Package:
mcp-server-fetch (PyPI), under src/fetch
- Tested: 2026.11.4, commit
36380287ed3b15d115b55f546617d723072b1296
- Comparison site:
src/fetch/src/mcp_server_fetch/server.py around the Protego can_fetch() call in check_may_autonomously_fetch_url() (about line 532 at that commit). Spot-checks of older revisions since robots support was added show the same non-canonical comparison; we are not claiming every intermediate release.
Reproduction (localhost only)
Uses the project's own FakeWeb harness — no network and no live site. Save as src/fetch/tests/test_repro_robots_bypass.py and run:
cd src/fetch && uv run --frozen pytest tests/test_repro_robots_bypass.py -v
import httpx
import pytest
from mcp.types import CallToolResult
from .harness import FakeWeb, connect
ROBOTS = "https://example.com/robots.txt"
PRIVATE = "the private page"
def page(body):
return httpx.Response(
200, content=body.encode(), headers={"content-type": "text/plain"}
)
def robots():
return page("User-agent: *\nDisallow: /private")
def text_of(result: CallToolResult) -> str:
return result.model_dump(by_alias=True, mode="json", exclude_none=True)[
"content"
][0]["text"]
async def test_control_is_refused(web: FakeWeb) -> None:
"""Control: the plain path is refused today, so the rule is enforced."""
web.add(ROBOTS, robots())
web.add("https://example.com/private", page(PRIVATE))
async with connect() as (session, _):
result = await session.call_tool("fetch", {"url": "https://example.com/private"})
assert result.is_error is True
assert PRIVATE not in text_of(result)
@pytest.mark.parametrize(
"url",
["https://example.com//private", "https://example.com/a%2f..%2fprivate"],
)
async def test_alternate_spelling_leaks(web: FakeWeb, url) -> None:
"""These spell the disallowed page differently; the origin still serves it."""
web.add(ROBOTS, robots())
web.add(url, page(PRIVATE))
async with connect() as (session, _):
result = await session.call_tool("fetch", {"url": url})
assert result.is_error is True, "the disallowed page was fetched"
assert PRIVATE not in text_of(result)
async def test_redirect_to_encoded_dots_leaks(web: FakeWeb) -> None:
"""A redirect target is not passed through Pydantic's AnyUrl."""
web.add(ROBOTS, robots())
web.add(
"https://example.com/go",
httpx.Response(302, headers={"location": "/%2e%2e/private"}),
)
web.add("https://example.com/%2e%2e/private", page(PRIVATE))
async with connect() as (session, _):
result = await session.call_tool("fetch", {"url": "https://example.com/go"})
assert result.is_error is True, "the disallowed page was fetched"
assert PRIVATE not in text_of(result)
On the tested revision the control passes and the three bypass tests fail (AssertionError: the disallowed page was fetched).
| Request path |
Path Protego compares |
Path HTTPX sends |
Path a normalizing origin serves |
/private |
/private |
/private |
/private (blocked) |
//private |
//private |
//private |
/private (fetched) |
/a%2f..%2fprivate |
/a%2F..%2Fprivate |
/a%2f..%2fprivate |
/private (fetched) |
/%2e%2e/private (via redirect Location) |
/../private |
/%2e%2e/private |
/private (fetched) |
We also confirmed against a local Python http.server origin (path resolution comparable to nginx/Apache-style serving): GET /private refused by robots; GET //private and the encoded-dot redirect target returned the private file on the wire. An origin that does not normalize returns 404, so there is no bypass there.
Notes on scope: /%2e%2e/private works through a redirect Location (Pydantic AnyUrl normalizes that form on a direct argument and correctly blocks it). //private works as a direct argument; as a redirect target HTTPX treats scheme-relative //private as host private. Cross-host redirects still re-check the target host's robots.txt, so an attacker-controlled site that 302s to https://victim.example/%2e%2e/private can miss the victim's rule the same way.
Impact assumptions
- Autonomous (model-initiated)
fetch with robots enforcement on (default); the intentional prompt-fetch robots exemption is out of scope.
- Attacker influences a URL the agent fetches (directly or via redirect).
- Target publishes a path-specific
Disallow, and its origin resolves the alternate spelling to that resource.
Suggested fix
Resolve the request path the way an origin does before matching rules, and require both the literal spelling and the resolved path to be allowed (and). Decode only separator/dot escapes (%2f/%2F/%2e/%2E), collapse duplicate slashes, and remove ./.. per RFC 3986 §5.2.4; leave ordinary escapes such as %20 alone. Derive the wire path with urlsplit/urlunsplit rather than from a URL object that HTTPX/Pydantic may already have normalized. Keep the existing can_fetch(str(url), …) check and add the second check — the change only ever adds refusals.
Happy to share a tested patch + regression file against the project's harness privately if useful.
This issue was identified with AI-assisted static code review and checked against the source before reporting; happy to answer questions or help with a fix.
Summary
mcp-server-fetchmatchesrobots.txtrules against the request path exactly as the client spelled it, while a typical origin resolves the path (collapse duplicate slashes, decode separator/dot escapes, remove./..segments) before serving. WithDisallow: /private, an autonomousfetchofhttps://example.com//private(and similar alternate spellings) therefore passes the robots check and returns the body of the page the site asked not to be crawled.This is a policy-enforcement bypass of the control the server documents for model-initiated fetches (
src/fetch/README.md, "Customization - robots.txt"). It is not an authentication bypass: the origin was already serving the content over plain HTTP.Severity: Low. No auth bypass, no SSRF / private-address bypass, no RCE, no crash.
robots.txtis advisory crawler policy; the impact is defeating the advertised autonomous-fetch guarantee.Distinct from GHSA-3jqg-r5pp-rmp2
GHSA-3jqg-r5pp-rmp2 was a redirect-hop robots re-check gap (fixed so each hop is checked). This report is a different root cause: even with per-hop checks present, the path comparison itself does not canonicalize. Tested on commit
36380287ed3b15d115b55f546617d723072b1296(mcp-server-fetch2026.11.4), which already includes the hop fix.Affected
mcp-server-fetch(PyPI), undersrc/fetch36380287ed3b15d115b55f546617d723072b1296src/fetch/src/mcp_server_fetch/server.pyaround theProtegocan_fetch()call incheck_may_autonomously_fetch_url()(about line 532 at that commit). Spot-checks of older revisions since robots support was added show the same non-canonical comparison; we are not claiming every intermediate release.Reproduction (localhost only)
Uses the project's own
FakeWebharness — no network and no live site. Save assrc/fetch/tests/test_repro_robots_bypass.pyand run:On the tested revision the control passes and the three bypass tests fail (
AssertionError: the disallowed page was fetched)./private/private/private/private(blocked)//private//private//private/private(fetched)/a%2f..%2fprivate/a%2F..%2Fprivate/a%2f..%2fprivate/private(fetched)/%2e%2e/private(via redirectLocation)/../private/%2e%2e/private/private(fetched)We also confirmed against a local Python
http.serverorigin (path resolution comparable to nginx/Apache-style serving):GET /privaterefused by robots;GET //privateand the encoded-dot redirect target returned the private file on the wire. An origin that does not normalize returns 404, so there is no bypass there.Notes on scope:
/%2e%2e/privateworks through a redirectLocation(PydanticAnyUrlnormalizes that form on a direct argument and correctly blocks it).//privateworks as a direct argument; as a redirect target HTTPX treats scheme-relative//privateas hostprivate. Cross-host redirects still re-check the target host's robots.txt, so an attacker-controlled site that 302s tohttps://victim.example/%2e%2e/privatecan miss the victim's rule the same way.Impact assumptions
fetchwith robots enforcement on (default); the intentional prompt-fetch robots exemption is out of scope.Disallow, and its origin resolves the alternate spelling to that resource.Suggested fix
Resolve the request path the way an origin does before matching rules, and require both the literal spelling and the resolved path to be allowed (
and). Decode only separator/dot escapes (%2f/%2F/%2e/%2E), collapse duplicate slashes, and remove./..per RFC 3986 §5.2.4; leave ordinary escapes such as%20alone. Derive the wire path withurlsplit/urlunsplitrather than from a URL object that HTTPX/Pydantic may already have normalized. Keep the existingcan_fetch(str(url), …)check and add the second check — the change only ever adds refusals.Happy to share a tested patch + regression file against the project's harness privately if useful.
This issue was identified with AI-assisted static code review and checked against the source before reporting; happy to answer questions or help with a fix.