Skip to content

mcp-server-fetch: robots.txt path-canonicalization bypass (e.g. //private fetched while Disallow: /private)

Low
cliffhall published GHSA-f9hh-cmx9-2f23 Nov 11, 2026

Package

pip mcp-server-fetch (pip)

Affected versions

>= 0.6.1, < 2026.11.11

Patched versions

2026.11.11

Description

Summary

mcp-server-fetch matches robots.txt rules against the request path exactly as the client spelled it, while a typical origin resolves the path (collapse duplicate slashes, decode separator/dot escapes, remove ./.. segments) before serving. With Disallow: /private, an autonomous fetch of https://example.com//private (and similar alternate spellings) therefore passes the robots check and returns the body of the page the site asked not to be crawled.

This is a policy-enforcement bypass of the control the server documents for model-initiated fetches (src/fetch/README.md, "Customization - robots.txt"). It is not an authentication bypass: the origin was already serving the content over plain HTTP.

Severity: Low. No auth bypass, no SSRF / private-address bypass, no RCE, no crash. robots.txt is advisory crawler policy; the impact is defeating the advertised autonomous-fetch guarantee.

Distinct from GHSA-3jqg-r5pp-rmp2

GHSA-3jqg-r5pp-rmp2 was a redirect-hop robots re-check gap (fixed so each hop is checked). This report is a different root cause: even with per-hop checks present, the path comparison itself does not canonicalize. Tested on commit 36380287ed3b15d115b55f546617d723072b1296 (mcp-server-fetch 2026.11.4), which already includes the hop fix.

Affected

  • Package: mcp-server-fetch (PyPI), under src/fetch
  • Tested: 2026.11.4, commit 36380287ed3b15d115b55f546617d723072b1296
  • Comparison site: src/fetch/src/mcp_server_fetch/server.py around the Protego can_fetch() call in check_may_autonomously_fetch_url() (about line 532 at that commit). Spot-checks of older revisions since robots support was added show the same non-canonical comparison; we are not claiming every intermediate release.

Reproduction (localhost only)

Uses the project's own FakeWeb harness — no network and no live site. Save as src/fetch/tests/test_repro_robots_bypass.py and run:

cd src/fetch && uv run --frozen pytest tests/test_repro_robots_bypass.py -v
import httpx
import pytest
from mcp.types import CallToolResult
from .harness import FakeWeb, connect

ROBOTS = "https://example.com/robots.txt"
PRIVATE = "the private page"

def page(body):
    return httpx.Response(
        200, content=body.encode(), headers={"content-type": "text/plain"}
    )

def robots():
    return page("User-agent: *\nDisallow: /private")

def text_of(result: CallToolResult) -> str:
    return result.model_dump(by_alias=True, mode="json", exclude_none=True)[
        "content"
    ][0]["text"]

async def test_control_is_refused(web: FakeWeb) -> None:
    """Control: the plain path is refused today, so the rule is enforced."""
    web.add(ROBOTS, robots())
    web.add("https://example.com/private", page(PRIVATE))
    async with connect() as (session, _):
        result = await session.call_tool("fetch", {"url": "https://example.com/private"})
    assert result.is_error is True
    assert PRIVATE not in text_of(result)

@pytest.mark.parametrize(
    "url",
    ["https://example.com//private", "https://example.com/a%2f..%2fprivate"],
)
async def test_alternate_spelling_leaks(web: FakeWeb, url) -> None:
    """These spell the disallowed page differently; the origin still serves it."""
    web.add(ROBOTS, robots())
    web.add(url, page(PRIVATE))
    async with connect() as (session, _):
        result = await session.call_tool("fetch", {"url": url})
    assert result.is_error is True, "the disallowed page was fetched"
    assert PRIVATE not in text_of(result)

async def test_redirect_to_encoded_dots_leaks(web: FakeWeb) -> None:
    """A redirect target is not passed through Pydantic's AnyUrl."""
    web.add(ROBOTS, robots())
    web.add(
        "https://example.com/go",
        httpx.Response(302, headers={"location": "/%2e%2e/private"}),
    )
    web.add("https://example.com/%2e%2e/private", page(PRIVATE))
    async with connect() as (session, _):
        result = await session.call_tool("fetch", {"url": "https://example.com/go"})
    assert result.is_error is True, "the disallowed page was fetched"
    assert PRIVATE not in text_of(result)

On the tested revision the control passes and the three bypass tests fail (AssertionError: the disallowed page was fetched).

Request path Path Protego compares Path HTTPX sends Path a normalizing origin serves
/private /private /private /private (blocked)
//private //private //private /private (fetched)
/a%2f..%2fprivate /a%2F..%2Fprivate /a%2f..%2fprivate /private (fetched)
/%2e%2e/private (via redirect Location) /../private /%2e%2e/private /private (fetched)

We also confirmed against a local Python http.server origin (path resolution comparable to nginx/Apache-style serving): GET /private refused by robots; GET //private and the encoded-dot redirect target returned the private file on the wire. An origin that does not normalize returns 404, so there is no bypass there.

Notes on scope: /%2e%2e/private works through a redirect Location (Pydantic AnyUrl normalizes that form on a direct argument and correctly blocks it). //private works as a direct argument; as a redirect target HTTPX treats scheme-relative //private as host private. Cross-host redirects still re-check the target host's robots.txt, so an attacker-controlled site that 302s to https://victim.example/%2e%2e/private can miss the victim's rule the same way.

Impact assumptions

  • Autonomous (model-initiated) fetch with robots enforcement on (default); the intentional prompt-fetch robots exemption is out of scope.
  • Attacker influences a URL the agent fetches (directly or via redirect).
  • Target publishes a path-specific Disallow, and its origin resolves the alternate spelling to that resource.

Suggested fix

Resolve the request path the way an origin does before matching rules, and require both the literal spelling and the resolved path to be allowed (and). Decode only separator/dot escapes (%2f/%2F/%2e/%2E), collapse duplicate slashes, and remove ./.. per RFC 3986 §5.2.4; leave ordinary escapes such as %20 alone. Derive the wire path with urlsplit/urlunsplit rather than from a URL object that HTTPX/Pydantic may already have normalized. Keep the existing can_fetch(str(url), …) check and add the second check — the change only ever adds refusals.

Happy to share a tested patch + regression file against the project's harness privately if useful.


This issue was identified with AI-assisted static code review and checked against the source before reporting; happy to answer questions or help with a fix.

Severity

Low

CVSS overall score

This score calculates overall vulnerability severity from 0 to 10 and is based on the Common Vulnerability Scoring System (CVSS).
/ 10

CVSS v3 base metrics

Attack vector
Network
Attack complexity
High
Privileges required
None
User interaction
Required
Scope
Unchanged
Confidentiality
Low
Integrity
None
Availability
None

CVSS v3 base metrics

Attack vector: More severe the more the remote (logically and physically) an attacker can be in order to exploit the vulnerability.
Attack complexity: More severe for the least complex attacks.
Privileges required: More severe if no privileges are required.
User interaction: More severe when no user interaction is required.
Scope: More severe when a scope change occurs, e.g. one vulnerable component impacts resources in components beyond its security scope.
Confidentiality: More severe when loss of data confidentiality is highest, measuring the level of data access available to an unauthorized user.
Integrity: More severe when loss of data integrity is the highest, measuring the consequence of data modification possible by an unauthorized user.
Availability: More severe when the loss of impacted component availability is highest.
CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:L/I:N/A:N

CVE ID

No known CVE

Weaknesses

Improper Resolution of Path Equivalence

The product is vulnerable to file system contents disclosure through path equivalence. Path equivalence involves the use of special characters in file and directory names. The associated manipulations are intended to generate multiple names for the same object. Learn more on MITRE.

Use of Incorrectly-Resolved Name or Reference

The product uses a name or reference to access a resource, but the name/reference resolves to a resource that is outside of the intended control sphere. Learn more on MITRE.