Skip to content

Commit 5bcdc4b

Browse files
cliffhallclaude
andcommitted
fix(fetch): run Readability.js under the server's own bounded subprocess
readabilipy's Node.js wrapper had no timeout and ran on the event loop (modelcontextprotocol#5306), let `npm install` and a failed run's stderr reach stdout, the stdio JSON-RPC channel (modelcontextprotocol#5310), and ran a bare `npm` that Windows cannot resolve to npm.cmd (modelcontextprotocol#5313). The new readability_js module runs ExtractArticle.js itself: node and npm resolved with shutil.which, stdin from the null device, output captured and forwarded to stderr, a timeout on every process with the whole process tree killed on expiry, the dependency install done at most once per process under a lock, and any failure falling back to the pure-Python extractor. fetch_url runs the extraction in a worker thread. The README states that the first HTML extraction with Node.js installs Readability.js's dependencies from the npm registry, what that means for an egress allowlist and for pinning, and how to install them ahead of time (modelcontextprotocol#4830). The Windows CI leg no longer pre-installs them, so it exercises the server's own install through npm.cmd. Closes modelcontextprotocol#5306 Closes modelcontextprotocol#5310 Closes modelcontextprotocol#5313 Closes modelcontextprotocol#4830 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent cd3ceb8 commit 5bcdc4b

8 files changed

Lines changed: 845 additions & 57 deletions

File tree

‎README.md‎

Lines changed: 36 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ The server speaks both eras of the Model Context Protocol over stdio, with nothi
4646
## Installation
4747

4848
Optionally: Install node.js, this will cause the fetch server to use a different HTML simplifier that is more robust.
49+
Its JavaScript dependencies are then installed with npm on first use, which contacts the npm registry.
4950
To keep the server from ever starting Node.js, see [Customization - HTML extraction](#customization---html-extraction).
5051

5152
### Using uv (recommended)
@@ -213,11 +214,34 @@ installed with the server). A `socks://` proxy URL, as many desktop proxy settin
213214

214215
### Customization - HTML extraction
215216

216-
HTML is converted to Markdown by [readabilipy](https://github.com/alan-turing-institute/ReadabiliPy). When Node.js is
217-
on the `PATH`, readabilipy runs Mozilla's Readability.js in a Node.js subprocess; otherwise it uses its own
218-
pure-Python extractor, which keeps headings and text but drops link targets. The Node.js subprocess runs with no
219-
timeout, so a Node.js that hangs, or fails in a way readabilipy does not detect, stalls the tool call until the client
220-
gives up.
217+
HTML is converted to Markdown with [readabilipy](https://github.com/alan-turing-institute/ReadabiliPy). When Node.js 14
218+
or newer is on the `PATH`, the server runs Mozilla's Readability.js (readabilipy's `ExtractArticle.js`) in a Node.js
219+
subprocess; otherwise it uses readabilipy's pure-Python extractor, which keeps headings and text but drops link
220+
targets. The subprocess runs off the server's event loop, so other requests are answered while it works, and under a
221+
30 second timeout: a Node.js that hangs is killed, together with anything it started, and that page is extracted with
222+
the pure-Python extractor instead. If Readability.js fails in any other way, the same fallback is used. Every reason
223+
for a fallback is written to stderr (the server's log), never to stdout, which carries the protocol.
224+
225+
**Readability.js's dependencies are installed with npm on first use, which reaches the npm registry.** readabilipy
226+
ships only the `package.json` for its JavaScript (`@mozilla/readability`, `jsdom`, `minimist`). The first time the
227+
server extracts HTML with Node.js on the `PATH` and those packages missing, it runs `npm install` in readabilipy's
228+
`javascript` directory, inside that tool call, so:
229+
230+
- `registry.npmjs.org` belongs in the egress allowlist of a deployment that restricts outbound traffic, unless the
231+
packages are installed ahead of time (below) or Readability.js is turned off. Without it the install fails and the
232+
server uses the pure-Python extractor.
233+
- The JavaScript that runs is resolved from readabilipy's unpinned version ranges when it is installed, so pinning
234+
`mcp-server-fetch` does not pin it.
235+
- The install runs at most once per server process, under a 120 second timeout. Its output goes to stderr. If it
236+
fails, or Node.js is missing or older than 14, the server uses the pure-Python extractor until it is restarted.
237+
238+
To install the packages ahead of time (in a container image, say), run `npm install` in readabilipy's `javascript`
239+
directory from the Python environment the server runs in; the server then finds them and installs nothing:
240+
241+
```bash
242+
cd "$(python -c "import os, readabilipy; print(os.path.join(os.path.dirname(readabilipy.__file__), 'javascript'))")"
243+
npm install --no-audit --no-fund
244+
```
221245

222246
Readability.js discards hidden elements. Pages rendered progressively on the server often send a visible loading
223247
shell, then the real content in a container hidden (`display: none`, `visibility: hidden`, `hidden`) until scripts
@@ -226,7 +250,8 @@ Readability.js extracted, the server unhides it and runs Readability.js again, k
226250
text. Other pages are extracted once, exactly as before.
227251

228252
To always use the pure-Python extractor, add the argument `--no-readability-js` to the `args` list in the
229-
configuration. The server then never starts Node.js (or npm), whether or not it is installed.
253+
configuration. The server then never starts Node.js (or npm), whether or not it is installed, and never contacts the
254+
npm registry.
230255

231256
## Windows Configuration
232257

@@ -270,6 +295,10 @@ If you're experiencing timeout issues on Windows, you may need to set the `PYTHO
270295

271296
This addresses character encoding issues that can cause the server to timeout on Windows systems.
272297

298+
Readability.js works on Windows as it does elsewhere: the server finds Node.js and npm (`npm.cmd`) on the `PATH`, and
299+
installs Readability.js's dependencies on first use as described in
300+
[Customization - HTML extraction](#customization---html-extraction).
301+
273302
## Debugging
274303

275304
You can use the MCP inspector to debug the server. For uvx installations:
@@ -294,7 +323,7 @@ uv sync --frozen --all-extras --dev
294323
uv run --frozen pytest
295324
```
296325

297-
The tests drive `serve()` through an MCP `ClientSession` in-process (and, in `tests/test_eras.py`, through a `Client` as a 2026-07-28 and as a 2025-11-25 client), with HTTP mocked, so they need no network. The HTML extraction tests need Node.js on `PATH` (readabilipy uses Readability.js when Node is present and a pure-Python fallback when it is not).
326+
The tests drive `serve()` through an MCP `ClientSession` in-process (and, in `tests/test_eras.py`, through a `Client` as a 2026-07-28 and as a 2025-11-25 client), with HTTP mocked, so they need no network. The HTML extraction tests need Node.js and npm on `PATH` (the server uses Readability.js when Node is present and a pure-Python fallback when it is not), and the first run installs Readability.js's dependencies from the npm registry.
298327

299328
Coverage, measured per file with branch coverage:
300329

Lines changed: 259 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,259 @@
1+
# Runs Mozilla's Readability.js (readabilipy's `ExtractArticle.js`) under
2+
# Node.js in a subprocess the server controls, instead of through readabilipy's
3+
# own wrapper (`simple_json_from_html_string(use_readability=True)`).
4+
#
5+
# readabilipy's wrapper had four defects for a stdio MCP server, each fixed
6+
# here rather than worked around at the call site:
7+
#
8+
# - No timeout (#5306). Its `subprocess.run` calls wait forever, so a Node.js
9+
# or npm that hangs stalled the tool call. Every process here runs under a
10+
# timeout, and on timeout its whole process tree is killed (a process group
11+
# on POSIX, `taskkill /T` on Windows), so nothing is left running.
12+
# - Child output on fd 1 (#5310). Its `npm install` inherits the server's
13+
# stdout, which over stdio is the JSON-RPC channel, and it `print`s Node's
14+
# stderr to stdout on a failure. Here every child gets stdin from the null
15+
# device (it must never read the server's JSON-RPC input either) and pipes
16+
# for stdout and stderr; whatever a child prints is forwarded to stderr.
17+
# - npm not found on Windows (#5313). It runs a bare `npm`, which Windows'
18+
# `CreateProcess` does not resolve to `npm.cmd`. Here `node` and `npm` are
19+
# resolved with `shutil.which`, which honours `PATHEXT`.
20+
# - A failed Readability.js run raised out of the tool call. Here any failure
21+
# (no Node.js, an old one, no npm, a failed install, a timeout, a crash, bad
22+
# output) returns None, and the caller falls back to readabilipy's
23+
# pure-Python extractor, with the reason written to stderr.
24+
#
25+
# readabilipy >= 0.3.0 ships only `package.json` for its JavaScript and runs
26+
# `npm install` on first use, which reaches the npm registry during a tool call
27+
# (#4830). That stays the behaviour (the README states it), but the install now
28+
# runs at most once per process, under a lock and a timeout. It installs into
29+
# readabilipy's own `javascript` directory, so dependencies an operator
30+
# installed there ahead of time (the README says how) are found and nothing is
31+
# fetched.
32+
#
33+
# Everything here is synchronous: the caller runs extraction in a worker
34+
# thread (`asyncio.to_thread`), so the server's event loop never waits on it.
35+
36+
from __future__ import annotations
37+
38+
import json
39+
import os
40+
import shutil
41+
import signal
42+
import subprocess
43+
import sys
44+
import tempfile
45+
import threading
46+
from pathlib import Path
47+
48+
import readabilipy
49+
50+
# How long one Readability.js run may take before it is killed.
51+
EXTRACT_TIMEOUT_SECONDS = 30.0
52+
# How long the one-time `npm install` of readabilipy's JavaScript may take.
53+
INSTALL_TIMEOUT_SECONDS = 120.0
54+
# How long `node -v` may take.
55+
VERSION_TIMEOUT_SECONDS = 10.0
56+
# readabilipy's `package.json` declares `"engines": {"node": ">=14.0.0"}`.
57+
MIN_NODE_MAJOR = 14
58+
# What `ExtractArticle.js` requires; all three present means installed.
59+
JS_DEPENDENCIES = ("@mozilla/readability", "jsdom", "minimist")
60+
61+
_probe_lock = threading.Lock()
62+
_probed = False
63+
_node: str | None = None
64+
65+
66+
def javascript_dir() -> Path:
67+
"""readabilipy's `javascript` directory: `ExtractArticle.js` and `node_modules`."""
68+
return Path(readabilipy.__file__).parent / "javascript"
69+
70+
71+
def _log(message: str) -> None:
72+
# stderr only: stdout is the JSON-RPC channel.
73+
print(f"mcp-server-fetch: {message}", file=sys.stderr, flush=True)
74+
75+
76+
def _forward(output: str) -> None:
77+
"""Copy a child's captured output to stderr, never to stdout."""
78+
if output:
79+
sys.stderr.write(output if output.endswith("\n") else output + "\n")
80+
sys.stderr.flush()
81+
82+
83+
def _kill_tree(proc: subprocess.Popen[str]) -> None:
84+
"""Kill ``proc`` and everything it started (npm runs node, `npm.cmd` runs cmd)."""
85+
if sys.platform == "win32":
86+
subprocess.run(
87+
["taskkill", "/F", "/T", "/PID", str(proc.pid)],
88+
stdin=subprocess.DEVNULL,
89+
stdout=subprocess.DEVNULL,
90+
stderr=subprocess.DEVNULL,
91+
check=False,
92+
)
93+
else:
94+
try:
95+
# start_new_session made the child a process-group leader.
96+
os.killpg(proc.pid, signal.SIGKILL)
97+
except ProcessLookupError:
98+
pass
99+
# Whatever the tree kill did, the direct child is certainly gone after this.
100+
proc.kill()
101+
102+
103+
def _run(
104+
argv: list[str], timeout: float, cwd: Path | None = None
105+
) -> subprocess.CompletedProcess[str] | None:
106+
"""Run ``argv`` detached from the server's stdio, or None if it could not
107+
start or did not finish within ``timeout`` seconds (it is killed then)."""
108+
try:
109+
proc = subprocess.Popen(
110+
argv,
111+
cwd=cwd,
112+
stdin=subprocess.DEVNULL,
113+
stdout=subprocess.PIPE,
114+
stderr=subprocess.PIPE,
115+
text=True,
116+
encoding="utf-8",
117+
errors="replace",
118+
# POSIX only (ignored on Windows): its own process group, so a
119+
# timeout can kill the processes it starts as well.
120+
start_new_session=True,
121+
)
122+
except OSError as e:
123+
_log(f"could not start {argv[0]}: {e}")
124+
return None
125+
try:
126+
stdout, stderr = proc.communicate(timeout=timeout)
127+
except subprocess.TimeoutExpired:
128+
_kill_tree(proc)
129+
proc.communicate()
130+
_log(f"{Path(argv[0]).name} did not finish within {timeout:g} s and was killed")
131+
return None
132+
return subprocess.CompletedProcess(argv, proc.returncode, stdout, stderr)
133+
134+
135+
def _node_major(version: str) -> int | None:
136+
"""The major version in `node -v` output (``v24.1.0``), or None."""
137+
head = version.strip().lstrip("v").split(".", 1)[0]
138+
return int(head) if head.isdigit() else None
139+
140+
141+
def _dependencies_installed(jsdir: Path) -> bool:
142+
modules = jsdir / "node_modules"
143+
return all((modules / name / "package.json").is_file() for name in JS_DEPENDENCIES)
144+
145+
146+
def _install_dependencies(jsdir: Path) -> bool:
147+
"""`npm install` readabilipy's JavaScript dependencies; whether it worked."""
148+
npm = shutil.which("npm")
149+
if npm is None:
150+
_log(
151+
"npm was not found on PATH, so Readability.js's dependencies cannot be "
152+
f"installed in {jsdir}; using the pure-Python extractor"
153+
)
154+
return False
155+
_log(
156+
f"installing Readability.js's dependencies with npm in {jsdir} (first use; "
157+
"this contacts the npm registry)"
158+
)
159+
done = _run(
160+
[npm, "install", "--no-audit", "--no-fund"],
161+
timeout=INSTALL_TIMEOUT_SECONDS,
162+
cwd=jsdir,
163+
)
164+
if done is None:
165+
return False
166+
_forward(done.stdout)
167+
_forward(done.stderr)
168+
if done.returncode != 0 or not _dependencies_installed(jsdir):
169+
_log(
170+
f"npm install failed (exit {done.returncode}); using the pure-Python extractor"
171+
)
172+
return False
173+
return True
174+
175+
176+
def _find_node() -> str | None:
177+
"""A Node.js ready to run `ExtractArticle.js`, installing its dependencies
178+
if they are missing, or None (the reason is logged)."""
179+
node = shutil.which("node")
180+
if node is None:
181+
_log("Node.js was not found on PATH; using the pure-Python extractor")
182+
return None
183+
done = _run([node, "-v"], timeout=VERSION_TIMEOUT_SECONDS)
184+
major = _node_major(done.stdout) if done and done.returncode == 0 else None
185+
if major is None or major < MIN_NODE_MAJOR:
186+
_log(
187+
f"Node.js >= {MIN_NODE_MAJOR} is needed for Readability.js "
188+
f"({node} did not report one); using the pure-Python extractor"
189+
)
190+
return None
191+
jsdir = javascript_dir()
192+
if not _dependencies_installed(jsdir) and not _install_dependencies(jsdir):
193+
return None
194+
return node
195+
196+
197+
def node_path() -> str | None:
198+
"""The Node.js to run Readability.js with, or None to use pure Python.
199+
200+
Probed once per process, under a lock, so concurrent first calls run one
201+
`npm install` between them. A failed probe is not retried until the server
202+
restarts, so a missing or broken Node.js costs one probe, not one per call.
203+
"""
204+
global _probed, _node
205+
with _probe_lock:
206+
if not _probed:
207+
_node = _find_node()
208+
_probed = True
209+
return _node
210+
211+
212+
def extract_article(html: str) -> str | None:
213+
"""Readability.js's article HTML for ``html`` ("" when it finds none), or
214+
None when Readability.js is unavailable or failed, to fall back on."""
215+
node = node_path()
216+
if node is None:
217+
return None
218+
jsdir = javascript_dir()
219+
with tempfile.TemporaryDirectory(
220+
prefix="mcp-server-fetch-", ignore_cleanup_errors=True
221+
) as tmp:
222+
page = Path(tmp, "page.html")
223+
result = Path(tmp, "article.json")
224+
page.write_text(html, encoding="utf-8")
225+
done = _run(
226+
[
227+
node,
228+
str(jsdir / "ExtractArticle.js"),
229+
"-i",
230+
str(page),
231+
"-o",
232+
str(result),
233+
],
234+
timeout=EXTRACT_TIMEOUT_SECONDS,
235+
cwd=jsdir,
236+
)
237+
if done is None:
238+
_log(
239+
"Readability.js was stopped; using the pure-Python extractor for this "
240+
"page. Start the server with --no-readability-js to never use it."
241+
)
242+
return None
243+
if done.returncode != 0 or not result.is_file():
244+
_forward(done.stderr)
245+
_log(
246+
f"Readability.js failed (exit {done.returncode}); using the "
247+
"pure-Python extractor for this page"
248+
)
249+
return None
250+
try:
251+
article = json.loads(result.read_text(encoding="utf-8"))
252+
except ValueError:
253+
_log(
254+
"Readability.js wrote unreadable output; using the pure-Python extractor"
255+
)
256+
return None
257+
# `Readability.parse()` returns null for a page with no article.
258+
content = article.get("content") if isinstance(article, dict) else None
259+
return content if isinstance(content, str) else ""

0 commit comments

Comments
 (0)