Skip to content

Commit 7e92b3a

Browse files
authored
feat!: port to MCP Python SDK v2 (2026-07-28 protocol revision) (#362)
Phases 1–5 of #360. Held in draft behind [python-sdk#3273](modelcontextprotocol/python-sdk#3273) until 2026-08-19; the shim described in "The release blocker, closed at our layer" ends the hold. **Merging cuts 3.0.0.** ## What this does Ports the server from MCP Python SDK 1.29 to 2.0.0, which implements the [2026-07-28 revision](https://modelcontextprotocol.io/specification/2026-07-28/changelog). The 1.x line is maintenance-only now. Most of the change is deletion. The features this project hand-rolled against 1.x internals are first-class in v2, so `subscriptions.py` drops ~350 lines: the URI-keyed registry of live `ServerSession` objects, the `WeakSet` bookkeeping that inferred disconnects from garbage collection, the concurrent fan-out with per-subscriber send timeouts and dead-session eviction, and the monkey-patch of `create_initialization_options` that advertised a capability the SDK hardcoded to `False`. Clients now opt in via `subscriptions/listen` and the SDK's `SubscriptionBus` owns delivery. What survives is the part the SDK cannot know: `MtimeWatcher`. The rest is mechanical: `FastMCP` → `MCPServer`, `version`/`instructions` as constructor kwargs (instructions now ride `server/discover`), transport config moved to the app builders, health routes onto the public `custom_route` hook, wire models to snake_case, and `ResourceTemplate` built via `from_function` so v2's parsed RFC 6570 template and `ResourceSecurity` policy are populated. ## Behavior changes a user can see **Notification kinds are now distinct.** A `.zim` file appearing or disappearing publishes `resources/list_changed`; a file replaced in place publishes `resources/updated` for its own URI. 1.x sent both as an `updated` for `zim://files`, because `resources/subscribe` gave clients no way to ask about list membership. **`zim://files` is therefore no longer a valid subscription target** — `list_changed` carries no URI. **`resources/subscribe` is no longer served.** A 2025-era client gets `-32601` and cannot receive change notifications; the rest of the handshake era (tools, resources, prompts) still works. This is a choice rather than a constraint — see the spike in the comments: the SDK keeps the method, and registering a handler through the public `add_request_handler` restores both the capability advertisement and real notification delivery to a legacy client. What it costs is the per-session subscriber registry this PR deletes. Subscriptions were HTTP-only before this change too, so no stdio client ever had the capability. **Results carry cache hints.** Registration-table endpoints get a one-hour TTL. Reads split by URI: `zim://{name}` and `zim://{name}/entry/{path}` are fixed by a sealed archive file and carry `OPENZIM_MCP_RESOURCE_CACHE_TTL_SECONDS` (new, default one hour), while `zim://files` is a live directory scan and stays bounded by the watcher's poll interval, so a cached listing is never staler than the server's own detection latency. The long TTL and subscriptions work together: a client on an open `subscriptions/listen` stream is told about a replacement immediately and re-reads. A client that is **not** subscribed — anything on stdio — can serve stale content until the TTL expires; set the knob to `0` to turn the override off. Scope is `private` throughout: payloads embed server-local paths. ## One deviation from the plan in #360 **The `sse` transport stays.** The spec deprecates HTTP+SSE and the SDK still serves it. Removing a working transport is a user-facing decision that deserves its own change and changelog entry, not a side effect of an SDK bump. Now tracked in #366. (The plan also called for dropping the legacy CORS headers. They stay, because the endpoint serves both eras — but note `Mcp-Method`/`Mcp-Name` are **required and enforced today**, so those entries are load-bearing rather than forward-compatibility. See the correction below.) ## Two bugs this nearly shipped **`SubscriptionBus.publish` is async.** The first version of `publish_change` called it without `await` — which delivers nothing, emits only a never-awaited warning, and would have passed every test as a silently dead subscriptions feature. mypy caught it. **The live subscription test drove `subscribe_resource`.** It could only have failed against this build, and invisibly, since `addopts` excludes live tests from every normal run and from CI. It now opens a real `subscriptions/listen` stream. ## A correction to an earlier claim on this PR An earlier revision of this description said mcp 2.0.0 "does not implement `Mcp-Method` / `Mcp-Name`". **That was wrong.** It came from a `rg` search that silently skipped the virtualenv because it is gitignored. They are implemented (`MCP_METHOD_HEADER` in `mcp/shared/inbound.py`) and enforced: a modern POST whose `Mcp-Method` disagrees with the body's method is rejected with `-32020`. The stale code comment is fixed in this branch. ## Verification - **3733 passed, 213 skipped, 0 failed**, against the real ZIM fixture corpus (3675/217 at first review, before the base refresh brought the corpus job in). Skip count matches `main` exactly. - **Live suite: 34 tests, 3 failures — the identical 3 that fail on `main`.** No live regressions. - End-to-end over real HTTP: a real server process, stateless `server/discover`, an open `subscriptions/listen` stream, and both notification kinds arriving with correct URIs after real file changes on disk. - Protocol conformance asserted over an in-memory client session in **both** eras: legacy `initialize` and modern `server/discover`. - Cache hints verified on the wire, which is how their era-gating surfaced: the SDK stamps them only for modern clients, and legacy sessions correctly see none. - black / isort / flake8 / mypy / bandit clean; docs rewritten and `test_docs_consistency` passing. ## Base refresh (2026-08-13) `origin/main` moved to 2.7.0 (#363, #364) after this branch's base, so main is merged in as of `ebbca89`. #364's identity kwargs landed in the 1.x `fastmcp_kwargs` dict this PR deletes; they are re-homed onto the v2 constructor call, which takes `website_url` and `icons` directly — plus `title` and `description`, the two v2-only fields on the same identity surface. Its `completion/complete` module carried over verbatim, and its test file re-wired onto the v2 stream pair (the 1.x memory-session helper no longer exists) with snake_case wire-model attributes. mypy caught a third near-miss in the same family as the two below: `Completion(hasMore=)` — runtime-valid via alias, wrong for a codebase that standardized on snake_case. Merged-tree verification: 3695 passed / 0 failed / 213 skipped against the real corpus, all linters clean. With #363, this PR's CI now also runs the corpus-backed job — the ~54 corpus-dependent tests that skipped silently in earlier runs cover exactly the resource-read and search paths this PR rewrites. ## Third review pass (2026-08-15) A review aimed at the paths tests don't reach — the branch was fully green at the time — found six defects and three pieces of drift. All fixed in `bb5e559`. **Validation errors on `resources/read` reached clients as "Internal server error."** A `zim://{name}/entry/{path}` whose path `sanitize_input` rejects (empty after sanitisation, or over `INPUT_LIMIT_ENTRY_PATH`) raised `OpenZimMcpValidationError`, which is neither `MCPError` nor a `ResourceError`. Nothing downstream converted it, because `ResourceManager.get_resource` invokes `create_resource` *outside* the `try` that wraps `resource.read()` — so `modern_error_data` replaced it wholesale with a generic `-32603`. The caller lost the only actionable part of the message and read a malformed request as a retryable server fault. Now `INVALID_PARAMS` with the text intact, converted at the only layer that can. This is the same failure the sibling `ResourceNotFoundError` conversion three lines above was added to prevent; the second call site was missed. **Replacement notifications never reached clients that subscribed with the spelling they had successfully read from.** `UriTemplate.match` percent-decodes before matching, so `zim://my archive` is a *working* `resources/read` — which means a client that built its URI from the `zim://files` listing rather than by RFC 6570 expansion gets a valid read and then subscribes with that same string. The watcher published only the encoded form, and SDK delivery is exact-string (`event_matches` does `event.uri in uris`), so that client sat on a URI nothing ever fires on: silence on every replacement, and with the new hour-long resource TTL, a stale overview for the life of the stream. Both spellings are now published when they differ; unreserved stems still publish exactly once. Note this is the flip side of the fix in `f547858` — that one made the *encoded* spelling deliver; this one covers the raw spelling that also reads. **`publish_change` mapped every unrecognised change kind to `resources/updated`.** Only `CHANGE_LIST_CHANGED` was compared; `CHANGE_REPLACED` was defined but never matched, so a third watcher kind added later (a removal carrying `zim://files`, say) would have gone out as an `updated` for that URI — reintroducing exactly the list-membership/content conflation the 2026-07-28 split exists to end, at the one seam nothing asserted on. Both kinds are matched explicitly now; anything else is dropped with a warning. **The per-stream subscription URI bound went missing, and the docstring claimed otherwise.** Deleting the 1.x registry took `MAX_URIS_TOTAL` / `MAX_URIS_PER_SESSION` with it, and this module's docstring said the SDK's `ListenHandler` was "the equivalent admission control." It is not: it caps concurrent streams (`max_subscriptions`) and per-stream backlog (`max_buffered_events`) and never inspects the requested URI set, which it then holds in a `frozenset` for the stream's lifetime. The real ceiling was 1024 streams × the transport's 4 MiB body cap — roughly three orders of magnitude looser than what it replaced, on a method the project's rate limiter does not cover (it wraps tool calls only). Restored as `install_bounded_listen_handler`, which swaps in a `ListenHandler` subclass that rejects an oversized set with `INVALID_PARAMS` before a slot is taken or an ack is sent — through the same private-registry seam already used to *withhold* the handler when subscriptions are off. The docstring now states what is and isn't bounded. **Title-mode promotion leaked the zero-result recovery hints onto the hit it produced.** The empty-page promotion branch popped the stale `reason="0_hits"` but kept `_meta.suggestions`, which `_assemble_find_response` fills only for the no-results and fuzzy cases — its stated rule being that a confident hit carries none "so confident matches aren't muddled by alt-spelling noise." A promoted canonical is exactly such a hit, so the model was handed "did you mean X?" alongside the answer, inviting a second lookup for a query already resolved. **An empty `M/<key>` metadata value rendered as a bare `## Content` heading.** The metadata presenter was a copy of `_render_entry_payload_text` that had dropped its `or "(No content)"` fallback (and diverged on the content-type default's casing), so an archive storing a key with an empty value produced output that read as a truncated response — the same silently-empty rendering the offset accounting was added to fix, in the surface that fix was written for. Both renderers now share one `_render_offset_and_body` builder, so the copy cannot drift again. Three pieces of drift alongside them: `serve_streamable_http` re-registered `/healthz` and `/readyz` on every call (`custom_route` appends unconditionally, so a retry after a failed bind stacked unreachable duplicates holding live closures for the process lifetime — the pre-port `_custom_starlette_routes.extend` had the same shape); the per-URI TTL decision fully re-parsed every overview body — metadata, namespace summary, and up to a 2000-character main-page preview — on the event loop to answer one boolean, now short-circuited by a substring probe that still falls through to the exact check; and `OPENZIM_MCP_SUBSCRIPTIONS_ENABLED`'s description still documented the removed `resources/subscribe` handlers, three website pages still named `EnvelopeAwareFastMCP`, and `TIMEOUTS.SUBSCRIPTION_SEND_SECONDS` still documented an `asyncio.gather` fan-out over a session registry that no longer exists. **Verification:** 16 new tests, each mutation-checked — every fix was reverted individually and its test confirmed to fail. **3733 passed / 0 failed / 213 skipped** against the real corpus (from 3717 before this pass; skip count unchanged). flake8 / isort / black / mypy / bandit clean, and the full pre-commit hook set passes. **Left unfixed, deliberately:** `completion/complete` recursively globs every allowed directory on every request and is not covered by the rate limiter — `list_zim_files_data` walks the tree to build its cache-invalidation signature *before* consulting its own cache, so even a warm listing rescans, and a picker sees one full walk per keystroke. A short TTL memo fixes the cost but fails `test_completion_reflects_archives_added_after_startup`, which deliberately pins live reads so an operator dropping in an archive sees it immediately. Bounding the cost means accepting a staleness window, which is a product decision rather than a bug fix, so it is left for its own change. (Also left: `scanned_count` is patched onto the payload by the W-namespace caller rather than being a parameter of the shared `_new_scheme_browse_payload` builder — a recurrence risk, but today's behavior is correct.) ## Ninth pass (2026-08-19): the sweep before the cut A four-dimension review — the new ping shim, docs drift, release mechanics, and a fresh-eyes tail over the full diff — with every finding adversarially verified, ran once the shim landed. The shim and the release mechanics came back clean. Four things did not: **Pagination stalled forever when a page landed entirely inside a whitespace run.** `paged_slice_length` — the single source both the footer hint and `_meta.more_at_offset` derive from — returned 0 for an all-whitespace mid-article slice (`lead` is 0 past the top of the article and `rstrip` empties the rest), so both told the caller to re-request the offset it was already at: an infinite loop for any client following either, on any preformatted body with a whitespace run one page wide. Deferral now has a floor: such a page consumes its full slice and ships the run verbatim — deferring it to the next page defers it to the same page forever, and dropping it would fuse the words on either side, the exact defect the reassembly suite exists to prevent. That suite's own `nxt > offset` assert now covers the case. **The suggestions cache kept its key while this release changed the payload.** The canonical-promotion fix (third pass) changed what `_generate_search_suggestions` produces, but `suggestions_data:v2b` survived — so an operator upgrading with `persistence_enabled` restored the 2.7.0 snapshot and kept serving the wrongly-promoted suggestions until TTL expiry, the embedded stat token blind to a change the archive didn't make. Bumped to `v2c` under the release's own rule (stated at the `entry:v3` and `browse_ns_data:v2e` bumps: a change to what a payload contains needs a new prefix), with a sweep-9 guard test pinning the retired spelling out of the source. **The dependency range promised what the code cannot keep.** `mcp>=2.0,<3.0` admits any 2.x, but `sdk_compat.py` and the handler-registry installs are surgery on 2.0.0 internals; CI resolves the lockfile and stays green either way, while fresh installs (pip/uvx, Smithery, Docker) resolve the newest wheel the range admits — the lockfile-masks-range-defects failure mode. Held to `<2.1`; raising the ceiling is a deliberate bump PR that re-audits each seam, and the ping-shim canary says when the biggest one can go. **Docs said things the branch made false.** The subscriptions page claimed delivery "no longer touches SDK internals" (the bounded listen handler — and now `sdk_compat.py` — are exactly that), told clients a raw non-ASCII stem "would never match" (the four-spelling publish exists precisely so it does), and presented the SDK's two stream caps as the only limits while the server's own pre-ack bound — 256 subscription URIs per stream, 2048 chars per URI, `INVALID_PARAMS` — went undocumented, including in the security page's defense-in-depth table, where the DoS-relevant numbers belong. And `FastMCP` — a name that does not exist in SDK v2 — survived in fourteen website references plus roughly twice that in code comments once a full grep ran; every one now names what ships, each behavior claim behind it re-verified against the installed 2.0.0 before renaming, with `sdk_compat.py`'s "FastMCP 4" (the third-party client framework) the one deliberate survivor. **Verification for this pass:** each behavioral fix mutation-checked (test red with the fix reverted, green with it). Full suite **4058 passed / 0 failed / 213 skipped** against the real corpus — skip count unchanged, the growth from the third pass's 3733 being the tests sweeps four through eight added. Live suite over real HTTP: **27 passed / 0 failed** (16 corpus-conditional skips), now including the modern-keepalive ping; the 3 failures earlier passes recorded did not reproduce. flake8 / isort / black / mypy / bandit and the full pre-commit hook set clean. ## The release blocker, closed at our layer (2026-08-19) **[python-sdk#3273](modelcontextprotocol/python-sdk#3273) held this PR in draft:** SDK 2.0.0's per-version method tables lack `("ping", "2026-07-28")`, so every keepalive from a modern-negotiated client — the clients this port exists to serve — died as `-32601` before the SDK's own ping handler was consulted. The gap is also wider than the issue states: `SERVER_RESULTS` lacks the row too, so patching only the request gate would have traded the clean error for an unhandled `KeyError` in the outbound sieve, mid-response. Both rows or neither. Upstream still has no fix (2.0.0 remains the only release; no commit touches the tables), so this ships the escape hatch the hold was defined with: `sdk_compat.install_ping_keepalive_shim()`, run at server construction, copies the 2025-11-25 ping rows into both tables — ping's wire shape did not change in 2026; the gap is a missing entry, not a missing schema, and reusing the previous revision's models keeps the shim off the `mcp_types._v*` internals. The tables are `MappingProxyType` views that the SDK binds as call-time *defaults*, so neither rebinding the module attribute nor passing `surface=` from our layer can reach the runner's calls; the one seam every consumer shares is the dict under the proxy, and that surgery is confined to `sdk_compat.py` and its tests. The wire result is spec-correct without further help: the sieve passes the empty result through and the runner's own era backfill stamps `resultType: "complete"` and the `serverInfo` `_meta`. **Retirement is enforced, not remembered.** A canary test captures the tables' pristine state at import — necessarily before any install can run — and fails the day the locked SDK defines the rows itself; deleting the shim is then a red build away, not a TODO. Wire tests assert the behavior in both eras in-memory, and the live suite now sends a keepalive over real HTTP against a spawned server process, where an in-memory test could not see an install that silently didn't happen. Mutation-checked both ways: with the install commented out, the in-memory test and the live HTTP test each fail with the exact upstream symptom (`-32601` from a real server). (Secondary upstream issue [#3300](modelcontextprotocol/python-sdk#3300) — a transport leaked per politely-DELETEd legacy session — remains open, slow, dual-era-path-only, and not release-blocking.) ## What merging means 1. **Dropping `resources/subscribe` for 2025-era clients ships as designed.** The decision the hold deferred is made; the `BREAKING CHANGE:` footer below is its changelog framing. 2. **Merging cuts 3.0.0.** `feat!` on 2.7.0 is a major bump — with the keepalive gap closed, the timing of this merge is deliberately the timing of the major release. ## Phase 5 status (verified 2026-08-11) - **`server.json` `$schema` needs no bump.** `2025-12-11` is still the only published registry revision — every 2026-dated URL 404s — and the registry schema does not encode MCP protocol revisions at all (zero occurrences of any protocol version string; it describes packaging and transport). The only unreleased draft change is a transport-URL template-variable relaxation, irrelevant to our stdio package entry. This checklist item rested on a false premise: server.json tracks the *registry's* schema evolution, not the protocol's. - **`.mcpb` re-cut: clean, and the listing does not change.** Built on the v2 runtime, the bundle captures all 8 tools and its `manifest.json` is byte-identical to the one `main` produces. This also exercises the legacy `initialize` handshake through a real subprocess — `build_mcpb.py` speaks `protocolVersion: 2025-06-18` — so it is independent end-to-end evidence of dual-era support. Worth noting because the only test covering that path, `test_capture_tools_live_matches_advanced_surface`, is `@pytest.mark.live` and runs in neither a default run nor CI. (The bundle's sha256 differs from main's purely because v2 emits tool-object keys in a different order; same keys, same values, same byte length.) - **CHANGELOG:** the repo squash-merges with `PR_BODY` as the commit message, so this description is the changelog source. The framing is the `BREAKING CHANGE:` footer below. BREAKING CHANGE: requires MCP Python SDK v2 (`mcp>=2.0.0,<2.1` — held to 2.0.x while `sdk_compat.py` and the handler-registry seams depend on SDK internals; raising the ceiling is a deliberate bump that re-audits them) and adopts the 2026-07-28 protocol revision. The server answers both protocol eras from one endpoint, so 2025-era clients keep working for tools, resources, and prompts with no client change. Two things do break for them. this build no longer serves `resources/subscribe`, so a legacy client gets `-32601` and cannot receive change notifications from it. That is this server's choice, not an SDK limitation: the lowlevel `Server` still accepts a handler for the method and advertises the capability when one is registered. Restoring it would mean restoring the per-session subscriber registry that this change deletes. Subscriptions have always been HTTP-only here, so no stdio client is affected. And `zim://files` is no longer a valid subscription target, because a `.zim` file appearing or disappearing is now published as `resources/list_changed`, which carries no URI. Clients on the 2026-07-28 revision subscribe through `subscriptions/listen`: ask for `resourcesListChanged` to hear about archives appearing or disappearing, and list `zim://{name}` under `resourceSubscriptions` to hear about in-place replacements.
1 parent ebbd78a commit 7e92b3a

123 files changed

Lines changed: 9823 additions & 1903 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.pre-commit-config.yaml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,15 @@ repos:
2424
# oneof_parse_benchmark) that are NOT pytest files; exclude every
2525
# non-``test_`` Python module there so editing one is committable.
2626
#
27-
# tests/ root carries a few support modules too. They are listed one
28-
# by one rather than blanket-excluded, so a genuinely misnamed test
29-
# module still trips the hook.
27+
# tests/ root carries a few support modules too, as does tests/live/
28+
# (``_stdio_helpers`` drives the JSON-RPC handshake the live modules
29+
# share). They are listed one by one rather than blanket-excluded, so
30+
# a genuinely misnamed test module still trips the hook.
3031
exclude: |
3132
(?x)^tests/(
3233
dispatch_eval/(?!test_).*\.py
3334
|(conftest_v2_fixtures|_promote_fixtures|zim_stubs)\.py
35+
|live/_stdio_helpers\.py
3436
)$
3537
3638
# Python code formatting — keep in lockstep with the dev-deps pin in

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ openzim-mcp/
106106
│ ├── content_processor.py # HTML→text, heading-id, link extraction
107107
│ ├── async_operations.py # asyncio helpers and timeouts
108108
│ ├── timeout_utils.py # Timeout primitives
109-
│ ├── subscriptions.py # MtimeWatcher and SubscriberRegistry
109+
│ ├── subscriptions.py # MtimeWatcher and publish_change (SDK subscription bus)
110110
│ ├── simple_tools.py # Simple-mode `zim_query` tool
111111
│ ├── intent_parser.py # Natural-language intent parsing
112112
│ ├── tool_schemas.py # Per-tool response TypedDicts

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636

3737
Built for research assistants, knowledge chatbots, and content-analysis systems that need *intelligent* access to vast knowledge repositories — not just a raw text dump. Smart navigation by namespace (articles, metadata, media), structure-aware retrieval (sections, tables of contents, related articles), full-text search with suggestions and multi-archive search, and link-graph extraction to map content relationships. Cached, paginated operations keep things responsive across massive archives; comprehensive input validation and path-traversal protection keep things safe.
3838

39-
Streamable HTTP transport, per-entry MCP resources with subscriptions, and dual Simple / Advanced modes are all built in.
39+
Streamable HTTP transport, per-entry MCP resources with live change notifications, and dual Simple / Advanced modes are all built in.
4040

4141
## Install
4242

@@ -116,7 +116,7 @@ For HTTP transport (long-running service with bearer auth, CORS, and health endp
116116

117117
- **8-tool advanced surface** — `zim_query`, `zim_search`, `zim_get`, `zim_get_section`, `zim_browse`, `zim_metadata`, `zim_links`, `zim_health`. Down from 22; advanced-mode schema drops from ~36KB to ~24.3KB, clearing the [MCP Tax](https://www.mmntm.net/articles/mcp-context-tax) pain band. [API reference →](https://cameronrye.github.io/openzim-mcp/docs/api-reference/)
118118
- **Streamable HTTP transport** — bearer-token auth, CORS, health endpoints, multi-arch Docker image. [HTTP & Docker deployment →](https://cameronrye.github.io/openzim-mcp/docs/http-and-docker-deployment/)
119-
- **Per-entry MCP resources + subscriptions** — `zim://{name}/entry/{path}` with native MIME types; clients subscribe and receive `notifications/resources/updated` when archives change. [Resources, prompts & subscriptions →](https://cameronrye.github.io/openzim-mcp/docs/resources-prompts-subscriptions/)
119+
- **Per-entry MCP resources + subscriptions** — `zim://{name}/entry/{path}` with native MIME types; clients open a `subscriptions/listen` stream and get `resources/list_changed` when a ZIM appears or disappears, `resources/updated` when one is replaced. [Resources, prompts & subscriptions →](https://cameronrye.github.io/openzim-mcp/docs/resources-prompts-subscriptions/)
120120
- **Simple-mode `zim_query`** — one natural-language tool that dispatches to the right operation, tuned for small-model deployment targets. [Quick start →](https://cameronrye.github.io/openzim-mcp/docs/quick-start/)
121121
- **Archive-type presets** — OpenZIM MCP detects the archive type (Wikipedia, Stack Exchange, and more) and auto-tunes retrieval and summarization for it — e.g. Stack Exchange dumps render as clean Q&A instead of vote-score noise. Operators can override the bundled defaults with a TOML file (`OPENZIM_MCP_PRESETS_OVERRIDE_PATH`).
122122
- **Native libzim introspection** — `zim_health(zim_file_path=...)` validates an archive's integrity (`Archive.check()` + checksum), and `zim_metadata` reports archive identity, full-text / title index capabilities, and an `M/Counter` mimetype breakdown. [API reference →](https://cameronrye.github.io/openzim-mcp/docs/api-reference/)

‎openzim_mcp/async_operations.py‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
from functools import partial
1212
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union, cast
1313

14+
from .meta import attach_meta
1415
from .zim_operations import ZimOperations
1516

1617
if TYPE_CHECKING:
@@ -795,11 +796,19 @@ async def get_archive_metadata_data(
795796
"has_fulltext_index": metadata_resp.get("has_fulltext_index"),
796797
"has_title_index": metadata_resp.get("has_title_index"),
797798
},
798-
"_meta": {},
799799
}
800800
counter_breakdown = metadata_resp.get("counter_breakdown")
801801
if counter_breakdown:
802802
result["counter_breakdown"] = counter_breakdown
803+
# The archive-type annotations are computed by the source metadata
804+
# call and ride its envelope; carry them onto the combined one so
805+
# they stay reachable on the Phase F surface.
806+
source_meta: Dict[str, Any] = dict(metadata_resp.get("_meta", {}) or {})
807+
attach_meta(
808+
result,
809+
detected_type=source_meta.get("detected_type"),
810+
detection_confidence=source_meta.get("detection_confidence"),
811+
)
803812
return cast("ArchiveMetadataResponse", result)
804813

805814
async def get_archive_validation_data(
@@ -844,9 +853,10 @@ async def get_health_data(
844853
asyncio.to_thread(_build_configuration_report, server),
845854
asyncio.to_thread(self._ops.list_zim_files_data, None),
846855
)
847-
return {
848-
"health": health, # type: ignore[typeddict-item]
849-
"configuration": configuration, # type: ignore[typeddict-item]
850-
"loaded_archives": loaded_archives, # type: ignore[typeddict-item]
851-
"_meta": {},
856+
result: Dict[str, Any] = {
857+
"health": health,
858+
"configuration": configuration,
859+
"loaded_archives": loaded_archives,
852860
}
861+
attach_meta(result)
862+
return cast("ServerHealthResponse", result)

‎openzim_mcp/bundle.py‎

Lines changed: 148 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616

1717
import logging
1818
import re
19+
from itertools import groupby
1920
from pathlib import Path
2021
from typing import TYPE_CHECKING, Any, Dict, Optional, cast
2122

@@ -39,7 +40,25 @@
3940
# v2d: SectionMeta grew ``heading_start`` — cached v2c bundles lack the
4041
# field, so the narrow-slice fix would silently keep serving the child
4142
# heading until TTL expiry without the key bump.
42-
_BUNDLE_KEY_PREFIX = "bundle:v2d"
43+
#
44+
# v2e: the heading LOCATOR changed, so the same entry now yields sections a
45+
# v2d bundle does not contain — an H1 whose title html2text italicises
46+
# (``# _2040_ (film)``) and a heading split across two lines by a ``<br>``
47+
# were both dropped before. Nothing else in the key encodes the extractor's
48+
# behaviour: it is keyed on the archive path, mtime, size, entry and render
49+
# mode, all unchanged by a code fix. So without this bump an operator running
50+
# with ``persistence_enabled`` restores a snapshot written by the previous
51+
# release and keeps getting ``heading_count: 0`` for every affected article
52+
# until the TTL expires — the fix silently inert on exactly the entries it
53+
# was written for. Same reasoning as the v2c bump above; the rule is that a
54+
# change to what a bundle CONTAINS needs a new prefix, not just a change to
55+
# its shape.
56+
# v2e -> v2f: the eighth pass fixed two more ways a real heading failed to be
57+
# located — inline markup abutting the text beside it, and a link to a
58+
# parenthetically-disambiguated article — so affected articles gain sections
59+
# they previously lacked. Same rule as the bumps above: a change to what a
60+
# bundle CONTAINS needs a new prefix.
61+
_BUNDLE_KEY_PREFIX = "bundle:v2f"
4362

4463

4564
def archive_stat_token(validated_path: Any) -> str:
@@ -109,8 +128,20 @@ def _loose_escaped_text(text: str) -> str:
109128
H1). Allowing an optional backslash before each character matches the
110129
escaped and unescaped forms alike. Used only by the relaxed fallback, so
111130
the larger pattern is paid only for headings the strict match missed.
131+
132+
A run of literal backslashes gets one bounded quantifier rather than a
133+
per-character optional escape: the surrounding inline-markup class also
134+
accepts backslashes, so per-character units let the engine split a run of
135+
n backslashes 2**n ways and backtrack for hours before conceding a miss.
112136
"""
113-
return "".join(r"\\?" + re.escape(ch) for ch in text)
137+
parts: list[str] = []
138+
for is_backslash, group in groupby(text, key=lambda ch: ch == "\\"):
139+
chars = "".join(group)
140+
if is_backslash:
141+
parts.append(rf"\\{{{len(chars)},{2 * len(chars)}}}")
142+
else:
143+
parts.append("".join(r"\\?" + re.escape(ch) for ch in chars))
144+
return "".join(parts)
114145

115146

116147
def _resolve_entry_html(
@@ -185,7 +216,26 @@ def _build_link_buckets(links_dict: Dict[str, Any]) -> LinkBuckets:
185216
# (``## [Linked](X) part``). The relaxed character-class pattern in
186217
# ``_compute_section_offsets`` can't tolerate the brackets/URL, so headings
187218
# containing an inline link were dropped from the bundle entirely.
188-
_MD_LINK_RE = re.compile(r"!?\[([^\]]*)\]\([^)]*\)")
219+
# ``(?:[^()\n\\]|\\.)*`` rather than ``[^)]*``: html2text backslash-escapes
220+
# parens inside a link target, so a heading linking to a
221+
# parenthetically-disambiguated article — ``Mercury_\(mythology\)``, which is
222+
# most disambiguated Wikipedia titles — stopped the old pattern at the first
223+
# ``)`` and left ``Mercury (mythology) "Mercury (mythology)")`` behind. That
224+
# never matches the soup-side visible text, so the heading was dropped, its
225+
# section vanished from the bundle, and the PRECEDING section's slice ran on
226+
# through it and swallowed its body. Each character belongs to exactly one
227+
# branch, so the alternation is ReDoS-safe — the same form
228+
# ``compact_format._MARKDOWN_LINK_RE`` already adopted for the CodeQL
229+
# ``py/redos`` finding.
230+
_MD_LINK_RE = re.compile(r"!?\[([^\]]*)\]\((?:[^()\n\\]|\\.)*\)")
231+
232+
233+
# Emphasis-shaped underscore runs: those leading or trailing a word, which
234+
# is the only position html2text emits them in for ``<i>``/``<em>``. An
235+
# underscore *between* two alphanumerics (``snake_case``) is literal text and
236+
# must survive — a blanket ``replace("_", "")`` would make the two sides of
237+
# the comparison disagree for any heading whose text really carries one.
238+
_MD_EMPHASIS_UNDERSCORE_RE = re.compile(r"(?<![^\W_])_+|_+(?![^\W_])")
189239

190240

191241
def _strip_md_inline_decorations(line: str) -> str:
@@ -195,13 +245,24 @@ def _strip_md_inline_decorations(line: str) -> str:
195245
links), drops emphasis/code markers, unescapes backslash-escaped
196246
punctuation, and collapses whitespace — the same visible text
197247
``_heading_visible_text`` produces from the soup side.
248+
249+
``_`` is html2text's marker for ``<i>``/``<em>``, and dropping only
250+
``*``/backtick left it in: Wikipedia italicises the work title in the
251+
H1 of every article about a film, book, album, or newspaper
252+
(``<h1><i>2040</i> (film)</h1>`` renders as ``# _2040_ (film)``), so
253+
this comparison saw ``'_2040_ (film)'`` against ``'2040 (film)'``, the
254+
locator missed, and the article's only heading — with its whole
255+
section — was dropped from the bundle. Applied to both sides of the
256+
comparison (see :func:`_match_decorated_heading_line`), so the
257+
word-boundary rule cannot desynchronise them.
198258
"""
199259
prev = None
200260
while prev != line:
201261
prev = line
202262
line = _MD_LINK_RE.sub(r"\1", line)
203263
line = line.replace("*", "").replace("`", "")
204264
line = re.sub(r"\\(.)", r"\1", line)
265+
line = _MD_EMPHASIS_UNDERSCORE_RE.sub("", line)
205266
return " ".join(line.split())
206267

207268

@@ -210,14 +271,83 @@ def _match_decorated_heading_line(
210271
) -> Optional[re.Match]:
211272
"""Last-resort heading locator: scan ``#`` lines of the right level and
212273
compare their link-stripped visible text against the heading text."""
213-
wanted = " ".join(text.split())
274+
# Normalise the soup side through the SAME reduction as the rendered
275+
# side. Only then is the emphasis-underscore rule above symmetric: a
276+
# heading whose text legitimately ends in ``_`` loses it on both sides
277+
# and still matches, instead of matching on neither.
278+
wanted = _strip_md_inline_decorations(text)
214279
line_re = re.compile(rf"^{'#' * level} (.+?)\s*$", re.MULTILINE)
215-
for m in line_re.finditer(rendered_markdown, cursor):
280+
candidates = list(line_re.finditer(rendered_markdown, cursor))
281+
for m in candidates:
216282
if _strip_md_inline_decorations(m.group(1)) == wanted:
217283
return m
284+
# Last resort within the last resort: compare with the spaces removed.
285+
#
286+
# html2text emits a space after emphasis it closes even when the source
287+
# had none, so ``<h2><b>Foo</b>bar</h2>`` renders as ``## **Foo** bar``
288+
# while the soup side reads ``Foobar``. The decoration-stripped forms are
289+
# then ``'Foo bar'`` and ``'Foobar'`` and the exact pass above misses, so
290+
# the section is dropped from the bundle entirely — ``view="toc"`` never
291+
# lists it and ``zim_get_section`` answers ``section_not_found`` for it.
292+
# Code spans do not gain the space, which is why a ``<code>`` fixture
293+
# matches either way and this went unnoticed.
294+
#
295+
# Only reached when the section would otherwise be lost, and the two
296+
# sides still have to agree on every non-space character, so this cannot
297+
# promote a match the exact pass would have rejected on content.
298+
squashed = wanted.replace(" ", "")
299+
if squashed:
300+
for m in candidates:
301+
if _strip_md_inline_decorations(m.group(1)).replace(" ", "") == squashed:
302+
return m
218303
return None
219304

220305

306+
def _locate_heading_text(
307+
rendered_markdown: str, level: int, text: str, cursor: int
308+
) -> Optional[re.Match]:
309+
"""Run the strict → relaxed → decorated matcher cascade for one spelling.
310+
311+
Split out so a heading can be retried under a second spelling (see
312+
``line_text`` in :func:`_compute_section_offsets`) without duplicating
313+
the cascade or reordering it.
314+
"""
315+
# Strict pattern first; relaxed fallback covers html2text decorating
316+
# the heading text with inline markup (italics, bold, code spans)
317+
# that the soup-level get_text() stripped — without the fallback
318+
# those sections are silently absent from the bundle and
319+
# ``get_section`` returns "not found".
320+
strict = re.compile(
321+
rf"^{'#' * level} {re.escape(text)}\s*$",
322+
re.MULTILINE,
323+
)
324+
match = strict.search(rendered_markdown, cursor)
325+
if match is None:
326+
# H17: the relaxed pattern previously read
327+
# ``[^\n]*{re.escape(text)}[^\n]*$`` — a substring match that
328+
# accidentally picked up a heading like ``## Notes and See also``
329+
# when the bundle was probing for ``See also``. Constrain the
330+
# prefix/suffix to inline-markup characters html2text actually
331+
# emits (``*``, ``_``, ``` ` ```, backslashes, whitespace) so
332+
# the relaxed branch only catches decorated-heading cases, not
333+
# any heading containing the text anywhere.
334+
# Inline markup (``**bold**`` etc.) is tolerated as a prefix/suffix
335+
# wrapper; ``_loose_escaped_text`` additionally tolerates html2text's
336+
# backslash-escaped interior punctuation (e.g. ``1\.`` for ``1.``).
337+
_MD_INLINE = r"[ \t\*_`\\]*"
338+
relaxed = re.compile(
339+
rf"^{'#' * level} {_MD_INLINE}{_loose_escaped_text(text)}"
340+
rf"{_MD_INLINE}\s*$",
341+
re.MULTILINE,
342+
)
343+
match = relaxed.search(rendered_markdown, cursor)
344+
if match is None:
345+
# Inline links in the heading (``## [Linked](X) part``) carry
346+
# brackets and a URL the relaxed character class can't cover.
347+
match = _match_decorated_heading_line(rendered_markdown, level, text, cursor)
348+
return match
349+
350+
221351
def _compute_section_offsets(
222352
rendered_markdown: str,
223353
headings: list[dict],
@@ -248,41 +378,19 @@ def _compute_section_offsets(
248378
id_source = h.get("id_source", "slug")
249379
if not text or not section_id:
250380
continue
251-
# Strict pattern first; relaxed fallback covers html2text decorating
252-
# the heading text with inline markup (italics, bold, code spans)
253-
# that the soup-level get_text() stripped — without the fallback
254-
# those sections are silently absent from the bundle and
255-
# ``get_section`` returns "not found".
256-
strict = re.compile(
257-
rf"^{'#' * level} {re.escape(text)}\s*$",
258-
re.MULTILINE,
259-
)
260-
match = strict.search(rendered_markdown, cursor)
261-
if match is None:
262-
# H17: the relaxed pattern previously read
263-
# ``[^\n]*{re.escape(text)}[^\n]*$`` — a substring match that
264-
# accidentally picked up a heading like ``## Notes and See also``
265-
# when the bundle was probing for ``See also``. Constrain the
266-
# prefix/suffix to inline-markup characters html2text actually
267-
# emits (``*``, ``_``, ``` ` ```, backslashes, whitespace) so
268-
# the relaxed branch only catches decorated-heading cases, not
269-
# any heading containing the text anywhere.
270-
# Inline markup (``**bold**`` etc.) is tolerated as a prefix/suffix
271-
# wrapper; ``_loose_escaped_text`` additionally tolerates html2text's
272-
# backslash-escaped interior punctuation (e.g. ``1\.`` for ``1.``).
273-
_MD_INLINE = r"[ \t\*_`\\]*"
274-
relaxed = re.compile(
275-
rf"^{'#' * level} {_MD_INLINE}{_loose_escaped_text(text)}"
276-
rf"{_MD_INLINE}\s*$",
277-
re.MULTILINE,
278-
)
279-
match = relaxed.search(rendered_markdown, cursor)
381+
match = _locate_heading_text(rendered_markdown, level, text, cursor)
280382
if match is None:
281-
# Inline links in the heading (``## [Linked](X) part``) carry
282-
# brackets and a URL the relaxed character class can't cover.
283-
match = _match_decorated_heading_line(
284-
rendered_markdown, level, text, cursor
285-
)
383+
# A ``<br>`` inside the heading becomes a real newline in the
384+
# rendered markdown, so only the text BEFORE it stays on the
385+
# heading line while ``text`` carries the whole subtree. Every
386+
# matcher above is line-anchored and none can bridge that, so
387+
# retry with the pre-break spelling ``_build_headings`` recorded
388+
# (absent unless the heading actually contains a ``<br>``).
389+
line_text = _normalize_heading_text(h.get("line_text", ""))
390+
if line_text and line_text != text:
391+
match = _locate_heading_text(
392+
rendered_markdown, level, line_text, cursor
393+
)
286394
if match is None:
287395
logger.warning(
288396
"Bundle: could not locate heading %r (level %d) in rendered markdown",
@@ -411,7 +519,7 @@ def extract_entry_bundle(
411519
# ``_extract_infobox`` decomposes the infobox, preserving the prior order
412520
# in which links were extracted ahead of infobox removal.
413521
content_root = select_main_content(soup)
414-
headings = _build_headings(content_root)
522+
headings = _build_headings(content_root, include_line_text=True)
415523
raw_links = content_processor.extract_html_links(str(content_root))
416524
link_buckets = _build_link_buckets(raw_links)
417525
infobox = _extract_infobox(content_root, content_processor)

0 commit comments

Comments
 (0)