Skip to content

Latest commit

 

History

History
145 lines (122 loc) · 7.9 KB

LOOKUP.md

File metadata and controls

145 lines (122 loc) · 7.9 KB

Lookup: has anyone else been shown this exact tool?

Chrome asks Safe Browsing about a site before it opens it. An MCP client can ask the same kind of question about a tool before an agent trusts it: has any public server ever shown this exact definition, and since when?

The drift feed has recorded every tool definition it read from the official MCP registry's servers -- npm packages it launched in a container, hosted servers it read as an anonymous client -- 230,763 distinct definitions at the 29 September 2026 checkpoint. It publishes all of them as static files: the whole record in one file, and the same record split into buckets.

heldfast verify does this for every tool in your lockfile. This page is the protocol, for any other client to do the same -- and what each way of asking tells whoever serves the files.

What an answer means

answer meaning
seen, since a date, on N servers this exact definition -- name, words, schemas, annotations, icons -- was shown publicly. One that thousands of people were shown was not crafted for you
not in the record as far as the public record goes, this definition exists only where you got it. Normal for a server you wrote or run privately; worth reading for one you installed

Not seen is not an accusation, and seen is not a clean bill of health: a public definition can still be a bad one. What the lookup tells you is whether you are looking at the same thing as everyone else.

Two ways to ask, and what each reveals

The whole record (lookup/all.json, the default for verify). One file, every fingerprint the log has seen. The client searches it locally. Whoever serves it learns that someone downloaded the record, and nothing about which tools or servers they have. It was about 32 MB on 2026-10-14 and grows as new definitions are logged. The same bytes are published gzipped as lookup/all.json.gz, about a third of the size, which is what verify asks for first. It keeps what it downloaded in your cache directory under the feed commit it read, since a commit's content never changes: verify against an unchanged feed downloads nothing.

In 16 parts (lookup/all-0.json.gz to lookup/all-f.json.gz), each the whole record for fingerprints starting with that hex character, in the same layout. heldfast from 2026-10-17 asks for the parts first, all sixteen, whatever it is looking for, so they reveal no more than the single file. They exist because the single file grows by about 0.4 MB a day towards the 90 MB every reader refuses past; each part is held to that bound alone. While the single file fits it is still written, for earlier clients; past it, it is no longer written, and an earlier client finds no whole record and says so (--lookup buckets still works for it).

all.json.gz is a run of gzip members, not one. It is the plain file compressed in pieces: the first line, one member for each bucket's lines, and the closing line. This is so that git, which stores the feed's history, can store each day as a small change: a gzip of the whole file shares no bytes with the one before it, and the file was most of the repository. Concatenated members are valid gzip (RFC 1952), and what they inflate to is byte for byte all.json, but a reader has to follow the members. gzip.GzipFile and gzip.decompress in Python, zcat and gzip -dc, and zlib.gunzipSync in Node do. A reader that stops after the first member, such as Python's zlib.decompressobj(31) used once, gets the first line and nothing else, and the JSON it parses is cut off; read all.json instead, or loop over unused_data. The file is about 3% larger than a single-member gzip. Every member is written with the same header (modification time 0, level 9, OS byte 255) and the same deflate settings, so a bucket that did not change compresses to the same bytes; if the runner's zlib changes, most buckets will compress to new bytes once, and the collector says so in its log.

Buckets (lookup/<abc>.json, verify --lookup buckets). The same record split 4,096 ways by the first three hex characters of the fingerprint, a few dozen definitions per bucket (a median of 29). One bucket hides one tool among its neighbours, the way Have I Been Pwned hides a password. It does not hide a server. A lookup asks about every tool a server has, at once, and the set of buckets those tools fall in is unique to that server for 85% of the 13,170 servers logged -- most of the rest are the same catalogue listed several times. Whoever serves the buckets, and anyone who can see those requests, can tell which public servers you use. Use buckets when that does not matter, or when downloading the whole record does.

A tool that is not in the record at all -- one you wrote -- is not revealed either way: its bucket name matches nothing anyone else has.

The protocol

  1. Fingerprint the tool exactly as the lockfile does: the tool digest in LOCK.md, lowercase hex. Implementations that already write .mcp-pin.lock have it on disk.
  2. Fetch the record, from https://raw.githubusercontent.com/rufat325/heldfast-feed/<commit>/lookup/all.json, (or lookup/all.json.gz, gzipped), where <commit> is the feed repository's main resolved to a commit -- git ls-remote https://github.com/rufat325/heldfast-feed refs/heads/main does that without the rate-limited REST API -- or main itself, if you do not need to record which state of the log you read. It is {"prefix_length": 3, "tools": {"<fingerprint>": {"first_seen": "YYYY-MM-DD", "servers": N}}}. Or, knowing what it reveals, fetch only lookup/<first three hex characters>.json, which is {"prefix": "abc", "tools": {...}} with the same entries.
  3. Search it locally. Present means seen; absent means not.

lookup/meta.json gives the prefix length and the count, so a client can check it is reading the layout it expects.

A client in 20 lines

import json, urllib.request

LOG = "https://raw.githubusercontent.com/rufat325/heldfast-feed/main/lookup"

# The whole record: nothing about your tools leaves this machine.
with urllib.request.urlopen(f"{LOG}/all.json", timeout=60) as resp:
    record = json.load(resp)["tools"]

# fingerprints from a lockfile:
lock = json.load(open(".mcp-pin.lock"))
for server in lock["servers"].values():
    for name, tool in (server.get("tools") or {}).items():
        row = record.get(tool["fingerprint"].lower())
        print(server["name"], name,
              f"public since {row['first_seen']} on {row['servers']} server(s)" if row
              else "never seen publicly")

The same in JavaScript is a fetch and a property lookup. Computing the fingerprint from a live tools/list is the one part with rules to follow: LOCK.md has them, with test vectors.

What it is built from, and its limits

  • Published daily by the feed's publishing job, from every catalogue it has held: those in the tree, and those retire took out of it, which it writes to lookup/retired/ first (the fingerprints, the first day, the servers). So a definition no longer in the tree keeps its date and its count. A bucket, and a line of the whole record, changes only when a new definition or a new server appears, so the files are stable and cache well.
  • Coverage is the registry's. Servers listed in the official MCP registry: npm packages with a stdio transport, and hosted servers that answer without credentials. A server that is not listed, or that needs a login, is not in it -- so "never seen" is expected for those.
  • The requests are ordinary HTTPS requests. Whoever serves the files -- GitHub today -- sees the address a request came from, for either way of asking.
  • One publisher. The record is derived from this project's feed, a git history anyone can fetch and recompute it from. Nothing yet has other parties independently reading the same servers; see TRANSPARENCY.md.