Checks that a quote an AI cites really appears word for word in the source document, before the answer reaches the user.
Who it's for: developers whose chatbot or RAG system (a bot that answers from your documents) shows citations. What you get: a free, instant verdict per quote (found, not found, or quoted from the wrong document), plus an optional second check by a model of your choice.
- Install Python 3.9 or newer.
- Run
pip install "git+https://github.com/tonydzi/verbatim-citation-gate". - Run the short example under "Quickstart - 30 seconds" in Details below. Or, from a clone of the repo, run
pip install -e .and thenpython examples/demo.py(the demo fails withModuleNotFoundErrorif the package is not installed).
Done when: a made-up quote prints not_found and a real quote prints found. No API key or internet is needed for this first check.
Want an AI agent to do it for you? Paste this to Claude Code / Codex / Cursor: "Read the 'For AI agents' section of https://github.com/tonydzi/verbatim-citation-gate and set it up for me."
- Python 3.9 to 3.13, git. The gate has zero dependencies. Not on PyPI: install from the repository.
- The optional judge stage needs a model you already use. Ask the human which model and where its API key lives. Never invent a key.
pip install "git+https://github.com/tonydzi/verbatim-citation-gate"
# from a clone, with the test extra:
pip install -e ".[test]"- Stage 1 needs nothing:
quote_gate(quote, doc_id, docs)wheredocsis a dict of doc id to full text. - Stage 2 is optional: pass
llm_call=<function(system, user) -> str>toaudit_citation(...). Wire it to the human's own model client. Seeexamples/quickstart.py. - Unparseable judge output fails closed: it never counts as support.
python -c "from verbatim_citation_gate import quote_gate; d={'a':'Body weight was unchanged in both arms.'}; print(quote_gate('Body weight was unchanged in both arms.','a',d), quote_gate('Dramatic weight loss.','a',d))"
# expected: found not_found
pytest -q # from a clone, after the test extra: 45 passedWithout installing, from a clone: PYTHONPATH=src python examples/demo.py (PowerShell: $env:PYTHONPATH="src"; python examples/demo.py) prints, among other lines, 4 citations audited - 3 rejected for free, 1 reached the model and exits 0. Without PYTHONPATH and without installing it exits 1 with ModuleNotFoundError. Both Verify commands can fail: the one-liner prints something other than found not_found if the gate is broken, and pytest -q reports failures; pytest needs pip install -e ".[test]" first (or PYTHONPATH=src python -m pytest -q tests).
pip uninstall verbatim-citation-gate- Do not add dependencies to the gate and do not loosen the fail-closed rule.
- Keep API keys out of the repo, out of commits and out of logs.
- A
foundverdict only means the words exist in the document. It does not mean the claim is true. - Publishing to PyPI or bumping the version needs the human's approval.
Catch fabricated RAG citations before they reach the user. A two-stage, framework-agnostic auditor for quote-style citations.
The deterministic half is src/verbatim_citation_gate/gate.py: zero dependencies, zero tokens.
The model half is src/verbatim_citation_gate/judge.py, and plugs into whatever LLM your pipeline already speaks.
pip install "git+https://github.com/tonydzi/verbatim-citation-gate"from verbatim_citation_gate import quote_gate
docs = {"veltranib-rct": "... Body weight was unchanged in both arms."}
quote_gate("Body weight was unchanged in both arms.", "veltranib-rct", docs) # 'found'
quote_gate("Veltranib produced dramatic weight loss.", "veltranib-rct", docs) # 'not_found'That is stage 1, and it is the whole offline half: no key, no network, no tokens.
The run in the animation above is examples/demo.py β python examples/demo.py
reproduces it, model calls counted by the demo itself.
Stage 2 (the judge) needs a model and is one argument away β see Use below.
π Docs: https://tonydzi.github.io/verbatim-citation-gate/ β the two stages, the API, the verdicts, and every known limit with its issue.
RAG systems love to cite. The problem is how they fail β a quote that reads as authoritative and word-for-word can still be one of three things, each with its own case in tests/test_gate.py:
- fabricated β a plausible sentence that appears in no source (
test_fabrication_is_not_found), - a frankenquote β every word is real, but the sentence was stitched together and never actually written (
test_frankenquote_is_not_found), or - misattributed β a real quote lifted from a different document (
test_misattributed_real_quote_from_wrong_doc).
A judge model, asked "does this quote support this claim?", waves all three through: they read as fluent and supportive.
The fix is to ask the cheaper, prior question first, deterministically β that is src/verbatim_citation_gate/gate.py.
βββββββββββββββββββββββββββ
claim + quote β β 1. verbatim gate (free) β β not_found / misattributed (rejected, 0 tokens)
+ doc id βββββββββββββββ¬ββββββββββββ
β found
βΌ
βββββββββββββββββββββββββββ
β 2. skeptical judge (LLM) β β supports / partial / unrelated / contradicts
βββββββββββββββββββββββββββ
Stage 1 β the verbatim gate, all of it in src/verbatim_citation_gate/gate.py.
Pure re + substring matching over a normalized form: case, smart quotes, dashes and whitespace folded, numbers and % preserved (test_normalize_preserves_numbers_and_percent in tests/test_gate.py).
It cannot be sweet-talked, and every fabrication it rejects costs zero model calls β proven by test_audit_fabrication_never_calls_judge in tests/test_gate.py.
Frankenquotes fail because the match is contiguous, not bag-of-words, and a real quote from the wrong document is surfaced as misattributed rather than collapsed into not_found β the two need different fixes upstream (src/verbatim_citation_gate/gate.py).
Stage 2 β the skeptical judge, all of it in src/verbatim_citation_gate/judge.py.
For quotes that do exist, a burden-of-proof prompt decides whether the passage actually establishes the claim. All three rules below live in that file's prompt:
- Default-refute (
judge.py) β the verdict starts at unsupported; the quote must earnsupports, and ties break against the claim. - Outside knowledge is inadmissible (
judge.py) β a claim can be true in the world and still unsupported by this source. - Full-strength support (
judge.py) β same population, direction, magnitude and certainty, or the verdict caps atpartial(subgroupβeveryone, correlationβcausation, "may"β"does").
If the judge's output does not parse it fails closed (src/verbatim_citation_gate/judge.py):
an unverifiable citation never counts as support.
Not on PyPI yet β install from the repository:
pip install "git+https://github.com/tonydzi/verbatim-citation-gate"
# from a clone, with the test extra:
pip install -e ".[test]" && pytest -qPure stdlib: the gate pulls in nothing, and the judge takes whatever llm_call
you already have.
from verbatim_citation_gate import audit_citation
docs = {"veltranib-rct": "... Body weight was unchanged in both arms."}
# Stage 1 only β no model, no key, fully offline:
audit_citation("Body weight did not change.", "veltranib-rct",
"Body weight was unchanged in both arms.", docs) # -> "found"
# Both stages β supply any model as llm_call(system, user) -> str:
def llm_call(system, user):
return client.messages.create( # Anthropic shown; any vendor works
model="claude-haiku-4-5", system=system, max_tokens=1024,
messages=[{"role": "user", "content": user}],
).content[0].text
audit_citation("Body weight did not change.", "veltranib-rct",
"Body weight was unchanged in both arms.", docs, llm_call=llm_call)
# -> "supports" | "partial" | "unrelated" | "contradicts" | "not_found" | "misattributed"llm_call is deliberately the smallest possible contract β (system, user) β text
β so the judge wires to Claude, GPT, Gemini, Mistral, Cohere, or a local Qwen
without adapters. OpenAI, Cohere, and other one-liners are in
examples/quickstart.py.
The number you page on for a citation auditor is recall of unfaithful citations β the fraction of bad citations you actually catch. The gate lifts that recall for free on the three failure modes a judge is worst at (fabricated, franken, misattributed), because those are exactly the ones that look supportive. What reaches the model is only the genuinely ambiguous case: a real quote whose sufficiency is a judgment call.
For citations expressed as character offsets, use the same fail-closed gate via
span_gate(doc_id, start, end, docs). Offsets are Python string indices and use
the half-open interval [start, end). Pass chunk=(chunk_start, chunk_end) when
the retriever returned one chunk; spans outside it are rejected rather than
silently clipped. An optional quote= must match inside the resolved span:
from verbatim_citation_gate import span_gate
span_gate("report", 10, 42, docs, quote="the cited text", chunk=(0, 100))Contributions wiring the gates into specific frameworks (Haystack, LlamaIndex, LangChain, Cohere, Qwen-Agent, β¦) are welcome.
Now β v0.1.0.
The deterministic gate (gate.py) and the burden-of-proof judge (judge.py), 24 tests green,
install from git. Known limits are open issues, not footnotes β read them before you rely on it,
especially #1.
Next, in the order we would take them:
| What | Why it matters | Issue |
|---|---|---|
| Minimum-evidence rule | a one-word quote currently returns found |
#4 |
| Citations by doc id / offset | the gate presumes quote-style citations | #9 |
| PyPI package | so pip install verbatim-citation-gate is real |
#8 |
Shipped from this table β all three fixed by outside contributor Guan Tong,
who claimed them off the issue list and had the first PR open within the hour: Unicode-aware normalization
(#1 β before it, the gate's
core promise was silently off in every non-Latin script), corpus normalization cached
(#2), and the py.typed marker
(#3).
Every noticeable change ships as a new release, so the release feed is the honest record of how far along this is β not the commit graph.
Stars are free and prove nothing. From 2026-08 this project is measured on three numbers, reviewed monthly, target horizon 4β8 months:
- Weekly unique clones β people pulling the code, not applauding it.
- Independent consumers β repos or teams that would have to replace this gate if it disappeared. This is the only number that really matters.
- Repeat outside contributors β engineers who came back a second time. First one already exists; that fact is why this repo became the lab's flagship.
This project is built by a human + AI team, and the git log says so: Claude writes most of the code, Codex and Grok review it, Gemini feeds the research. Each is credited on a commit only if its output changed that commit's content β no decorative credits. Lab-wide policy, one source for every repo: AI-CONTRIBUTORS.md.
MIT Β© Palo Alto AI Research Lab
Built and battle-tested at Palo Alto AI Research Lab β a fleet of Claude Code machines running 24/7 as a second brain and synthetic cofounder. This gate was extracted after it survived production, not written as a demo.
Questions, war stories, or you want to run this on your own fleet:
- π€ Author: Anton Dziatkovskii β Telegram @tonydzi Β· WhatsApp +1 341 222 9178 Β· X @Tony_Stef_
- π£ Channels: @ClawRus (RU) Β· @ClawEng (EN)
- π palo-alto.ai Β· Palo Alto AI Research Lab
- π§ͺ Engineers: want to test-drive this setup? Message me β I hand out free starter seeds to engineers who test and report back.
The queue is visible: verbatim-citation-gate β roadmap β Now (an open PR exists), Next (scoped, free to take), Later (deferred, with the reason on the card), Shipped β and Shipped now starts with v0.1.0.
Issues labelled accepted
are scoped, free to take, and nobody is on them. Comment "claiming this" β no permission needed β
and it is yours for 7 days. New here? Start with
good first issue.
You keep the copyright to your code. No CLA, no assignment, ever β your contribution goes in under this repo's existing license, the same terms as ours. We answer every issue and PR within 48 hours, including "no, and here is why"; our silence is our bug, so ping the thread.
Full deal: CONTRIBUTING.md
This repository is one piece lifted out of a live operation: one engineer running a fleet of machines, built with Claude as implementation collaborator; the machines reach consensus with each other and wake the human only for money or the irreversible. It was extracted after it survived production, not written as a demo β and it runs on its own: nothing here phones home to the rest.
See how the whole thing fits together β SYSTEM.md
Its closest neighbours in the gates layer: verdict-contract Β· claim-check Β· verified-ops-starter
One click and an agent reads the repo, pulls out the patterns and helps you apply them to your own work.
Copy the prompt (works in any agent: Gemini, Grok, a local model, your own CLI)
Read this repo: https://github.com/tonydzi/verbatim-citation-gate (βverbatim-citation-gateβ - Catch fabricated RAG citations before they reach the user: a zero-token verbatim gate + a burden-of-proof judge, framework-agnostic). Work out what problem it actually solves, pull out the reusable patterns and help me apply them to my own setup. Start by asking what I am working on.
β TonyDzi, Palo Alto AI Research Lab Β· second brain, agent coordination, persistent memory: github.com/tonydzi
