verdict-contract

A small Python file that reads an AI code reviewer's answer and turns it into a safe pass/fail exit code.

Who it's for: anyone who lets an AI model review code and wants automation to merge only on a real APPROVE. What you get: one stdlib-only file, verdict_contract.py. It gives the model one strict way to answer, and it treats anything unclear as "do not approve".

Quick start

1. Install Python 3.8 or newer. Nothing else.
2. Clone it: git clone https://github.com/tonydzi/verdict-contract and open the folder.
3. Run the tests: python3 test_verdict_contract.py (on Windows use python).
4. Try it on a fake review: save a text file report.txt whose only line is ===VERDICT=== APPROVE, then run python3 verdict_check.py report.txt.

Done when: the test run's last line starts with OK - 42 contract cases and the exit code is 0, and step 4 prints VERDICT: APPROVE. Change the file to looks fine and it prints a (not stated) warning instead, with exit code 4: that is the safe default, not a failure. To gate a real reviewer, run git diff | python3 examples/review_gate.py 'codex exec --model gpt-5' and read the exit code: 0 approve, 3 changes requested, 4 the model broke the format.

Want an AI agent to do it for you? Paste this to Claude Code / Codex / Cursor: "Read the 'For AI agents' section of https://github.com/tonydzi/verdict-contract and set it up for me."

For AI agents

Preconditions

• Python 3.8+ (checked with 3.12), standard library only. Use python instead of python3 on Windows.
• No account, key or network for the module itself.
• Ask the human: which reviewer command they use (it must take a prompt on stdin and print a review), and whether they want the GitHub Action.

Install

git clone https://github.com/tonydzi/verdict-contract
cd verdict-contract

To use it in another project, copy verdict_contract.py there. Always import the prompt rule, never retype it:

import sys
from verdict_contract import PROMPT_RULE, find_verdict, exit_code
# MY_REVIEW_TASK (your review instructions) and my_model (your model call) are placeholders
prompt = MY_REVIEW_TASK + "\n\n" + PROMPT_RULE
report = my_model.run(prompt, diff)
sys.exit(exit_code(find_verdict(report)))

Configure

• Wrap a CLI reviewer: git diff | python3 examples/review_gate.py '<reviewer command>' (command supplied by the human).
• Check a saved report file: python3 verdict_check.py <report-file> (or - for stdin).
• GitHub Actions: this repo ships action.yml with one input, report (path to the reviewer report). Use uses: tonydzi/verdict-contract@main with with: report: <path>; ask the human which ref to pin.
• Exit codes: 0 approve, 3 request changes, 4 ambiguous or no verdict, 1 unreadable report, 2 usage error.

Verify

python3 test_verdict_contract.py
python3 _test_verdict_check.py
python3 _test_review_gate.py
python3 _test_prompt_guard.py

Expected: all four exit 0 (real run 2026-11-13: yes). The first ends with OK - 42 contract cases. Quick check by hand: a file containing the single line ===VERDICT=== APPROVE passed to python3 verdict_check.py <file> prints VERDICT: APPROVE and exits 0; a file without a verdict prints VERDICT: (not stated) !! ... and exits 4.

Uninstall / rollback

Delete the cloned folder, or remove the copied verdict_contract.py and the workflow step that calls the action.

Rules for the agent

• Do not loosen the parser so that APPROVE is easier to reach. Blocking must always win; see AGENTS.md.
• Never retype PROMPT_RULE into a caller; import it.
• Treat exit 4 as "do not merge", never as a pass.
• Any change needs a new counterexample case in test_verdict_contract.py.

Details

Your LLM reviewer said APPROVE. Did it?

If you pipe a diff to a model and grep its reply for VERDICT: APPROVE, you do not have a gate — you have a coin flip that fails in the dangerous direction, and test_verdict_contract.py shows five ways it does.

This repo is the stdlib-only module that replaced ours — 205 lines in verdict_contract.py — plus the 42 counterexamples that forced every line of it, all of them runnable as test_verdict_contract.py.

MIT. No dependencies. Python 3.8+. Copy one file and go.

---

The failure it fixes

We ran a real two-vendor review gate over our own diffs, then asked the two vendors to review the gate itself; both independently answered ROOT-FIXED: NO. Every counterexample below is reproduced live by test_verdict_contract.py, and the afternoon they came from is written up in devlog/2026-08-04-seven-rounds-to-a-verdict.md:

what the model wrote	what the old parser read	why it matters
I refuse to give a VERDICT APPROVE would be wrong	APPROVE	\s in the gap class quietly includes \n
Example output: VERDICT: APPROVE	APPROVE	a quoted example shipped as a real pass
the prompt text itself, echoed back	APPROVE	the instruction contained both literals: prompt self-poisoning
## VERDICT: REQUEST_CHANGES	(nothing)	a silent miss: review happened, no verdict recorded, nobody shouting
a blocking review, printed then exit 0	"the review passed"	the verdict never reached the exit code

The last one is the whole category: a gate whose verdict lives only in stdout is not a gate, it is a log line. The exit-code half of the fix lives in verdict_contract.py.

And one that only shows up in production: a reviewer quoted a bare ``` inside a ```text block, our fence tracker desynced, a quoted APPROVE became "visible", and the gate approved a diff the reviewer had just rejected. Fence detection is undecidable on nested or unbalanced fences, so the strict layer does not use it at all — the case is pinned as fence-desync in test_verdict_contract.py.

The fix, in four parts

All four are implemented in verdict_contract.py and each has its own cases in test_verdict_contract.py:

1. Structured marker (_SENTINEL_MARK in verdict_contract.py). The model must emit ===VERDICT=== <value> alone on its final line; the parser matches whole lines only, so prose can never glue into a verdict and a newline gap is impossible by construction.
2. Closed enum, explicit AMBIGUOUS, position enforced (find_verdict in verdict_contract.py). APPROVE or REQUEST_CHANGES, nothing else — a junk value, contradicting markers, a repeated marker, or an APPROVE that is not the final non-blank line all return AMBIGUOUS loudly. Never a guess, never a silent fail-safe.
3. The weak layer can block but never approve (_WEAK in verdict_contract.py). Old-style prose verdicts still block (safe direction); a prose approve degrades to AMBIGUOUS. The asymmetry is the design: a false block costs a re-read, a false approve ships a bug.
4. Exit-code gate (exit_code in verdict_contract.py). 0 approve, 3 request-changes, 4 ambiguous or missing — automation can no longer read "it printed something" as "it passed".

And the structural bit that makes it hold: the instruction the model receives and the parser that reads the reply live in the same file — PROMPT_RULE and find_verdict are both exported from verdict_contract.py. Two copies of a contract are one drift away from failing silently. The self-poisoning case asserts the prompt itself parses as nothing, so an echoed instruction is inert.

Quickstart

git clone https://github.com/tonydzi/verdict-contract
cd verdict-contract
python3 test_verdict_contract.py        # 42 cases, exit 0

Gate any CLI reviewer you already use — the wiring is examples/review_gate.py:

git diff | python3 examples/review_gate.py 'codex exec --model gpt-5'
echo "exit $?"      # 0 approve / 3 blocked / 4 the model broke the contract

In your own code, both sides come from one import of verdict_contract.py:

from verdict_contract import PROMPT_RULE, find_verdict, exit_code

prompt = MY_REVIEW_TASK + "\n\n" + PROMPT_RULE     # never retype the rule
report = my_model.run(prompt, diff)
sys.exit(exit_code(find_verdict(report)))

What you are looking at

verdict_contract.py is the sanitized twin of a module that has gated every code review across a 6-machine agent fleet since 2026-07-21, unchanged.

The test cases are not hypotheticals: each one is a counterexample a vendor model produced while trying to break the fix, over seven adversarial rounds — the round-by-round record is devlog/2026-08-04-seven-rounds-to-a-verdict.md. Four of those rounds found a real hole in code we already believed was correct.

Not included, on purpose: our reviewer prompts, machine topology, and approval routing. The contract is the reusable part; what changed when is in CHANGELOG.md.

Known limits

• Non-English prompts (reported, not yet explained). In one external run (issue #8) two German review prompts were sent with PROMPT_RULE as the last block. Two of three model lines ended with a localized or unmarked verdict line (URTEIL: ..., bare REQUEST_CHANGES) instead of the sentinel, and verdict_check.py exited 4. Exit 4 blocks, so nothing was approved. Sample: 2 prompts, 1 call per model line, 6 replies. The same reporter then ran an English control (same material, instructions translated): none of the 6 replies exited 4 (3 ended on the sentinel, 3 on VERDICT: REQUEST_CHANGES, read as exit 3 by the weak layer), against 4 of 6 in German. That is two prompts and one call per cell, so it does not establish that the language caused the difference.
• The parser does not accept localized labels on purpose: a wider gate trusts more. A sender-side guard that requires PROMPT_RULE verbatim and rejects look-alike patterns in the prompt is the pattern the reporter uses; examples/prompt_guard.py (contributed by @b7n0de, #9) is that guard.

Roadmap

Now — v0.1.0. PROMPT_RULE + strict/weak parser + exit_code() in one stdlib file, 42 contract cases, CI on three OSes × three Pythons plus a weekly run so rot surfaces without a push.

Next, in the order we would take them:

• More counterexamples. This contract only knows the failures 42 cases have shown it. The most valuable thing anyone can send is a reviewer reply that fools it — open an issue with the exact text.
• Wrappers for more reviewers. examples/review_gate.py wraps any CLI reviewer today; the common asks are a GitHub Action and a pre-commit hook.
• Retiring the weak legacy layer. It can block but never approve — deliberate, and it should stop being needed once callers emit the sentinel.

Every noticeable change ships as a new release, so the release feed — not the commit graph — is where you can see whether the contract has moved under you.

Who made this

Palo Alto AI Research Lab — one engineer running operations and an AI cofounder running a fleet of autonomous Claude machines, shipping the useful pieces for free. How this repo expects a change to be proven, human or agent, is in AGENTS.md.

If this saves you one bad merge, a star helps more than you would think — we are looking for the first ten people who actually run it. Found a case that breaks it? Open an issue at https://github.com/tonydzi/verdict-contract/issues with the exact reply text; a counterexample is the most valuable thing you can send us.

Related: verified-ops-starter (your job exited 0, prove it did the work) and awesome-verified-agents.

Devlog

• Seven rounds to a verdict - how two vendor models spent an afternoon proving our review gate was lying, with every counterexample they produced.

---

Contributors welcome — and here is what we are missing

We spend a lot of time answering other people's issues. It was fair to say out loud what we have not built ourselves:

• Ship a GitHub Action that wraps the verdict contract
• Ship a pre-commit hook
• Send us a reviewer reply that fools the parser

Issues labelled accepted are scoped, free to take, and nobody is on them. Comment "claiming this" — no permission needed — and it is yours for 7 days. New here? Start with good first issue.

You keep the copyright to your code. No CLA, no assignment, ever — your contribution goes in under this repo's existing license, the same terms as ours. We answer every issue and PR within 48 hours, including "no, and here is why"; our silence is our bug, so ping the thread.

Full deal: CONTRIBUTING.md

---

🧩 One piece of a working system

This repository is one piece lifted out of a live operation: one engineer running a fleet of machines, built with Claude as implementation collaborator; the machines reach consensus with each other and wake the human only for money or the irreversible. It was extracted after it survived production, not written as a demo — and it runs on its own: nothing here phones home to the rest.

See how the whole thing fits together → SYSTEM.md

Its closest neighbours in the gates layer: claim-check · verified-ops-starter · oss-publish

AI contributors

This project is built by a human + AI team, and the git log says so: Claude writes most of the code, Codex and Grok review it, Gemini feeds the research. Each is credited on a commit only if its output changed that commit's content — no decorative credits. Lab-wide policy, one source for every repo: AI-CONTRIBUTORS.md.

READ THIS WITH AI

One click and an agent reads the repo, pulls out the patterns and helps you apply them to your own work.

Copy the prompt (works in any agent: Gemini, Grok, a local model, your own CLI)

Read this repo: https://github.com/tonydzi/verdict-contract (“verdict-contract” - Your LLM reviewer said APPROVE. Did it? A structured verdict contract: prompt rule + parser + exit-code gate in one stdlib file, with the 42 counterexamples that forced every line. Free, MIT). Work out what problem it actually solves, pull out the reusable patterns and help me apply them to my own setup. Start by asking what I am working on.

_{— TonyDzi, Palo Alto AI Research Lab · second brain, agent coordination, persistent memory: github.com/tonydzi}