pyvulnpatch-probe

Checks whether a test that scores AI tools on fixing Python security bugs is fair, by measuring how much the test itself gives away.

Who it's for: researchers and engineers who read or build benchmarks for automated vulnerability repair in Python. What you get: three scripts (leak census, date census, reproducibility audit) plus a self-report scorer, and a README with the measured results on the public Vul4Py corpus. This repo reports no solve rate.

Quick start

1. git clone https://github.com/tonydzi/pyvulnpatch-probe and cd pyvulnpatch-probe.
2. Run the tests: python3 tests/test_probe.py (no network, no API keys).
3. Read "What this measures" in Details to see the numbers, for example that 13 of 100 cases leak the advisory id into a supposedly blind prompt.

Done when: the tests print 27 passed, 0 failed and python3 tests/run_killlist.py ends with 22/22 mutations provably caught.

Want an AI agent to do it for you? Paste this to Claude Code / Codex / Cursor: "Read the 'For AI agents' section of https://github.com/tonydzi/pyvulnpatch-probe and set it up for me."

For AI agents

Preconditions

• Python 3 (the author used 3.11 and 3.12; no minimum version is declared in the repo). No pip packages are listed in the repo: standard library only.
• The tests need no network. The two censuses read the fix commits of the repos in data/corpus-vul4py.csv through the GitHub CLI (probe/census.py runs gh api repos/<slug>/commits/<sha>), so they need gh installed and logged in (gh auth status), plus internet access, and they spend GitHub API rate limit (about 100 calls each). Ask the human before running them.
• On Windows python3 is often a Microsoft Store stub that does nothing: type python instead of python3 in every command below.
• The reproducibility audit needs a local clone of the Vul4Py harness (https://github.com/tabudz/vul4py), which ships no licence file. Ask the human whether to clone it and whether they accept its terms.
• probe/run_subscription.py drives a coding agent through a subscription and spends the human's quota. Do not run it without explicit approval; this repo does not document an outsider setup for it, so ask the human.

Reproduce

git clone https://github.com/tonydzi/pyvulnpatch-probe
cd pyvulnpatch-probe
python3 tests/test_probe.py
python3 tests/test_repro_audit.py
python3 tests/test_install_cmds.py
python3 tests/run_killlist.py

With the human's approval, the censuses (commands from the README):

python3 -m probe.census leaks --corpus data/corpus-vul4py.csv --out data/leaks.jsonl
python3 -m probe.census dates --corpus data/corpus-vul4py.csv --out data/dates.jsonl --cutoff "GPT-4o 2023-10"=2023-10-01
python3 -m probe.repro_audit --workspaces <path-to-vul4py>/workspaces
python3 -m probe.score runs/<agent>/

python3 -m pytest tests/ -q also works if pytest is installed.

Configure

No environment variables are read by the probe scripts. Ask the human for the corpus CSV (default data/corpus-vul4py.csv), the model cutoff dates for the date census, and the path to their Vul4Py clone.

Verify

Run the four test commands above. Expected, as run locally with Python 3.12 (exit code 0 each): 27 passed, 0 failed, 7 passed, 0 failed, 11 passed, 0 failed, and 22/22 mutations provably caught. Warnings about run_killlist.py: it deliberately edits the five scorer files under probe/ (score.py, census.py, repro_audit.py, install_cmds.py, ablate.py) in place, one mutation at a time, and restores each afterwards; it runs all three suites 23 times, which takes many minutes on a slow Windows machine (about 5 minutes here). Run it only in a clean checkout, let it finish, and afterwards check git status shows no modified files. If it was interrupted, restore with git checkout -- probe/ (otherwise the suites fail from then on).

Census run: the census prints a JSON summary and exits 0 even when GitHub lookups failed, so read the summary, not the exit code. Expected for the leak census on the shipped corpus: "cases": 100, "errors": 0 and "leak_either": 13; for the date census "errors": 0 and after_cutoff of 71 (GPT-4o), 30 (Claude Sonnet 4) and 0 (2026 models). A non-zero errors means gh is missing or not logged in, or rate limited; the other numbers are then not comparable. Published numbers to compare: leaks 13 of 100 cases (either channel), post-cutoff slice 71 for GPT-4o, 30 for Claude Sonnet 4, 0 for 2026-cutoff models.

Uninstall / rollback

Delete the cloned folder, plus any data/*.jsonl output and the Vul4Py clone you made. Nothing is installed system-wide.

Rules for the agent

• Do not cite a solve rate or false-success rate from this repo. It reports none.
• Do not copy Vul4Py code or data into this repo, and do not publish exploit text. Only public, already fixed advisories are in scope.
• Do not run probe/run_subscription.py or any model-calling step without explicit approval.

Details

Three measurements that a Python vulnerability-patching benchmark needs before its solve rate means anything, and that the existing ones do not report.

This is not another benchmark. It does not ship a corpus. It is an audit and ablation layer that runs on top of an existing Python AVR corpus and answers questions about the measurement, not about the models.

Why this exists

As of October 2026 the landscape is:

benchmark	language	patching?	paired oracle?
PatchBench (Sep 2026)	C/C++ only	yes	yes
Vul4Py (Aug 2026)	Python	yes	yes
CyberChainBench (Jun 2026)	Solidity	yes	economic
Vulnerability Localization Benchmark (Sep 2026)	multi	no (localisation)	n/a

Python is covered, by Vul4Py — 100 real vulnerabilities, 60 projects, 60 CWEs, each with a paired oracle (exploit test must flip from fail to pass; the project's own pytest suite must stay green). Vul4Py reports OpenHands repairing 41/100, the best directly-prompted LLM 4/100, and a specialised AVR tool 2/100.

What is missing is not another corpus. It is the arithmetic that tells you whether those numbers measure repair ability:

1. Every agent is handed the answer's address. Vul4Py gives every condition the advisory identifier, the fix commit SHA, and a pre-selected set of source files drawn from the human fix. Choosing the files is line-level localisation. There is no condition without it, so the hint's contribution to 41/100 is unmeasured.
2. Contamination is named, not measured. The paper lists "leakage of fix commits into LLM training data" under threats to validity. No date split, no post-cutoff slice.
3. The agent is never asked what it thinks it did. The oracle is compared only to itself (exploit-only vs paired). Nobody measures the agent asserting a hole is closed while the oracle disagrees — operationally the worst outcome, because a confident wrong answer closes the incident.

What this measures

1. Leak census — is a "no hints" condition actually hintless?

python3 -m probe.census leaks --corpus data/corpus-vul4py.csv --out data/leaks.jsonl

Withholding the CVE id from the prompt means nothing if the repository says it out loud. Measured over Vul4Py's 100 cases (100/100 resolved, 0 errors):

channel	cases
advisory id in a test file's path	4 / 100
advisory id in the test's added lines	13 / 100
either — a blind prompt is not blind here	13 / 100
in the fix commit message (control; never shown to the agent)	32 / 100
fix commit adds no test file at all	4 / 100

The path leaks are fixture payloads named after the advisory (tests/data2/GHSA-3gf5-cxq9-w223.pkl) — one such directory names ten advisories at once, so a single ls hands over nine neighbours too.

This number is worth stating because the common assumption is worse than reality. A reviewer on this design argued the leak is near-universal and makes hint ablation "an illusion"; measured, it is 13%. Material, and cheap to sanitise — not fatal. probe/ablate.py masks those ids and blind_is_honest() refuses to label a case blind until it does.

2. Date census — how large can a post-cutoff claim be?

python3 -m probe.census dates --corpus data/corpus-vul4py.csv --out data/dates.jsonl \
  --cutoff "GPT-4o 2023-10"=2023-10-01 \
  --cutoff "Claude Sonnet 4 ~2025-03"=2025-03-01 \
  --cutoff "2026 frontier models ~2026-05"=2026-05-01

Fix commits by year: 2017:2 · 2018:1 · 2020:3 · 2021:3 · 2022:5 · 2023:18 · 2024:34 · 2025:34.

Post-cutoff slice: 71 cases for GPT-4o, 30 for Claude Sonnet 4, 0 for any 2026-cutoff model.

This result kills the clean version of measurement 2, and that is the finding. Vul4Py's corpus stops in 2025, so on a current model there is no uncontaminated slice at all — not a small one, zero. Any post-cutoff claim about a 2026 model on this corpus is arithmetically unavailable. For the models Vul4Py itself used the slice is real (30 and 71), so the split is worth running against their numbers; it is not a path to a fresh result on current models. Doing that needs either new 2026 advisories or semantics-preserving perturbation of the existing ones, and both are larger projects than this one.

3. Reproducibility audit — how many cases can grade a patch at all?

python3 -m probe.repro_audit --workspaces <vul4py>/workspaces

Before any solve rate, one question: on how many cases is the oracle actually trustworthy? The upstream harness answers that itself — OK means the exploit test fails before the fix, passes after, and the project suite stays green — but it does not say why the others fail, which is the difference between "the benchmark is broken", "my machine is broken", and "the patch is bad".

Measured on the 10-case pilot subset (macOS, Python 3.11, upstream harness at 2a20f54, October 2026):

	cases
usable oracle (OK)	1 / 10
INFRA_BROKEN	7 / 10
UNEXPECTED	2 / 10

Cause, from the harness's own logs: 7 of 10 are missing_test_extras, 2 are unexplained and reported as unknown rather than folded into a bucket.

Two independent defects produce this, both demonstrated rather than inferred:

1. install_cmds is identical across all 10 cases — pip install -U pip wheel then pip install -e . — and is not derived from the project. That installs runtime dependencies only, while test dependencies live in extras the harness never installs. starlette's suite then cannot import its own conftest.py without httpx; authlib's needs flask; h2's needs hypothesis; python-socketio's needs requests.
2. Dependencies are unpinned. Patching case 1 for starlette (to pip install -e ".[full]") got httpx installed and moved the failure to DeprecationWarning: The anyio.abc.BlockingPortal alias is deprecated, raised as an error because the project builds with filterwarnings = ["error"]. A 2025 fix commit is being tested against October 2026 dependency resolutions. The code is fine; the world moved.

Note the ordering trap this audit is built around: real logs are full of deprecation chatter, so a classifier that tests the loose pattern first blames nearly everything on unpinned dependencies. missing_test_extras is tested first, and a mutation proving that is in the kill-list.

Fixing cause 1 is necessary and not sufficient

probe/install_cmds.py derives real per-project install commands from each project's own pyproject.toml and requirements files — never inventing a package name, always printing what it chose. It changed the commands for 6 of 10 cases (.[full], .[dev], --group dev, --group testing, -r requirements.txt). After applying them and re-scanning the whole set:

cause	before	after
missing_test_extras	7	5
unpinned_dependency	0	1
unknown	2	3
usable oracle (OK)	1	1

The usable count did not move. What the derivation did, case by case:

• starlette got httpx installed, and in the live run the failure moved to the anyio deprecation. The classifier still files it under missing_test_extras on a conftest ImportError — see the correction below.
• authlib's dev group turns out to be a lint/tooling group (tox, ruff, coverage) and has no flask; its real test environment is assembled by tox. Name-based selection picked the only group there was, and picked wrong — visible in the output, which is why the derivation prints its choice.
• pypdf took .[dev] and now fails on a missing PIL.
• fastmcp installed a full test stack and the live log ends in an asyncio cancellation during lifespan teardown.
• four cases were never treated at all (found_nothing: skops, internetarchive, mkdocs-include-markdown-plugin, python-socketio) — they declare no test dependencies anywhere the deriver reads, so it correctly refused to guess. python-socketio still fails on the missing requests the table above names.

Correction (2026-10-28)

An earlier version of this section drew a headline from the table above — "a benchmark whose entries install unpinned dependencies decays" — and put the 9/10 figure next to it. An adversarial review of this repo's own shipped data showed that overstates what was measured, and the overstatement went upstream before it was caught. Three things were wrong:

1. The decay thesis rests on n=1, not on 9/10. In data/repro_audit_after.jsonl exactly one case is classified unpinned_dependency (fastmcp). Five are still missing_test_extras, three are unknown, one is OK. missing_test_extras is a time-independent harness-config defect: identical install_cmds on every entry would have failed the same way on publication day in August. Writing "published in August, by October nine of ten could no longer grade a patch" fused a static config bug with a time-dependent decay claim and presented the sum as evidence for decay. The 9/10 is real — it is the count of entries that cannot grade a patch — but it is not the decay claim's denominator.

2. That one case's evidence is the weakest in the set. fastmcp is filed as unpinned_dependency on an AuthlibDeprecationWarning line, while its fresh functional_fixed.log ends in CancelledError. Logs accumulate across runs, so the classifier may have matched a stale line — exactly the caveat this section already carried, landing on the one row the headline leaned on.

3. "Necessary but not sufficient" is partly a statement about this tool. The deriver changed commands on 6 of 10 and left 4 untouched; of the 6, at least two got a demonstrably wrong extra (authlib's lint group, pypdf's dev without PIL). So it was correctly applied on roughly 2–3 cases. "Fixing install_cmds does not recover the benchmark" is not established by that — "our deriver needs work" fits the same data, and the two point at opposite next actions.

What survives. The measured facts stand: 1 of 10 entries can grade a patch; install_cmds is one hardcoded pair on all 10 and never installs the projects' test dependencies; the Dockerfile pins nothing. The recommendation also survives on its own merits — a pip freeze captured at curation time subsumes both causes, because a locked environment carries the test dependencies too. What does not survive is ranking pinning above the config defect on the strength of this data: by measured case count the config defect dominates, 5 to 1.

Corrected upstream in the same thread rather than only here.

Reported upstream: tabudz/vul4py#1.

Caveat, stated rather than buried: logs accumulate across runs, so a case re-scanned after a fix can still match a stale line from an earlier failure. The cause is per-case evidence, not a timestamped verdict — and as the correction above shows, that caveat bit the one row the headline rested on.

4. Self-report gap — does the agent know whether it succeeded?

python3 -m probe.score runs/<agent>/

Each case records two independent verdicts and crosses them:

	oracle: repaired	oracle: not repaired
agent claims FIXED	true_success	false_success
agent claims NOT_FIXED	unclaimed_success	honest_failure
agent states nothing	no_verdict	no_verdict

false_success_rate is false_success / claimed_fixed — the share of the agent's own assertions that the oracle refused. It is reported as null, never 0.0, when the agent made no assertions, so an empty run cannot read as a flawless one.

Honest status

Measured and reproducible now (no model calls, no API keys, no money): the leak census and the date census, both over all 100 cases, both green end to end. Every number in this README came out of the code in this repo.

Pipeline proven, measurement blocked. probe/run_subscription.py drives the three conditions through a flat-rate subscription agent and does produce patches — two of three conditions generated a real diff on the first case tried. The measurement is blocked one step later, on the oracle: only 1 of the 10 pilot cases can grade a patch at all (section 3). A hint-ablation delta over one case is not a result, so this repo reports no solve rate and no false-success rate, and you should not cite one from here. The scorers are verified against synthetic patches only.

That blocker is the current finding rather than a setback: the reason 9 cases cannot grade anything is reproducible, has two named causes, and is fixable upstream.

What such a run costs, grounded in Vul4Py's own Table 2 (mean API spend per instance on a Claude Sonnet 4 backbone: OpenHands $1.32, SWE-agent $1.43, direct prompting $0.03–0.05):

scope	runs	expected	hard ceiling with a per-case cap
smoke test, 3 cases × 3 conditions	9	≈$12	$27 at $3/case
minimum real measurement, 10 × 3	30	≈$40	$45 at $1.50/case
publishable, 30 × 3	90	≈$119	$135 at $1.50/case

The direct-prompting arm is ~30× cheaper but scores 4–5/100 upstream, so an ablation delta measured there sits on the floor and is noise. The number only means something on an agentic backbone.

Not done: perturbation-based contamination measurement (the replacement for measurement 2 suggested above), open-weight backbones, languages other than Python.

Verification

tests/ is a kill-list: 45 assertions, each naming a specific way a measurement could lie.

tests/run_killlist.py is the red-first proof. It breaks each scorer on purpose — oracle ignoring the functional suite, false-success rate divided by the wrong denominator, blind prompt forgetting to redact quoted source, leak audit counting removed lines — reruns the suite, and requires it to go red via the named test. 22 of 22 mutations are provably caught. A test that has only ever been green proves nothing about the scorer.

python3 tests/test_probe.py       # 27 passed
python3 tests/test_repro_audit.py # 7 passed
python3 tests/test_install_cmds.py # 11 passed
python3 tests/run_killlist.py     # 22/22 mutations provably caught

One bug was found this way in production rather than by the kill-list: the GitHub slug parser forbade dots, so changedetection.io became changedetection and two cases reported 404. That read as "GitHub lost the commit" until the slug was printed. Both the fix and a mutation guarding it are now in place — see KILL-LIST.md.

Licence

This repo is MIT (LICENSE). Citing it: CITATION.cff.

Attribution and licensing of the corpus

The corpus this operates on is Vul4Py, by Tan Bui, Ting Zhang, Ferdian Thung, Yunpeng Xiong, Penghao Jiang, Xin Zhou and David Lo (Singapore Management University / Monash / UNSW) — arXiv:2608.00692, harness at github.com/tabudz/vul4py. The paired-oracle design is theirs; this repo adds measurements around it and claims nothing about the corpus itself.

⚠️ That repository ships no LICENSE file, which under default copyright means all rights reserved. This repo therefore contains none of their code and none of their data — only our own scripts plus data/corpus-vul4py.csv, a list of public (advisory id, repo URL, fix commit) triples. Anyone wanting to run this should clone their harness themselves and comply with whatever terms they set. Asking them to add an explicit licence is the first thing worth doing here.

Benchmarks referenced for scope comparison: PatchBench (arXiv:2609.04075), CyberChainBench (arXiv:2606.26216), Vulnerability Localization Benchmark (arXiv:2609.15939). Advisory data originates from OSV and the GitHub Advisory Database.

Scope and safety

Every vulnerability referenced here is already public and already fixed — each one has a published advisory and an upstream patch, both predating this work. Nothing here produces a new exploit, and no exploit text is included: the reproducers live in the upstream projects' own test suites. This is defensive measurement of whether automated repair works.

---

🧩 One piece of a working system

This repository is one piece lifted out of a live operation: one engineer running a fleet of machines, built with Claude as implementation collaborator; the machines reach consensus with each other and wake the human only for money or the irreversible. It was extracted after it survived production, not written as a demo — and it runs on its own: nothing here phones home to the rest.

See how the whole thing fits together → SYSTEM.md

AI contributors

This project is built by a human + AI team, and the git log says so: Claude writes most of the code, Codex and Grok review it, Gemini feeds the research. Each is credited on a commit only if its output changed that commit's content — no decorative credits. Lab-wide policy, one source for every repo: AI-CONTRIBUTORS.md.