red-first-review-skill

A Claude Code skill and a small Python tool that check someone else's pull request by deleting each safety check the PR adds, then seeing whether the tests notice.

Who it's for: people who review pull requests and want proof that a fix is really tested. What you get: a table of which removed guards nothing caught (the real findings), which a review can be built on.

Quick start

1. Put the skill where Claude Code can find it: git clone https://github.com/tonydzi/red-first-review-skill ~/.claude/skills/red-first-review
2. Open the project you want to review and ask Claude: "review this PR" or "is this change actually tested?"
3. Claude runs mutmatrix.py on the PR's guards and reports the ones nothing caught.

Done when: you see a table with each guard marked killed or SURVIVED, ending in a line like 2 killed, 0 SURVIVED, 0 invalid. A SURVIVED row is not an error: it is the finding (a guard the tests never notice). A table cut short by a message about baseline or canary means nothing was measured, so do not trust it.

Want an AI agent to do it for you? Paste this to Claude Code / Codex / Cursor: "Read the 'For AI agents' section of https://github.com/tonydzi/red-first-review-skill and set it up for me."

For AI agents

Preconditions

• Python 3.8 or newer. Standard library only. Nothing to pip install.
• For the skill route: Claude Code with a skills folder at ~/.claude/skills/. For other agents, ask the human where skills live, or just use mutmatrix.py on its own.
• A target repository whose test suite is green and runs from one shell command. Ask the human for that test command and for the source file(s) the PR changes.
• Ask the human for a clean git status in the target repo before any run.

Install

git clone https://github.com/tonydzi/red-first-review-skill ~/.claude/skills/red-first-review

Windows PowerShell does not expand ~ for native commands, so use: git clone https://github.com/tonydzi/red-first-review-skill "$HOME\.claude\skills\red-first-review". Plain tool only: copy the single file mutmatrix.py anywhere. On Windows python3 is usually a Microsoft Store stub: type python instead of python3 in every command below.

Configure

No config files. Each run needs these from the human or the PR:

python3 mutmatrix.py --repo . --test "<their test command>" --file <path/to/changed_file.py> --call-identity <guard_function_name>

Other targeting flags (all in python3 mutmatrix.py --help): --mutant 'LABEL::FILE::FIND::REPLACE', --line 'LABEL::FILE::N', --sep '@@' when the code contains ::, --canary-file, --timeout. Pass -p no:randomly inside a pytest command so test order is stable. Follow SKILL.md for the full review procedure.

Verify

python3 mutmatrix.py --help
python3 _test_mutmatrix.py

Expected: --help exits 0 and prints the usage with --test. The self-test (about 50 seconds on Windows; it plants four deliberate bugs in a copy of the tool and checks each is caught) exits 0 and ends with all cases behaved as expected. (cases C1 to C7 PASS). It tests the tool itself, not your target repo; run it from the folder that holds mutmatrix.py.

Uninstall / rollback

Delete ~/.claude/skills/red-first-review. mutmatrix.py restores the files it mutates; if a run was killed, check git status in the target repo and look for a leftover .mutmatrix-backup/ folder (the tool refuses to start while one exists). Restore from it, then remove it.

Rules for the agent

• Never run mutmatrix in a repo with uncommitted work.
• Exit codes: 0 no survivors, 1 survivors found (the finding), 3 nothing measured (do not quote the result), 2 usage, 4 tool crashed.
• Post reviews with gh pr review --comment, not gh pr comment. Do not post on someone else's PR without the human's yes.
• Read the whole changed files, not only the diff (step 0 of SKILL.md).

Details

A test that has never been shown failing is a claim, not a check.

This is a Claude Code skill (SKILL.md) plus a small standalone tool (mutmatrix.py) for reviewing someone else's pull request by measuring the one thing you cannot get by reading it: which of the guards this PR adds would survive being deleted?

mutmatrix.py neutralises each guard in turn, runs the project's own test suite, and records what nothing notices. Survivors are the finding. Everything else in the review — the suggested test, the argument, the ranking — hangs off that table, and SKILL.md is the procedure that turns it into a review.

Standard library only. Python 3.8+. Nothing to install.

python3 mutmatrix.py \
  --repo . \
  --test "pytest tests/unit -q -p no:randomly" \
  --file src/pkg/api.py \
  --call-identity redact_secrets

What that produces, from the hand-rolled run this repository was built out of (google/adk-python#6957, a PR redacting OAuth secrets at eight call sites):

mutant	verdict	killed by
/run	SURVIVED	nothing
/run_sse	killed	test_agent_run_sse_redacts_oauth2_client_secret
/run_live (websocket)	SURVIVED	nothing
_redacted_session_response	killed	test_get_session_redacts_oauth2_client_secret
_redacted_sessions_response (list)	SURVIVED	nothing
dev get_eval_result_legacy	SURVIVED	nothing
dev get_eval	killed	test_get_eval_redacts_oauth2_client_secret
dev get_eval_result	SURVIVED	nothing

The title of adk-python#6957 named three endpoints. One of them was actually guarded by a test. Reverting the /run line alone left the entire suite of adk-python#6957 green.

---

Why it refuses to guess

Most of mutmatrix.py is not the mutation loop — that part is twenty lines. Most of it is the checks in mutmatrix.py that stop it printing a table it cannot stand behind.

gate	what it catches	exit
baseline	suite already red → nothing is attributable to the PR	3
canary	test command never imports the file → every "killed" is another test's	3
compile check	mutant does not parse → every test fails, which looks exactly like perfect coverage	3
no-canary	you skipped the canary → the result is UNPROVEN, never clean	3

Exit 0 means no survivors. Exit 1 from mutmatrix.py means survivors — that is your review. Exit 3 means mutmatrix.py measured nothing and you should not quote it.

The compile check exists because of a measurement made building this: a delimiter collision produced two unparseable mutants on anthropic-sdk-python#1906, and both were scored killed. The review of adk-python#6957 would otherwise have certified coverage nobody had. Now:

| `sync delete-root guard`  | INVALID | mutant does not compile -- not a coverage result |
0 killed, 0 SURVIVED, 1 invalid.

---

A real run, end to end

anthropics/anthropic-sdk-python#1906 — "fix(memory): compare resolved paths in the delete root guard". In anthropic-sdk-python#1906 the memory tool guarded its root directory with if command.path == "/memories", comparing the raw string the model sent, so /memories/, /memories/. and /memories/subdir/.. all reached shutil.rmtree.

Step 1, baseline.

$ .venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q
79 passed in 5.66s

Step 3, neutralise both guards (sync and async) with mutmatrix.py, one at a time:

python3 mutmatrix.py --repo . \
  --test ".venv/bin/python -m pytest tests/lib/tools/memory_tools/ -q -p no:randomly" \
  --canary-file src/anthropic/lib/tools/_beta_builtin_memory_tool.py --sep '@@' \
  --mutant 'sync delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if full_path == self.memory_root.resolve():@@if False:' \
  --mutant 'async delete-root guard@@src/anthropic/lib/tools/_beta_builtin_memory_tool.py@@if Path(str(full_path)) == Path(str(self.memory_root)).resolve():@@if False:'

BASELINE: green (7s) -- 79 passed in 5.29s
CANARY  : red (1s) -- the test command does reach _beta_builtin_memory_tool.py

  [1/2] sync delete-root guard    killed   (7s)
  [2/2] async delete-root guard   killed   (7s)

2 killed, 0 SURVIVED, 0 invalid.

Both guards are real. Six named tests in anthropic-sdk-python#1906 die when the sync guard goes:

test_delete_not_allow_deleting_memories_directory
test_delete_not_allow_deleting_memories_directory_via_symlink
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories//]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/.]
test_delete_not_allow_deleting_memories_directory_via_alias[/memories/subdir/..]

A clean table is a result, not a wasted run. "I removed your guard and your tests caught it, here are the six by name" is worth more to a maintainer than another opinion about style, which is why SKILL.md keeps that verdict.

Step 0 of SKILL.md is where the finding came from. Reading the whole file rather than the diff turned up a sibling the PR's own reasoning applies to and the diff never touches — rename, whose old_path can also be the root, and which has no guard at all:

rename("/memories", "/memories/backup")
  -> OSError(22, 'Invalid argument')   # not a ToolError; logged at ERROR with a traceback

Every other refusal in that class is a clean ToolError the model can act on. That became the one actionable finding in anthropic-sdk-python#1906, and it exists because of step 0, not because of the mutation table.

Where the method comes from

The table at the top of this README is that run against adk-python#6957. Five of eight redaction sites in adk-python#6957 survived, including two of the three endpoints its own title claimed to fix. The author's reply:

Thanks for this - genuinely one of the most useful reviews I've gotten on this PR. All three findings confirmed and fixed, verified the same way you found them (neutering each call site and checking the suite).

— prasanna8585, 1 Sep 2026 (comment id 5489421299)

That review was hand-rolled. This repository is that procedure made repeatable, as SKILL.md and mutmatrix.py.

---

Install

As a Claude Code skill:

git clone https://github.com/tonydzi/red-first-review-skill \
  ~/.claude/skills/red-first-review

Then ask Claude to review a PR; SKILL.md triggers on "review this PR", "is this change actually tested?", "mutation test this".

As a plain tool, mutmatrix.py is one file with no dependencies. Copy it in.

Targeting

flag	use
--call-identity NAME --file PATH	one mutant per call site of NAME; rewrites NAME(x) → x
--mutant 'LABEL::FILE::FIND::REPLACE'	an exact textual edit; FIND must be unique in the file
--line 'LABEL::FILE::N'	comment out line N
--sep '@@'	change the separator when your code contains :: (Python usually does)
--canary-file PATH	the file the test command must import (default: first mutated file)
--no-canary	skip the reach check; the run is then reported UNPROVEN, never clean

Gotchas paid for in advance

• Bytecode caching will lie to you, so mutmatrix.py defends against it. CPython validates a cached .pyc by source size and mtime in whole seconds. Neutralising the same guard at two call sites usually yields two files of identical length — so two mutants inside one second run the first one's bytecode and both score SURVIVED, a trap pinned by _test_mutmatrix.py. Measured on the self-test before the fix: 1 killed became 0 killed. mutmatrix gives every write its own mtime second.
• Random test ordering makes runs incomparable. Pass -p no:randomly.
• gh pr comment is not a review. It creates no review object, so the work lands in no contribution graph and no reviewed-by: filter — while still looking like noise to spam heuristics. Use gh pr review --comment.
• Do not mutate a repository you have uncommitted work in. The tool restores from an in-memory copy and a backup directory, and refuses to start if a previous run left one behind, but git status before you start is free.

Does the tool's own test suite pass this bar?

It has to, or the whole thing is a joke. _test_mutmatrix.py builds a throwaway project whose right answer is known by construction (three call sites, one covered), and it ships with a kill-list — the mutations of itself it claims to catch:

$ python3 _test_mutmatrix.py
  C1 PASS  C2 PASS  C3 PASS  C4 PASS  C5 PASS  C7 PASS  C6 PASS
all cases behaved as expected.

$ python3 _test_mutmatrix.py --red M3     # killed/survived inverted
  C3 FAIL  C4 FAIL  C5 FAIL                # exactly the cases it declared

mutation	what it breaks	must fail
M1	baseline check disabled	C1
M2	canary verdict hardcoded green	C2
M3	killed/survived inverted	C3, C4, C5
M4	compile check on the mutant removed	C7

C6 is the control: green from the start and under every mutation above. Without it, a mutation that merely broke the tool would look like a working red mode.

Licence

MIT. Built for one person's fleet — adapt the paths and the test commands to yours.

---

🧩 One piece of a working system

This repository is one piece lifted out of a live operation: one engineer running a fleet of machines, built with Claude as implementation collaborator; the machines reach consensus with each other and wake the human only for money or the irreversible. It was extracted after it survived production, not written as a demo — and it runs on its own: nothing here phones home to the rest.

See how the whole thing fits together → SYSTEM.md

AI contributors

This project is built by a human + AI team, and the git log says so: Claude writes most of the code, Codex and Grok review it, Gemini feeds the research. Each is credited on a commit only if its output changed that commit's content — no decorative credits. Lab-wide policy, one source for every repo: AI-CONTRIBUTORS.md.

READ THIS WITH AI

One click and an agent reads the repo, pulls out the patterns and helps you apply them to your own work.

Copy the prompt (works in any agent: Gemini, Grok, a local model, your own CLI)

Read this repo: https://github.com/tonydzi/red-first-review-skill (“red-first-review-skill” - Review a PR by measuring which of its own guards its own test suite actually catches. Neutralise each guard, run their tests, report the removals nothing notices. Stdlib-only, no install). Work out what problem it actually solves, pull out the reusable patterns and help me apply them to my own setup. Start by asking what I am working on.

_{— TonyDzi, Palo Alto AI Research Lab · second brain, agent coordination, persistent memory: github.com/tonydzi}