redgreen-check

Prove that a change's tests catch the bug.

redgreen-check runs the tests a change adds or edits twice:

1. with the change (they must pass: green), and
2. in a temporary git worktree where only the non-test changes are reverted (they should fail: red).

Then it reports, per test, whether the test catches the bug, and prints a markdown block you can paste into the pull request. Your checkout is never modified.

Maintainers get more and more pull requests, many of them written with AI, whose new tests pass whether or not the fix is there. Checking that by hand means stashing the fix, running the tests, and putting the fix back. This tool does that in one command, and the evidence block is cheap proof for the reviewer.

It is not mutation testing. Mutation testing makes up small code changes and checks that some test notices. This tool reverts the actual change and checks that the new tests notice.

Install

Node 20 or newer and git. There is no npm release yet, so run it from GitHub:

npx github:b84330808/redgreen-check main..

or clone the repo and run node bin/redgreen-check.js. It has no dependencies.

The test runners are yours: the tool runs pytest, jest, vitest, go test or cargo test with the dependencies you already installed in your checkout.

Usage

redgreen-check main..                  # this branch against main (from the merge base)
redgreen-check 1a2b3c4                 # one commit: 1a2b3c4^..1a2b3c4 (works for merge commits too)
redgreen-check origin/main...HEAD --exec "uv run pytest"
redgreen-check HEAD~3.. --cmd "make test"

Option
<range>	base..head, base...head, base.., or one commit. The diff is always taken from the merge base. Default: origin/HEAD...HEAD, or HEAD^..HEAD.
-r, --runner <name>	pytest, jest, vitest, go, cargo (repeatable). Default: detected per test file.
--cmd <command>	Run any shell command instead of an adapter. {files} expands to the changed test files. Exit 0 = pass.
--exec <prefix>	How to start the runner, e.g. "uv run pytest", "poetry run pytest", "pnpm exec vitest".
--setup <command>	Run in the worktree before each of the two runs, e.g. "npm run build" when tests import built files.
--test-glob <glob>	Also treat matching paths as tests (kept at head). Repeatable.
--source-glob <glob>	Always revert matching paths, even if they look like tests. Repeatable.
--keep <glob>	Keep matching non-test paths at head, e.g. "*.lock". Repeatable.
--strict	Also fail when a test fails without the change only because it needs the change to compile or import.
--timeout <sec>	Per run. Default 1800.
--format md|text|json	What to print on stdout. Default md (the evidence block). Progress and test output go to stderr.
--json <file>, --markdown <file>	Also write the full result / the evidence block to a file.
--keep-worktree	Leave the temporary worktree in place for debugging.
-C <dir>	Repository. Default: the current directory.
-q, --quiet	Do not stream test output.

Exit status: 0 the changed tests fail without the change and pass with it, 1 they don't (or fail with it), 2 error, 3 nothing to check (only tests changed, or no tests changed).

What the verdicts mean

Verdict	With the change	Without it	Meaning
✅ catches the bug	pass	fail	What you want.
⚠️ doesn't catch the bug	pass	pass	The test would pass on the old code too. Either it does not test the bug, or it guards behaviour that already worked. Worth a look either way.
🧱 needs the fix to build	pass	does not compile / import	The test uses API that the change adds (a Go build error, a Rust compile error, an ImportError, a missing export). It proves the new API works, not that the old bug is gone. Fails the check only with --strict.
❌ fails with the fix	fail	—	The test is broken at head.
❓ did not run	—	—	The runner did not report the test (e.g. a dynamic test name).

The check passes when every changed test is ✅ (or 🧱) and at least one test goes red.

How it works

1. Split the change. git diff <merge-base> <head> gives the changed files. Test files are recognised by name and directory: test_*.py, *_test.py, conftest.py, *_test.go, *.test.ts/*.spec.js (any JS/TS flavour), files under test/, tests/, __tests__/, spec/, testdata/, fixtures/, __snapshots__/, Rust tests/, Java *Test.java, and so on. Everything else is the fix.
2. Find the changed tests. Diff hunks are mapped to the test functions they touch: pytest node ids (tests/test_x.py::TestClass::test_y), jest/vitest describe › it titles, Go TestXxx functions, Rust #[test] functions. New test files count in full. When a test file changed only outside any test (imports, helpers), every test in it runs and is reported as (file). Changed snapshots and Go testdata/ put the tests that read them in scope.
3. Green run. git worktree add at head in a temporary directory, then run just those tests: pytest <node ids>, jest --runTestsByPath <files>, vitest run <files>, go test -run '^(TestA|TestB)$' ./pkg, cargo test -p <crate> -- <names>.
4. Red run. In the same worktree, put the base version of every non-test file back (git checkout <merge-base> -- …, and delete files the change added), keep the tests at head, and run the same tests again. Rust files with an inline #[cfg(test)] module are split by hunk: hunks inside the test module are kept, the rest are reverted.
5. Compare per test, print the report, and remove the worktree.

Using the dependencies you already installed

The worktree has no node_modules or virtualenv of its own, so the tool points it at yours, carefully, so that the worktree's code is what gets tested:

• Node: node_modules folders are linked into the worktree. Workspace packages that are symlinked back into your checkout (npm, yarn and pnpm workspaces) are re-pointed at the worktree copy, so a reverted workspace package is really reverted. Jest gets its own cache directory per run.
• Python: the interpreter is .venv/venv in your checkout if there is one, else python on PATH. sys.path entries and setuptools editable-install mappings that point into your checkout are re-pointed at the worktree through PYTHONPATH, so pip install -e . does not leak the fixed code into the red run. .pyc writing is turned off, so a revert with the same file size and timestamp cannot be masked by stale bytecode.
• Go: the module cache is shared; nothing else is needed.
• Rust: both runs share one temporary CARGO_TARGET_DIR.

If your tests need a build step from sources (C extensions, a dist/ folder that tests import), pass it as --setup; it runs before both runs.

GitHub Action

The Action runs on pull requests and writes the evidence block to the job summary. It does not comment on the pull request. Tests that pass without the change get a warning annotation on the test line.

name: red/green
on: pull_request
permissions:
  contents: read
jobs:
  redgreen:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: '3.13'
      - run: pip install -e . pytest
      - uses: b84330808/redgreen-check@v0

Install your dependencies first, as you would for your normal test job (npm ci, go mod download, …). With the default shallow checkout the Action fetches the base commit itself.

Input	Default
base	the PR base	For pull requests: the first parent of GitHub's merge commit. For pushes: the previous commit.
head	HEAD
runner, cmd, exec, setup, test-glob, source-glob, keep, strict, timeout		Same as the CLI options. List inputs take commas or newlines.
fail-on	fail	fail: fail the step when tests don't catch the change or fail with it. Add no-tests to also fail when code changed but no tests did. never: report only.
working-directory	.
annotations	true	Warning annotations for tests that pass without the change.

Outputs: outcome (pass, fail, no-tests, nothing, error), summary, caught, not-caught, cant-build, and report (path of the markdown file; the full JSON result is next to it).

Test output from the red run is printed with workflow commands disabled, so failures that the tests themselves print as ::error annotations (some reporters do this in CI) do not end up on the pull request.

Limitations

• Tests are matched by name. Tests with names built at run time (loops, test.each with computed titles) may show as "did not run".
• In Go a test file that uses new API makes the whole package fail to compile without the change, so every changed test in that package is reported as 🧱, including ones that would otherwise catch the bug. Rust behaves the same per test target.
• Tests that need a fresh build of the package (Python C extensions, packages installed non-editable from the checkout) need --setup.
• Yarn Plug'n'Play is not supported.
• Flaky tests give flaky verdicts; nothing is retried.
• Reverting is per file (plus Rust test modules). A test helper in a non-test file is reverted with the fix; use --test-glob or --keep to keep it.

Related work

• Mutation testing (mutmut, Stryker, cargo-mutants, PIT) measures how well a test suite notices random code changes. It answers a different question.
• SWE-bench-style harnesses record FAIL_TO_PASS tests for benchmark tasks; this tool does the same check on your own pull requests.
• kwlck/PatchProof is a GitHub App with a similar goal.

License

MIT