Skip to content

About

A defect-hunting pipeline: proves each defect with a reproduction that fails before the patch and passes after, then opens the fix upstream under the receiving project's rules.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

17 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

Defect Hunting pipeline

(The search is not deterministic. The gate is.)

It reads open-source codebases it does not own, isolates real defects, proves each one with a reproduction that fails before the patch and passes after, and opens the fix upstream under the rules of the project receiving it.

The finding is the work of a language model, so it is not repeatable: run it again tomorrow on the same repository and it will not surface the same defects. What is repeatable is the gate every candidate has to pass — a test that fails on the unpatched tree, passes on the patched one, in the project's own environment, and that anyone can re-run to the same result. That distinction is the whole design. Nothing downstream of it is trusted because a model said so.

It is built and operated by one person, Leo Camus, on hardware he owns. Every figure below is measured, and every one of them can be checked against public GitHub data by anyone who wants to.

Seven days

10–16 September 2026, one week of continuous operation.

Agent runs 357 over 151 hours
Pull requests opened, proven, written up 181 across 103 repositories
Merged 83, in 44 projects nobody here maintains
Closed without merging 7
Acceptance among decided pull requests 92.2 %
Cost $909 total, $10.95 per merged fix
Hunting runs that end with no pull request 14 % — they are included in the cost above

Checkable: merged · closed without merging. Those searches cover the whole account, so they include work that predates the pipeline.

Those figures were counted on 16 September at about 18:00 UTC, as this page was being written, which is why they stop mid-day rather than at midnight: 14 more pull requests went out that evening. Re-counting the same week on 18 September gives 195 opened, 108 merged, 8 closed without merging, an acceptance of 93.1 % among the decided ones. The table above is left as it was measured rather than quietly raised. As of 31 October the whole account stands at 310 merged of 434 pull requests opened, in 136 repositories, with 36 closed without merging and 88 still open: an acceptance of 89.6 % among the decided ones. Still one person, on hardware he owns, still the same gate. The weeks since have kept landing: a merged HLS URL-resolution fix on Cloudstream (“great fix and test … a very cool project”), and NASA's F´/FPP maintainers signing off on issues and pull requests with “Good catch!” and “Looks good!”.

One caveat about those searches, and it works against the figures rather than for them. Some maintainers apply a patch by rebasing it into the default branch themselves and then close the pull request, which GitHub can only display as closed, never as merged. Five contributions across the account are in that state, two of them in this week: osrg/gobgp #3604 and #3609, whose commits are in master with me as their author. The re-count above puts those two back on the right side. The platform searches do not.

The hardest review it has passed is the JavaScript engine v8/v8: two changelists through Gerrit, CLA and committer review, into the ECMA-262 implementation behind Chrome and Node.js. The fix it would point to first is NASA F´ #5972, where a framer dropped its status signal on a failed buffer allocation and stalled the downlink chain instead of reporting the error.

Two rules decide what is allowed to leave

  1. Nothing ships without a reproduction that fails before the patch and passes after. Not a plausible argument, not a static-analysis hit, not a linter opinion. A failing test that turns green, run in the project's own environment.
  2. Nothing ships until the receiving project's contribution rules are satisfied. Their rules beat ours, always, without asking. If a project forbids naming an AI tool, the disclosure line comes off. If it requires a trailer, a template section, a signed agreement or an accepted issue first, that comes first — or nothing goes out at all.

Every pull request says where it came from, in one sentence, in the body:

Found by a defect-hunting pipeline I build and run (Dev-next-gen), using Claude Code with Anthropic's Claude Opus 5.

That line is not optional, and it is not marketing. A maintainer deciding how much of their attention to spend is entitled to know what they are reading.

And a maintainer who would rather receive none of it is entitled to that too. One line is enough, no reason is needed, and it does not expire: opting out. How projects have actually reacted so far, the merges and the two refusals, with what was said in each case, is in reception.

The clearest measure is not the hardest review, it is a project that ships. On samuelgursky/davinci-resolve-mcp, nine fixes from this pipeline were merged and released in one week, each one cut into a version the same day, the last of them tonight as v4.7.10. The maintainer keeps the count himself, in the last line of every message:

Nine releases from your pipeline this week.

He does not take them on trust either, and that turns out to be the pattern rather than the exception. Three maintainers, on three unrelated projects, did the same thing before accepting a fix: they re-derived the proof instead of believing it.

  • samuelgursky, before merging the ninth, merged main into the branch in a scratch worktree, ran the full suite, then brute-forced the arithmetic both ways for every frame in two hours at 29.97 and 59.94, 648,000 conversions, and found his own measurement against DaVinci Resolve and the patch agree on the same number.
  • ChuckHend, on pgmq #593, opened his answer with "I was able to reproduce" and merged one minute later.
  • SirAllap, on agentglass, re-ran the measurements rather than take the table on trust, put each defect back one at a time to confirm the new guard caught it, and checked the two limits the pull request declared about itself.

None of them argued about where the finding came from. They ran it. That is what a proof is for: not to be believed, to be re-run. A patch that arrives with one turns a question of trust into a question of arithmetic, and arithmetic is the only part of this a stranger can settle alone. The cases, with the refusals beside them, are in reception.

When a maintainer asks for something

Since 18 September, on a project that has already merged something from this pipeline, a person with write access who describes work to be taken gets it taken. Not on the next pass, not after somebody reads the thread: the work starts and the pull request that follows quotes the request that caused it.

What counts as asking is deliberately wider than a formal request. An explicit invitation to open a separate pull request counts. So does a defect mentioned in passing without being fixed, a series of the same kind someone says they can see, and a difficulty they say they are stuck on. Those are the three shapes that kept appearing in threads where the work was wanted and nobody had asked for it in so many words.

This was built because it was measured, not because it sounded good. The two contributions started by hand from an invitation on 17 September, 0xJacky/nginx-ui #1933 and SirAllap/agentglass #597, were both merged within the hour. Nothing sent unasked has ever moved that fast.

Three conditions, all of them required:

  1. Write access. A passer-by's suggestion does not start anything, however good. A project's direction belongs to the people who carry it. This is the same signal that governs opting out, so the same word opens the door and closes it.
  2. A project that has already merged something. The relationship comes before the invitation is treated as one.
  3. The request has to be quoted word for word. If what the request is understood to be cannot be found verbatim in what was written, nothing starts. A paraphrase would end up in a public pull request putting words in a maintainer's mouth, and no amount of usefulness is worth that.

And one thing it deliberately will not do: a request about the pull request in front of you stays in that pull request. It never becomes a second one. Changing what you already have in hand is not the same act as opening something new, and only the second one needs your invitation.

What comes next

An agent a maintainer can call into a discussion, instead of receiving contributions nobody asked for: @Defect-Hunting-Pipeline. It has to be installed to be called, so consent is given once, by the project. It will never ask for write access to your repository, and it refuses to publish a vulnerability, to claim without proof, and to take instructions from the content it reads.

The rest

  • Journal — what this system becomes, dated, with the fact that prompted each change and what was expected of it.
  • Thesis — why the bottleneck everyone expects is not the one that shows up, and what we now measure about it.
  • Evidence — the cases behind the claims, with links.
  • Reception — how projects receive this, the refusals included, with the quotes.
  • Opting out — how a maintainer excludes a project, and what is not built yet.
  • Disclosure — what happens when a finding should not be published.
  • Agent — the invocable agent: what it does, what it refuses, what it will never ask for.
  • Roadmap — what would have to be true for this to run at a different size.

What this repository is not

This repository describes what the system produces, the rules it obeys, and the evidence for its claims. It does not describe how it is built. There is no source, no architecture, no prompts and no operational detail here, and there will not be.

About

A defect-hunting pipeline: proves each defect with a reproduction that fails before the patch and passes after, then opens the fix upstream under the receiving project's rules.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors