Repository navigation
Benchmark other review tools as attempts: PR-Agent and roborev #193
Description
Context
The review-quality benchmark (#179) scores attempts: one review tool with one provider and model. Today the only tool is marginal. To know whether a score is good, we need other reviewers on the same cases, with the same model and the same key, so a difference comes from the tool, not the model.
Proposal
Add two open-source reviewers as attempts, each in its own PR:
attempts:
- {tool: marginal, provider: anthropic, model: claude-opus-5-5}
- {tool: pr-agent, provider: anthropic, model: claude-opus-5-5}
- {tool: roborev, provider: anthropic, model: claude-opus-5-5}| Tool | License | How it runs on a case | Posts anything? |
|---|---|---|---|
| PR-Agent | MIT | reviews the case's upstream PR (source:) with the Anthropic key |
no: publish_output=false, and the workflow's token can't write to other repos |
| roborev | MIT | reviews the head commit in a git checkout, through its claude-code agent |
no, it's local |
Each tool gets an adapter that turns its review into the benchmark's finding shape (file, line, severity), so the scorer and the summary treat every attempt the same way. A tool that gives no confidence or severity shows those metrics as n/a rather than a guessed value.
GitHub Copilot code review is left out: its bring-your-own-key option covers Copilot Chat and the CLI, not code review, so it can't run on our key or model.
Acceptance criteria
- PR-Agent runs as an attempt in the workflow matrix and appears as its own column in the summary (Benchmark PR-Agent with Claude Opus 5.5 next to marginal #195).
- roborev the same, in a second PR.
- Neither tool can post to an upstream repo during a run.
- Before any comparison is published, each tool's terms are checked.
Activity