Repository navigation
Description
Context
The review-quality benchmark (#179, ADR 0008) answers "does marginal find the right bugs?". It doesn't answer "does marginal behave right on a PR?": a fixed finding gets resolved, a disputed one gets closed with credit, a draft or lockfile-only PR gets no model call, a PR can't switch off its own review. Today those are checked by hand, by opening PRs from the marginal-client release checklist and reading the result (ADR 0007). Nothing is scored, and a regression shows up only if someone happens to run the right scenario. #140's last run left scenarios 2–5 unchecked.
Proposal
Turn the marginal-client checklist into a scripted, scored suite that runs on the same triggers as review quality: release PRs and workflow_dispatch.
Each scenario is a script of steps on marginal-client plus checks on what GitHub shows afterwards:
# tests/integration/pr_experience/scenarios.yml
- id: fix-after-finding
steps:
- open_pr: {branch: bug-round-down} # scenario 1's bug
- expect: {inline_findings: ">=1"}
- push: {patch: fix-round-down.patch}
- expect: {thread_resolved: true, reply_contains: "✅ Resolved"}
- id: lockfile-only
steps:
- open_pr: {branch: lock-bump}
- expect: {model_calls: 0, summary_contains: "1 file(s) not reviewed"}| Metric | What it measures |
|---|---|
| Scenarios passed | expectations met ÷ all expectations, per scenario |
| Model calls | calls made vs. expected (skips must cost nothing) |
| Time to review | seconds from push to posted review |
The runner opens each PR, waits for the review, checks the expectations through the GitHub API, then closes the PR and deletes its branches, so marginal-client is left as it started.
Scope
- Start with the checklist scenarios that need no human: 1 (real bug), 3 (clean), 4 (fix after a finding), 6 (lockfile-only), 7 (draft), 8 (opt-out label), 9 (secret in a URL), 10 (PR silences its own review).
- Scenario 5 (disputed finding) needs a reply from a second account; follow-up.
- Soft gate like review quality: warns in the run summary, never fails the build.
- Results go to the run summary and
score.json, next to the review-quality results.
Cost
Scenarios 6, 7 (draft part) and 8 make no model call. The others are one review each, plus one more per later push (scenario 4), so about 8 reviews per run. Run it only on release PRs and on demand, like review quality.
Acceptance criteria
- One command (
make pr-experience) runs the scenarios above on marginal-client and writes a pass/fail table. - The workflow runs on a release PR and on demand, and warns without failing.
- A run leaves no open PRs or scenario branches on marginal-client.
- The marginal-client README checklist points to the suite for the scripted scenarios.
Activity