Skip to content

Benchmark the PR experience: script the marginal-client release checklist #208

Open

Description

@exactml

Context

The review-quality benchmark (#179, ADR 0008) answers "does marginal find the right bugs?". It doesn't answer "does marginal behave right on a PR?": a fixed finding gets resolved, a disputed one gets closed with credit, a draft or lockfile-only PR gets no model call, a PR can't switch off its own review. Today those are checked by hand, by opening PRs from the marginal-client release checklist and reading the result (ADR 0007). Nothing is scored, and a regression shows up only if someone happens to run the right scenario. #140's last run left scenarios 2–5 unchecked.

Proposal

Turn the marginal-client checklist into a scripted, scored suite that runs on the same triggers as review quality: release PRs and workflow_dispatch.

Each scenario is a script of steps on marginal-client plus checks on what GitHub shows afterwards:

# tests/integration/pr_experience/scenarios.yml
- id: fix-after-finding
  steps:
    - open_pr: {branch: bug-round-down}          # scenario 1's bug
    - expect: {inline_findings: ">=1"}
    - push: {patch: fix-round-down.patch}
    - expect: {thread_resolved: true, reply_contains: "✅ Resolved"}
- id: lockfile-only
  steps:
    - open_pr: {branch: lock-bump}
    - expect: {model_calls: 0, summary_contains: "1 file(s) not reviewed"}
Metric What it measures
Scenarios passed expectations met ÷ all expectations, per scenario
Model calls calls made vs. expected (skips must cost nothing)
Time to review seconds from push to posted review

The runner opens each PR, waits for the review, checks the expectations through the GitHub API, then closes the PR and deletes its branches, so marginal-client is left as it started.

Scope

  • Start with the checklist scenarios that need no human: 1 (real bug), 3 (clean), 4 (fix after a finding), 6 (lockfile-only), 7 (draft), 8 (opt-out label), 9 (secret in a URL), 10 (PR silences its own review).
  • Scenario 5 (disputed finding) needs a reply from a second account; follow-up.
  • Soft gate like review quality: warns in the run summary, never fails the build.
  • Results go to the run summary and score.json, next to the review-quality results.

Cost

Scenarios 6, 7 (draft part) and 8 make no model call. The others are one review each, plus one more per later push (scenario 4), so about 8 reviews per run. Run it only on release PRs and on demand, like review quality.

Acceptance criteria

  • One command (make pr-experience) runs the scenarios above on marginal-client and writes a pass/fail table.
  • The workflow runs on a release PR and on demand, and warns without failing.
  • A run leaves no open PRs or scenario branches on marginal-client.
  • The marginal-client README checklist points to the suite for the scripted scenarios.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions