Skip to content

verifier headline ranks a medium trust-root notice above critical release blockers #365

Closed

Description

@pengfei-threemoonslab

Summary

When a PR both touches the trust root and contains critical release blockers, verifier.json.headline reports the trust-root fact and the blockers are not mentioned. On first adoption — where touching the trust root is unavoidable — the headline is guaranteed to describe governance plumbing instead of the safety finding that actually blocks the merge.

Evidence (reproduced 2026-08-10, CLI 0.16.0b7, contract 21)

verify --base <merge-base> --head sg-head on google/adk-samples PR #1917, with shipgate.yaml introduced by the same diff (first adoption):

merge_verdict:           blocked
control.state:           human_review_required   (must_stop: true)
base_status:             missing_manifest
headline: "This PR weakens the release policy that evaluates it; a coding agent cannot
           self-approve that change — a human must review it."

Meanwhile, in the same artifact:

release_decision.decision: blocked
release_decision.reason:   4 active findings block release
blockers[0..3]: SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING on spraay_batch_eth,
                spraay_batch_eth_variable, spraay_batch_token, spraay_batch_token_variable

The two findings driving the headline are both medium:

  • SHIP-VERIFY-TRUST-ROOT-TOUCHED — Release trust root touched: …/shipgate.yaml
  • SHIP-VERIFY-POLICY-WEAKENED — Policy change cannot be proven safe (no base snapshot)

SHIP-VERIFY-POLICY-WEAKENED here does not mean the policy was weakened; it means there was no base manifest to compare against, which is definitionally true for every first adoption (base_status: missing_manifest). So on first adoption the headline is always the governance message, regardless of what the PR does.

The subject of this PR is an agent that batch-pays ETH and ERC-20 on Base mainnet whose only confirmation is prompt text ("Reply 'confirm' to execute"). That does not appear in the headline.

Impact

headline is the one line that reaches a PR comment, a chat reply, or a triage list. Ranking a medium governance notice above four critical financial-write blockers understates severity at the exact moment a human is deciding how much attention to spend — and it does so most reliably for new adopters, whose first impression is that the tool is about its own configuration rather than about their agent.

Proposal

  1. Rank headline by the strongest signal: critical/high release blockers outrank trust-root and policy notices. When both are present, lead with the blocker and append the review requirement.
  2. Distinguish "policy weakened relative to a base" from "no base snapshot exists." The latter is a first-adoption condition, not a weakening claim; give it its own reason code and a message that says so.

Before / After

Before

This PR weakens the release policy that evaluates it; a coding agent cannot self-approve that change — a human must review it.

After

Release blocked: 4 tools can move funds with no enforced controls. This PR also introduces the Shipgate policy that evaluates it, so a human — not the coding agent — must review both.

Acceptance criteria

  • Repro's headline names the financial-write blockers first and retains the human-review requirement.
  • control.state, must_stop, merge_verdict, can_merge_without_human, and permissions are unchanged — ordering only.
  • First adoption (base_status: missing_manifest) emits a distinct reason code from a real base-relative weakening, with distinct copy.
  • A trust-root-only PR with no blockers still leads with the trust-root message.

Scope / non-goals

No change to gating, control state, or human-review routing. This is headline selection and one reason-code split. It does not relax the first-adoption human-review requirement, which stays correct.

Headline metric

Merge-decision comprehension on first adoption — the fraction of blocked verdicts whose headline names the blocking cause. Currently 0% for first-adoption PRs that also carry blockers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Picked up inside the current milestone window; keep about 15 openarea:verifyverify / preview orchestration and SHIP-VERIFY-* checksbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions