Skip to content

Roll up human-facing findings by subject; derive recommendation from evidence.missing #364

Closed

Description

@pengfei-threemoonslab

Summary

Human-facing output lists findings flat, ordered by severity, so one fact about four tools is presented as 17 separate rows across 5 check families. Separately, recommendation text is templated and restates requirements the user has already satisfied, contradicting the same finding's evidence.missing.

Evidence (reproduced 2026-08-10, CLI 0.16.0b7)

Final scan on google/adk-samples PR #1917 — 17 findings, 4 subjects:

check_id sev count subject
SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING critical 4 the 4 spraay tools
SHIP-AUTH-MISSING-SCOPE high 4 the same 4
SHIP-SIDEFX-IDEMPOTENCY-MISSING high 4 the same 4
SHIP-MANIFEST-HIGH-RISK-OWNER-MISSING high 4 the same 4
SHIP-ADK-GUARDRAIL-EVIDENCE-MISSING high 1 aggregate over the same 4

Top findings: shows 3 rows — three instances of the same check on three different tools — so the human summary spends its entire budget on one check family and never mentions idempotency, scope, owner, or guardrails.

Recommendation drift, same finding:

"evidence": {"missing": ["safeguards.audit_log", "safeguards.idempotency"]},
"recommendation": "Declare approval.required, safeguards.audit_log, and safeguards.idempotency
                   for this financial write action."

approval.required: true was already declared in action_surface.actions; that is precisely why it is absent from evidence.missing. The recommendation instructs the user to re-declare it.

Impact

Direct user report from the walkthrough: "痛点不聚焦…看到了 finding 也不知道改什么" ("the pain isn't focused… I see the findings and still don't know what to change"). The underlying report is correct and complete; the rendering scatters a single decision ("these four tools move money with no enforced controls") across 17 rows sorted by an axis (severity) that is orthogonal to how the reader acts. Following the drifting recommendation wastes a round and teaches distrust of the text.

Proposal

  1. Group the human summary (stdout, report.md, pr-comment.md) by subject — tool, then aggregate findings — with severity as an attribute rather than the sort key.
  2. Render recommendation from evidence.missing rather than a static per-check template; drop satisfied requirements.
  3. Top findings: selects distinct subjects (or distinct check families), never N instances of one check on sibling tools.

Before / After

Before

Top findings:
- SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING: spraay_batch_eth - ...
- SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING: spraay_batch_eth_variable - ...
- SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING: spraay_batch_token - ...

After

4 tools move funds with no enforced controls (financial_write, ambient authority):
  spraay_batch_eth, spraay_batch_eth_variable, spraay_batch_token, spraay_batch_token_variable
    BLOCKS RELEASE  missing: audit_log, idempotency
    review          no auth scopes · no idempotency evidence · no owner declared
    review          no ADK callback/plugin guardrail evidence for this group

Acceptance criteria

  • Repro's stdout presents 4 tool groups + 1 aggregate instead of 17 flat rows; blocking status stays visible per group.
  • No recommendation names an item absent from that finding's evidence.missing.
  • report.json findings[] is byte-identical — per-finding structure and count unchanged.
  • report.md / pr-comment.md goldens updated; SARIF unchanged.

Scope / non-goals

Presentation only. No change to finding identity, fingerprints, counts, severities, blocks_release, or the machine contract. report.json remains the flat per-finding record that automation consumes.

Headline metric

Time-to-first-correct-remediation on a blocked verdict, and removal of one guaranteed wasted iteration (the already-satisfied approval.required). No schema bump; renderer plus recommendation-construction change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Queued; valuable but not blockingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions