Skip to content

Commit 4055234

Browse files
fix(evidence): a proven read is bounded, not unread (#419 review)
Four findings from review of the ordering change. **[P1] `effect_is_measured` was the wrong predicate to rank with.** It is a proposal-safety rule: it returns False for every read-only reading, however authoritative, because a pre-filled `effect: read` is the one direction where a confirmed guess loses safety (#357). An OpenAPI `GET`, a trusted `readOnlyHint: true`, and a reviewed `effect: read` are all *proven* reads, and all three read as "nothing to propose" — so ranking with it sent them to the ceiling and let the name band break the tie among them. A structural `GET` named `delete_account` led the questionnaire ahead of a genuinely unknown effect: this issue's own defect inverted, with a repository-chosen name ordering something the scan had established. Ordering now asks `effect_is_bounded`, which is two clauses because the resolver records the two cases differently: an effect status of `declared`, `structural`, or `conflicting` — which is what catches a reviewed `effect: read`, since a declaration leaves no reading behind — or an observed side effect, which is the `inferred` action whose keyword hint reads `external_communication`. A heuristic reading of `read` is in neither: this resolver may not act on it, so the answer stays open. Three regressions cover it, all named to look as mutating as the band can score, and all three fail against the old predicate. `_PendingQuestion` is now seeded from its first action rather than from a zero floor, which a bounded `read` reaches exactly. **[P2] The published ordering contract said the old thing.** `DeclarationQuestionRow`'s docstring is emitted verbatim into the report, packet, and verifier schemas, and it — with `docs/agent-contract-current.md` — still said "highest-acting action first". Both now describe the ranking and state that position is not severity: the action at the top is the one *least* is known about. Schemas and `llms-full.txt` regenerated. No field shapes change. **[P2] The header still excluded one member of its own tier.** "could read nothing about" is false for an action whose only reading is a heuristic `read` — it prints that reading above its block and is still unbounded. The header now describes the tier as what nothing has pinned down, "no effect evidence at all, or only a reading this scan is not allowed to act on", and the rendered acceptance test carries all three shapes: nothing read, a protocol default, and a heuristic read. **[P3] The block note claimed a position it may not have.** `build_declaration_scaffold(..., questions=None)` is supported for older reports and preserves gap emission order, so a blank can follow a bounded question. The sentence claiming placement is now emitted only when the block is actually numbered by coverage; the rest of the note, which says what the silence means, is unconditional. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent a8951f9 commit 4055234

12 files changed

Lines changed: 441 additions & 126 deletions

‎CHANGELOG.md‎

Lines changed: 31 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,23 @@
1616

1717
*Rank by the ceiling, not by the floor.* Observed risk and "how much can
1818
answering this move the verdict" are not the same quantity, and the header
19-
claimed the second. An action nothing was observed about is not a low-risk
20-
action; it is an unmeasured one, its answer can still turn out to be
21-
`destructive`, and it is exactly where a human answer carries new
22-
information. A question about an unmeasured action now sorts above every
23-
measured one, and the measured ones keep their old order among themselves —
24-
strongest first. On the same walk the financial write moves from Q6 to Q3 and
25-
all three drafts move to the end.
19+
claimed the second. An action nothing has bounded is not a low-risk action;
20+
it is an unmeasured one, its answer can still turn out to be `destructive`,
21+
and it is exactly where a human answer carries new information. A question
22+
about an unbounded action now sorts above every bounded one, and the bounded
23+
ones keep their old order among themselves — strongest first. On the same
24+
walk the financial write moves from Q6 to Q3 and all three drafts move to the
25+
end.
26+
27+
*Bounded is not the same test as draftable.* A reviewed declaration and
28+
policy-eligible source evidence bound an action even when what they establish
29+
is `read`, and the rule that decides whether to pre-fill a value cannot say
30+
so: it refuses to draft `effect: read` from anything, because a confirmed
31+
guess of `read` is the one direction that loses safety. Ranking on that rule
32+
would send an OpenAPI `GET` named `delete_account` to the top of the file
33+
with its name breaking the tie — the same defect inverted — so ordering asks
34+
its own question. A heuristic reading of `read` still bounds nothing: this
35+
resolver may not act on it, so the answer remains open.
2636

2737
*And a name breaks the tie among blanks.* Where nothing was observed there is
2838
nothing to rank by, so the questionnaire falls back to the shape of the
@@ -33,8 +43,20 @@
3343
or a verdict. Getting it wrong costs a reader one place in a list they have
3444
to finish either way.
3545

36-
The header sentence now states the order the file actually uses, and a test
37-
renders the file and checks the two against each other.
46+
The header sentence now states the order the file actually uses — including
47+
the heuristic-read case, where a block prints a reading and is still
48+
unbounded — and a test renders the file and checks the two against each
49+
other. A blank with no reading at all now says so at the block, since the
50+
header explains that the top of the file is the unbounded half and silence
51+
read as "nothing to see here".
52+
53+
*Published contract.* `report.json` /
54+
`semantic_coverage.declaration_questions.open_questions[]` documented itself
55+
as "highest-acting action first", which is no longer what it is. The model
56+
docstring is emitted verbatim into the report, packet, and verifier schemas,
57+
so it and `docs/agent-contract-current.md` now describe the ranking above and
58+
say plainly that position is not severity: the action at the top is the one
59+
*least* is known about. Field shapes are unchanged.
3860

3961
- **One action, one permission list, with no reviewed authority either.** A
4062
manifest row that listed `scopes:` and declared no `authority:` block at

‎docs/agent-contract-current.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -415,7 +415,7 @@ In `agents-shipgate-reports/report.json`:
415415
- `release_decision.{blockers,review_items}[].capability_refs` (v0.24+) — stable capability IDs copied from the originating finding when a policy or policy-pack rule matched a `CapabilityFactV1`. Empty for findings that are not capability-policy matches. This is audit metadata only; `release_decision.decision` remains the gate.
416416
- `release_decision.{blockers,review_items}[].capability_trace_refs` (v0.25+) — stable local trace-evidence IDs copied from the originating finding when an existing trace/evidence check used declared local trace artifacts. Empty when no local trace row is relevant. This is audit metadata only; `release_decision.decision` remains the gate.
417417
- `release_decision.evidence_coverage.semantic_coverage` (v0.29+) — `{total_actions, pass_eligible_actions, gap_count, review_concern_count, reason_counts}`. A non-zero semantic `gap_count` prevents `passed`; a non-zero `review_concern_count` prevents an automatic pass and routes known review concerns to human review — unscoped/ambient authority, and (v0.36+) `acknowledged_effect_override`, a declared effect a reviewer acknowledged as weaker than the evidence inferred for it. Read `reason_counts` for which; the count is of concerns, so one action can contribute more than one. Semantic gaps are not Findings and cannot be suppressed, baselined, severity-overridden, waived by `--no-heuristics`, or satisfied by `human_ack`.
418-
- `release_decision.evidence_coverage.semantic_coverage.declaration_questions` (v0.37+) — the same action surface counted as a questionnaire: `{total, answered, open, open_by_dimension, open_questions[]}`. A *question* is one `(action, dimension)` a reviewed `action_surface.actions` row has to answer, and only `effect` and `authority` are counted — an action whose effect the scan established by itself (an OpenAPI method, an MCP annotation) was never asked and is not in `total`, and an inventory or `agent_bindings` declaration has no per-action counterfactual to score against. `answered` is exact rather than optimistic: it counts dimensions that gap when the same action is re-resolved *without* its declaration. `total == answered + open`, and `open_by_dimension` sums to `open`. `open_questions[]` is the answer order — highest-acting action first, `effect` before `authority` — and joins to `evidence_gaps[].subject_id`. Nothing here gates; it is a projection of counts the decision already made, published so a coding agent (and the generated questionnaire) can report progress instead of a gap tally.
418+
- `release_decision.evidence_coverage.semantic_coverage.declaration_questions` (v0.37+) — the same action surface counted as a questionnaire: `{total, answered, open, open_by_dimension, open_questions[]}`. A *question* is one `(action, dimension)` a reviewed `action_surface.actions` row has to answer, and only `effect` and `authority` are counted — an action whose effect the scan established by itself (an OpenAPI method, an MCP annotation) was never asked and is not in `total`, and an inventory or `agent_bindings` declaration has no per-action counterfactual to score against. `answered` is exact rather than optimistic: it counts dimensions that gap when the same action is re-resolved *without* its declaration. `total == answered + open`, and `open_by_dimension` sums to `open`. `open_questions[]` is the answer order and joins to `evidence_gaps[].subject_id`. v0.38 ranks it by the ceiling of what an answer can establish: the actions nothing has bounded first — no effect evidence, a protocol default standing in for its absence, or only a heuristic reading the scan may not act on — then the actions a reviewed declaration or policy-eligible source evidence established, strongest-acting first, with `effect` before `authority` within one action. Position is not severity: the action at the top is the one *least* is known about. Nothing here gates; it is a projection of counts the decision already made, published so a coding agent (and the generated questionnaire) can report progress instead of a gap tally.
419419
- `release_decision.evidence_coverage.policy_gap_count` and top-level `policy_evidence_gaps[]` (v0.33+) — policy applicability that is heuristic-only, mixed, unknown, or conflicting. These rows are outside Findings and cannot be suppressed, baselined, severity-overridden, acknowledged, or removed by `--no-heuristics`; any row prevents `passed`.
420420
- `release_decision.evidence_coverage.identity_coverage` (v0.30+) — `{total_observations, canonical_tools, bound_tools, pass_eligible_tools, ambiguous_name_count, gap_count, reason_counts}`. Provider-scoped observations remain separate unless an exact reviewed `tool_identity.bindings[]` entry joins them. Any ambiguous selector, invalid binding, or conflicting identity prevents `passed`.
421421
- `release_decision.evidence_coverage.evidence_gaps[]` (v0.26+; semantic kinds added v0.29) — one structured row per measurable gap: `{kind, subject, source_type, source_ref, why, next_action}`. In addition to `low_confidence_tool` and `source_warning`, v0.29 adds `incomplete_surface`, `missing_effect_evidence`, `inferred_effect_only`, `conflicting_effect_evidence`, `missing_authority_evidence`, `partial_authority_evidence`, `conflicting_authority_evidence`, and `invalid_semantic_annotation`. v0.36 adds `declaration_below_inferred_evidence`: the declared effect is weaker than evidence this scan inferred for the same action. The declaration still stands as the operative effect — heuristics never drive a verdict — but the action is not evidence-backed-pass until a reviewer raises the declared effect or adds `action_surface.actions[].override` with the `evidence` they checked and the `reason` it does not apply. An acknowledged override keeps the action pass-eligible and is reported as one semantic review concern, so the run can never read `passed`. Each acknowledgement is also emitted as a row in `release_decision.evidence_coverage.semantic_coverage.acknowledged_overrides[]` (v0.36+) naming the action, both readings, the hint source, any source evidence that agrees, and the reviewer's evidence and reason — the packet's §1 and the PR comment render it, because a count is not a review surface. The acknowledgement is consumed by policy applicability as well, so applying it reaches the review route rather than trading one gap for another. Semantic next actions use `declare_action_effect`, `declare_action_authority`, `declare_tool_inventory`, `provide_complete_inventory`, or `resolve_semantic_conflict`, include accepted values and exact source/manifest pointers, and are always human-routed. v0.37 adds `next_action.observed_readings[]` on effect rows — `{effect, sources[], observed}`, the distinct readings this scan's non-declaration evidence supports, so the row can be answered without opening `action_surface_facts`. Where those readings support one conservative answer, `next_action.declaration_template` carries it **pre-filled** instead of a `<REVIEW_REQUIRED>` blank. That value is a proposal, not an assertion: it is drawn from the closed `ActionEffect` vocabulary (never from source content), it is never weaker than any reading, and it is offered only where something was observed — a protocol default standing in for the absence of evidence, or a heuristic reading of `read`, keeps the blank. Their declaration placeholders carry `suggested_patch_kind="manual"`, `auto_apply=false`, and `requires_human_review=true`; they are not Patch objects, and a pre-filled template is still operative only once a human merges it into the manifest. v0.37 also re-routes `partial_authority_evidence`: it is raised when the *source's* authority evidence is ambiguous or incomplete, and the resolver preserves it whatever the manifest declares ("reviewed authority cannot replace ambiguous or incomplete source authority alternatives"). Its `next_action.kind` is therefore `provide_source` with no declaration template, rather than a `declare_action_authority` block that could not close the row it was printed on. It is excluded from `declaration_questions` for the same reason. Work the rows in order instead of guessing; Agents Shipgate never auto-asserts effect or authority.

‎docs/engineering/insufficient-evidence-cold-start.md‎

Lines changed: 56 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -574,18 +574,50 @@ action; it is an unmeasured one. Its answer can still turn out to be
574574
### Rank by the ceiling
575575

576576
A question is now ranked by the *ceiling* of what its answer can establish
577-
rather than by the floor the scan already inferred. Where the scan measured a
578-
side effect, that measurement bounds the answer from below and the old rank
579-
still applies. Where it measured nothing, there is no bound at all, so the
580-
question sorts above every measured one (`UNMEASURED_EFFECT_RANK`). On the same
581-
walk the financial write moves Q6 → Q3, and all three drafts move to the end.
582-
583-
"Measured" and "could be drafted" are one fact, so they are one predicate:
584-
`effect_is_measured` is the gate `propose_effect_declaration` already applied,
585-
now read from both ends. A heuristic reading of `read` is deliberately *not* a
586-
measurement — this resolver refuses to establish a read-only action from a
587-
heuristic, so such an action is exactly as unproven as one with no reading at
588-
all.
577+
rather than by the floor the scan already inferred. Where something already
578+
bounds the effect, the answer cannot go below it and the old rank still
579+
applies. Where nothing does, there is no bound at all, so the question sorts
580+
above every bounded one (`UNBOUNDED_EFFECT_RANK`). On the same walk the
581+
financial write moves Q6 → Q3, and all three drafts move to the end.
582+
583+
### "Bounded" is not "draftable", and the difference is a whole class of bug
584+
585+
The first draft of this used `effect_is_measured` — the gate
586+
`propose_effect_declaration` already applies — on the reasoning that "measured"
587+
and "could be drafted" are one fact. They are, and that is exactly why it was
588+
the wrong predicate to rank with.
589+
590+
`effect_is_measured` is a **proposal-safety** rule. It returns `False` for every
591+
read-only reading, however authoritative, because a pre-filled `effect: read`
592+
is the one direction where a reviewer confirming without thinking loses safety.
593+
An OpenAPI `GET`, an MCP `readOnlyHint: true` the manifest trusts, and a
594+
reviewed `effect: read` are all *proven* reads — and all three read as "nothing
595+
to propose".
596+
597+
Ranked with it, they went to the **ceiling**, and the name band then broke the
598+
tie among them. A structural `GET` named `delete_account` led the questionnaire,
599+
ahead of a genuinely unknown effect — this round's own defect inverted, with a
600+
repository-chosen name ordering something the scan had established. Found in
601+
review, not by the tests, because every fixture had been built from the walk,
602+
where nothing is a proven read.
603+
604+
`effect_is_bounded` is the ordering question, and it is two clauses because the
605+
resolver records the two cases differently:
606+
607+
- an effect status of `declared`, `structural`, or `conflicting` — the resolver
608+
established it (or established that its sources disagree, which is still a
609+
bound). This clause is what catches a reviewed `effect: read`, which leaves
610+
*no reading at all* behind: declaration claims are excluded from readings on
611+
purpose, because a row is not evidence about itself;
612+
- otherwise, an observed side effect — the `inferred` action whose keyword hint
613+
reads `external_communication`.
614+
615+
A heuristic reading of `read` is deliberately in neither: this resolver refuses
616+
to establish a read-only action from a heuristic, so such an action is exactly
617+
as unproven as one with no reading at all. It is the one tier member that
618+
*prints* a reading above its block, which is why the header describes the tier
619+
as what nothing has pinned down rather than as what the scan "read nothing
620+
about".
589621

590622
### A name may order what it may not judge
591623

@@ -610,6 +642,11 @@ trust, and is given none:
610642
where a claim, an issue, or a verdict can see it, an unreviewed reading of a
611643
repository-chosen string has become evidence.
612644

645+
The `delete_account` GET above is what that second point is worth. The band was
646+
correctly scoped to one tier the whole time; what put a name in charge of a
647+
proven action was the *tier* being wrong, one predicate away. A guard on the
648+
band alone would not have caught it.
649+
613650
Getting a band wrong costs a reader one place in a list they must finish
614651
either way.
615652

@@ -635,3 +672,10 @@ what the header now says.
635672
you already inferred.* Where those two differ, the second one ranks a
636673
questionnaire by how much it already knows, which is the reverse of what the
637674
reader is there to do.
675+
676+
**And the one it adds twice:** *a safety rule is not a general-purpose
677+
predicate.* `effect_is_measured` is exactly right for deciding what may be
678+
pre-filled and exactly wrong for deciding what is known, and the two read as
679+
the same sentence right up until an OpenAPI `GET` sorts above an unknown. When
680+
a predicate's name describes its answer rather than its question, borrowing it
681+
is how the borrower inherits a rule it never wanted.

‎docs/packet-schema.v0.15.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -547,7 +547,7 @@
547547
},
548548
"DeclarationQuestionRow": {
549549
"additionalProperties": false,
550-
"description": "v0.37: one open declaration question, in the order to answer it.\n\nOrdered strongest-acting action first. Answering the two that move money\nand communicate outward is what reached a verdict on the fourth\n``adk-samples#1745`` walk, so the list leads with them rather than with\nwhatever sorts first alphabetically. Ordering is ranking only \u2014 it decides\nwhat to read first and can never change a verdict.\n\nv0.38: a question is one blank a reviewer fills, so ``answer_path`` \u2014 the\nmanifest block that blank lives in \u2014 is its identity. Actions answered by\nthe same block are one question: a source of 117 actions with no authority\nevidence owes one ``tool_sources[].authority`` block, and counting that as\n117 questions describes one edit as a backlog (#410 increment 3).",
550+
"description": "v0.37: one open declaration question, in the order to answer it.\n\nOrdered by how much answering can move the verdict, rather than by what\nsorts first alphabetically. Answering the two actions that moved money and\ncommunicated outward is what reached a verdict on the fourth\n``adk-samples#1745`` walk. Ordering is ranking only \u2014 it decides what to\nread first and can never change a verdict.\n\nv0.38: a question is one blank a reviewer fills, so ``answer_path`` \u2014 the\nmanifest block that blank lives in \u2014 is its identity. Actions answered by\nthe same block are one question: a source of 117 actions with no authority\nevidence owes one ``tool_sources[].authority`` block, and counting that as\n117 questions describes one edit as a backlog (#410 increment 3).\n\nv0.38 also ranks by the **ceiling** of what an answer can establish rather\nthan by the effect already inferred. The actions nothing has bounded lead \u2014\nno effect evidence at all, a protocol default standing in for its absence,\nor only a heuristic reading this scan may not act on \u2014 because an unbounded\naction is unmeasured rather than safe, and its answer can still turn out to\nbe anything. Actions whose effect a reviewed declaration or policy-eligible\nsource evidence established keep their evidence rank behind them, strongest\nfirst. Position is not severity: the action at the top is the one *least*\nis known about.",
551551
"properties": {
552552
"answer_path": {
553553
"default": "",

0 commit comments

Comments
 (0)