You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4055234
Browse filesBrowse the repository at this point in the historyBrowse files
fix(evidence): a proven read is bounded, not unread (#419 review)
Four findings from review of the ordering change.
**[P1] `effect_is_measured` was the wrong predicate to rank with.** It is a
proposal-safety rule: it returns False for every read-only reading, however
authoritative, because a pre-filled `effect: read` is the one direction where a
confirmed guess loses safety (#357). An OpenAPI `GET`, a trusted
`readOnlyHint: true`, and a reviewed `effect: read` are all *proven* reads, and
all three read as "nothing to propose" — so ranking with it sent them to the
ceiling and let the name band break the tie among them. A structural `GET`
named `delete_account` led the questionnaire ahead of a genuinely unknown
effect: this issue's own defect inverted, with a repository-chosen name
ordering something the scan had established.
Ordering now asks `effect_is_bounded`, which is two clauses because the
resolver records the two cases differently: an effect status of `declared`,
`structural`, or `conflicting` — which is what catches a reviewed `effect:
read`, since a declaration leaves no reading behind — or an observed side
effect, which is the `inferred` action whose keyword hint reads
`external_communication`. A heuristic reading of `read` is in neither: this
resolver may not act on it, so the answer stays open.
Three regressions cover it, all named to look as mutating as the band can
score, and all three fail against the old predicate. `_PendingQuestion` is now
seeded from its first action rather than from a zero floor, which a bounded
`read` reaches exactly.
**[P2] The published ordering contract said the old thing.**
`DeclarationQuestionRow`'s docstring is emitted verbatim into the report,
packet, and verifier schemas, and it — with
`docs/agent-contract-current.md` — still said "highest-acting action first".
Both now describe the ranking and state that position is not severity: the
action at the top is the one *least* is known about. Schemas and `llms-full.txt`
regenerated. No field shapes change.
**[P2] The header still excluded one member of its own tier.** "could read
nothing about" is false for an action whose only reading is a heuristic `read`
— it prints that reading above its block and is still unbounded. The header now
describes the tier as what nothing has pinned down, "no effect evidence at all,
or only a reading this scan is not allowed to act on", and the rendered
acceptance test carries all three shapes: nothing read, a protocol default, and
a heuristic read.
**[P3] The block note claimed a position it may not have.**
`build_declaration_scaffold(..., questions=None)` is supported for older
reports and preserves gap emission order, so a blank can follow a bounded
question. The sentence claiming placement is now emitted only when the block is
actually numbered by coverage; the rest of the note, which says what the
silence means, is unconditional.
Co-Authored-By: Claude Opus 5 <[email protected]>
Copy file name to clipboardExpand all lines: docs/agent-contract-current.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -415,7 +415,7 @@ In `agents-shipgate-reports/report.json`:
415
415
-`release_decision.{blockers,review_items}[].capability_refs` (v0.24+) — stable capability IDs copied from the originating finding when a policy or policy-pack rule matched a `CapabilityFactV1`. Empty for findings that are not capability-policy matches. This is audit metadata only; `release_decision.decision` remains the gate.
416
416
-`release_decision.{blockers,review_items}[].capability_trace_refs` (v0.25+) — stable local trace-evidence IDs copied from the originating finding when an existing trace/evidence check used declared local trace artifacts. Empty when no local trace row is relevant. This is audit metadata only; `release_decision.decision` remains the gate.
417
417
-`release_decision.evidence_coverage.semantic_coverage` (v0.29+) — `{total_actions, pass_eligible_actions, gap_count, review_concern_count, reason_counts}`. A non-zero semantic `gap_count` prevents `passed`; a non-zero `review_concern_count` prevents an automatic pass and routes known review concerns to human review — unscoped/ambient authority, and (v0.36+) `acknowledged_effect_override`, a declared effect a reviewer acknowledged as weaker than the evidence inferred for it. Read `reason_counts` for which; the count is of concerns, so one action can contribute more than one. Semantic gaps are not Findings and cannot be suppressed, baselined, severity-overridden, waived by `--no-heuristics`, or satisfied by `human_ack`.
418
-
- `release_decision.evidence_coverage.semantic_coverage.declaration_questions` (v0.37+) — the same action surface counted as a questionnaire: `{total, answered, open, open_by_dimension, open_questions[]}`. A *question* is one `(action, dimension)` a reviewed `action_surface.actions` row has to answer, and only `effect` and `authority` are counted — an action whose effect the scan established by itself (an OpenAPI method, an MCP annotation) was never asked and is not in `total`, and an inventory or `agent_bindings` declaration has no per-action counterfactual to score against. `answered` is exact rather than optimistic: it counts dimensions that gap when the same action is re-resolved *without* its declaration. `total == answered + open`, and `open_by_dimension` sums to `open`. `open_questions[]` is the answer order — highest-acting action first, `effect` before `authority` — and joins to `evidence_gaps[].subject_id`. Nothing here gates; it is a projection of counts the decision already made, published so a coding agent (and the generated questionnaire) can report progress instead of a gap tally.
418
+
- `release_decision.evidence_coverage.semantic_coverage.declaration_questions` (v0.37+) — the same action surface counted as a questionnaire: `{total, answered, open, open_by_dimension, open_questions[]}`. A *question* is one `(action, dimension)` a reviewed `action_surface.actions` row has to answer, and only `effect` and `authority` are counted — an action whose effect the scan established by itself (an OpenAPI method, an MCP annotation) was never asked and is not in `total`, and an inventory or `agent_bindings` declaration has no per-action counterfactual to score against. `answered` is exact rather than optimistic: it counts dimensions that gap when the same action is re-resolved *without* its declaration. `total == answered + open`, and `open_by_dimension` sums to `open`. `open_questions[]` is the answer order and joins to `evidence_gaps[].subject_id`. v0.38 ranks it by the ceiling of what an answer can establish: the actions nothing has bounded first — no effect evidence, a protocol default standing in for its absence, or only a heuristic reading the scan may not act on — then the actions a reviewed declaration or policy-eligible source evidence established, strongest-acting first, with `effect` before `authority` within one action. Position is not severity: the action at the top is the one *least* is known about. Nothing here gates; it is a projection of counts the decision already made, published so a coding agent (and the generated questionnaire) can report progress instead of a gap tally.
419
419
-`release_decision.evidence_coverage.policy_gap_count` and top-level `policy_evidence_gaps[]` (v0.33+) — policy applicability that is heuristic-only, mixed, unknown, or conflicting. These rows are outside Findings and cannot be suppressed, baselined, severity-overridden, acknowledged, or removed by `--no-heuristics`; any row prevents `passed`.
420
420
-`release_decision.evidence_coverage.identity_coverage` (v0.30+) — `{total_observations, canonical_tools, bound_tools, pass_eligible_tools, ambiguous_name_count, gap_count, reason_counts}`. Provider-scoped observations remain separate unless an exact reviewed `tool_identity.bindings[]` entry joins them. Any ambiguous selector, invalid binding, or conflicting identity prevents `passed`.
421
421
- `release_decision.evidence_coverage.evidence_gaps[]` (v0.26+; semantic kinds added v0.29) — one structured row per measurable gap: `{kind, subject, source_type, source_ref, why, next_action}`. In addition to `low_confidence_tool` and `source_warning`, v0.29 adds `incomplete_surface`, `missing_effect_evidence`, `inferred_effect_only`, `conflicting_effect_evidence`, `missing_authority_evidence`, `partial_authority_evidence`, `conflicting_authority_evidence`, and `invalid_semantic_annotation`. v0.36 adds `declaration_below_inferred_evidence`: the declared effect is weaker than evidence this scan inferred for the same action. The declaration still stands as the operative effect — heuristics never drive a verdict — but the action is not evidence-backed-pass until a reviewer raises the declared effect or adds `action_surface.actions[].override` with the `evidence` they checked and the `reason` it does not apply. An acknowledged override keeps the action pass-eligible and is reported as one semantic review concern, so the run can never read `passed`. Each acknowledgement is also emitted as a row in `release_decision.evidence_coverage.semantic_coverage.acknowledged_overrides[]` (v0.36+) naming the action, both readings, the hint source, any source evidence that agrees, and the reviewer's evidence and reason — the packet's §1 and the PR comment render it, because a count is not a review surface. The acknowledgement is consumed by policy applicability as well, so applying it reaches the review route rather than trading one gap for another. Semantic next actions use `declare_action_effect`, `declare_action_authority`, `declare_tool_inventory`, `provide_complete_inventory`, or `resolve_semantic_conflict`, include accepted values and exact source/manifest pointers, and are always human-routed. v0.37 adds `next_action.observed_readings[]` on effect rows — `{effect, sources[], observed}`, the distinct readings this scan's non-declaration evidence supports, so the row can be answered without opening `action_surface_facts`. Where those readings support one conservative answer, `next_action.declaration_template` carries it **pre-filled** instead of a `<REVIEW_REQUIRED>` blank. That value is a proposal, not an assertion: it is drawn from the closed `ActionEffect` vocabulary (never from source content), it is never weaker than any reading, and it is offered only where something was observed — a protocol default standing in for the absence of evidence, or a heuristic reading of `read`, keeps the blank. Their declaration placeholders carry `suggested_patch_kind="manual"`, `auto_apply=false`, and `requires_human_review=true`; they are not Patch objects, and a pre-filled template is still operative only once a human merges it into the manifest. v0.37 also re-routes `partial_authority_evidence`: it is raised when the *source's* authority evidence is ambiguous or incomplete, and the resolver preserves it whatever the manifest declares ("reviewed authority cannot replace ambiguous or incomplete source authority alternatives"). Its `next_action.kind` is therefore `provide_source` with no declaration template, rather than a `declare_action_authority` block that could not close the row it was printed on. It is excluded from `declaration_questions` for the same reason. Work the rows in order instead of guessing; Agents Shipgate never auto-asserts effect or authority.
Copy file name to clipboardExpand all lines: docs/packet-schema.v0.15.json
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -547,7 +547,7 @@
547
547
},
548
548
"DeclarationQuestionRow": {
549
549
"additionalProperties": false,
550
-
"description": "v0.37: one open declaration question, in the order to answer it.\n\nOrdered strongest-acting action first. Answering the two that move money\nand communicate outward is what reached a verdict on the fourth\n``adk-samples#1745`` walk, so the list leads with them rather than with\nwhatever sorts first alphabetically. Ordering is ranking only \u2014 it decides\nwhat to read first and can never change a verdict.\n\nv0.38: a question is one blank a reviewer fills, so ``answer_path`` \u2014 the\nmanifest block that blank lives in \u2014 is its identity. Actions answered by\nthe same block are one question: a source of 117 actions with no authority\nevidence owes one ``tool_sources[].authority`` block, and counting that as\n117 questions describes one edit as a backlog (#410 increment 3).",
550
+
"description": "v0.37: one open declaration question, in the order to answer it.\n\nOrdered by how much answering can move the verdict, rather than by what\nsorts first alphabetically. Answering the two actions that moved money and\ncommunicated outward is what reached a verdict on the fourth\n``adk-samples#1745`` walk. Ordering is ranking only \u2014 it decides what to\nread first and can never change a verdict.\n\nv0.38: a question is one blank a reviewer fills, so ``answer_path`` \u2014 the\nmanifest block that blank lives in \u2014 is its identity. Actions answered by\nthe same block are one question: a source of 117 actions with no authority\nevidence owes one ``tool_sources[].authority`` block, and counting that as\n117 questions describes one edit as a backlog (#410 increment 3).\n\nv0.38 also ranks by the **ceiling** of what an answer can establish rather\nthan by the effect already inferred. The actions nothing has bounded lead \u2014\nno effect evidence at all, a protocol default standing in for its absence,\nor only a heuristic reading this scan may not act on \u2014 because an unbounded\naction is unmeasured rather than safe, and its answer can still turn out to\nbe anything. Actions whose effect a reviewed declaration or policy-eligible\nsource evidence established keep their evidence rank behind them, strongest\nfirst. Position is not severity: the action at the top is the one *least*\nis known about.",
0 commit comments