Skip to content

Commit 94743b5

Browse files
fix(evidence): address the PR review — orthogonal effects, frozen schemas, and reviewer-safe prose (#409)
Eight review comments. Five affected gate correctness or wire compatibility. **Effects are not totally ordered by obligation.** Declaring `financial_write` over an inferred `external_communication` outranks it on risk, but requires no confirmation — which is exactly what communicating outward requires. Reading rank alone made that action pass-eligible with no gap and no external-communication finding while the external-write risk tag sat untouched in the same report. `declaration_covers` now requires the declaration to rank at or above the observation under **both** published rank orders *and* to oblige at least that observation's built-in controls. The obligations move into `BUILTIN_EFFECT_OBLIGATIONS`, pinned against the branches they mirror by a test that walks each entry through a real scan. Coverage also reads the whole reviewed surface rather than the `effect` field alone: a `risk_tags: [financial_action]` entry produces a policy-eligible `financial_write` claim and applies the financial-write controls, so a heuristic saying the same thing is already accounted for. The shipped `ai_generated_refund_pr` fixture declares exactly that pair and found this. **Four published schemas were mutated in place.** The new enum value propagated into packet v0.12, verifier v0.9, capability-lock v0.6, and capability-lock-diff v0.7, all keeping their version identifiers — so a consumer pinned to any of them rejected artifacts that document was supposed to describe. `generate_schemas.py --check` proves committed == generated and cannot catch it. Those four are restored byte-for-byte and the versions advance to 0.13, 0.10, 0.7, and 0.8. Prior artifacts still read: packet gains a v0.12 upgrade arm, the verifier's legacy reader gains 0.9, and a v0.6 lock is advanced on load instead of being rejected, which the bump would otherwise have done to every committed `capabilities.lock.json`. A parametrised test now asserts the frozen documents match `origin/main` byte-for-byte and that emitted artifacts validate against their own current schema. **The two comparators disagreed.** `EFFECT_EVIDENCE_RANK` orders `write` below `privileged_data_access` and `ACTION_EFFECT_RANK` the other way, so a declared `privileged_data_access` read as covered here while the policy path raised `mixed_policy_evidence` — a verdict no override could reach, contradicting the documented promise. Both consumers now call `declaration_covers`, which requires both orders to agree; nothing that gated before stops gating. **Two documented policy paths did not exist.** `ActionFact` has no `override` projection, so `require: {override.reason: ...}` resolved to `unknown_path` and produced a blocking false positive. Both rows are removed and the docs say where a recorded override *is* readable. **A visually blank reason satisfied the reviewed-reason requirement.** `str.strip()` accepts U+200B. The reason now requires visible content and rejects invisible and direction-altering code points; `has_visible_content` moved to `schemas/_text.py` so a manifest validator can reach it without `schemas` importing `core`. Reviewer surface: the reason is folded through `one_line` before truncation, so a `\n- Agent may merge: true` cannot render as a separate authority-looking line; and each observation renders beside the producers that actually made it rather than the strongest effect beside a union of every source. Also: `benchmark/safety-qualification/README.md` required report schema 0.35 while the runner now requires 0.36, so following the checked-in procedure produced receipts the runner rejects. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent f23abaf commit 94743b5

63 files changed

Lines changed: 9348 additions & 259 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.well-known/agents-shipgate.json‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -189,8 +189,8 @@
189189
"agent_boundary_result_schema_version": "shipgate.agent_boundary_result/v2",
190190
"agent_boundary_result_schema_path": "docs/agent-boundary-result-schema.v2.json",
191191
"report_schema_version": "0.36",
192-
"packet_schema_version": "0.12",
193-
"verifier_schema_version": "0.9",
192+
"packet_schema_version": "0.13",
193+
"verifier_schema_version": "0.10",
194194
"verify_run_schema_version": "shipgate.verify_run/v4",
195195
"verification_plan_schema_version": "shipgate.verification_plan/v1",
196196
"verification_unit_result_schema_version": "shipgate.verification_unit_result/v1",
@@ -288,8 +288,8 @@
288288
"scenario": "scenario.json",
289289
"governance_benchmark_result": "benchmark/agent-pr-governance/results/result.json"
290290
},
291-
"capability_lock_schema_version": "0.6",
292-
"capability_lock_diff_schema_version": "0.7",
291+
"capability_lock_schema_version": "0.7",
292+
"capability_lock_diff_schema_version": "0.8",
293293
"preflight_schema_version": "0.4",
294294
"attestation_schema_version": "0.5",
295295
"registry_schema_version": "0.4",
@@ -484,18 +484,18 @@
484484
"agent_result": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/agent-result-schema.v3.json",
485485
"agent_boundary_result": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/agent-boundary-result-schema.v2.json",
486486
"codex_boundary_result": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/codex-boundary-result-schema.v2.json",
487-
"verifier": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/verifier-schema.v0.9.json",
487+
"verifier": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/verifier-schema.v0.10.json",
488488
"verify_run": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/verify-run-schema.v4.json",
489489
"verification_plan": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/verification-plan-schema.v1.json",
490490
"verification_unit_result": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/verification-unit-result-schema.v1.json",
491491
"verification_artifact_manifest": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/verification-artifact-manifest-schema.v1.json",
492492
"verification_receipt": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/verification-receipt-schema.v1.json",
493493
"human_authorization": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/human-authorization-schema.v1.json",
494494
"agent_handoff": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/agent-handoff-schema.v7.json",
495-
"packet": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/packet-schema.v0.12.json",
495+
"packet": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/packet-schema.v0.13.json",
496496
"preflight": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/preflight-schema.v0.4.json",
497-
"capability_lock": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/capability-lock-schema.v0.6.json",
498-
"capability_lock_diff": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/capability-lock-diff-schema.v0.7.json",
497+
"capability_lock": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/capability-lock-schema.v0.7.json",
498+
"capability_lock_diff": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/capability-lock-diff-schema.v0.8.json",
499499
"governance_benchmark_catalog": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/governance-benchmark-catalog-schema.v0.2.json",
500500
"governance_benchmark_result": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/governance-benchmark-result-schema.v0.2.json",
501501
"feedback": "https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/docs/feedback-schema.v0.1.json",

‎AGENTS.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -688,7 +688,7 @@ For the short, current statement of "which fields to read", see [`docs/agent-con
688688
| Report schema (v0.8 frozen reference) | [`docs/report-schema.v0.8.json`](docs/report-schema.v0.8.json) | `0.8` |
689689
| Report schema (v0.7 frozen reference) | [`docs/report-schema.v0.7.json`](docs/report-schema.v0.7.json) | `0.7` |
690690
| Report schema (v0.6 frozen reference) | [`docs/report-schema.v0.6.json`](docs/report-schema.v0.6.json) | `0.6` |
691-
| Packet schema (Release Evidence Packet, latest) | [`docs/packet-schema.v0.12.json`](docs/packet-schema.v0.12.json) | `0.12` |
691+
| Packet schema (Release Evidence Packet, latest) | [`docs/packet-schema.v0.13.json`](docs/packet-schema.v0.13.json) | `0.13` |
692692
| Agent result schema (current) | [`docs/agent-result-schema.v2.json`](docs/agent-result-schema.v2.json) | `agent_result_v2` |
693693
| Verifier schema (current) | [`docs/verifier-schema.v0.5.json`](docs/verifier-schema.v0.5.json) | `0.5` |
694694
| Agent handoff schema (current) | [`docs/agent-handoff-schema.v5.json`](docs/agent-handoff-schema.v5.json) | `shipgate.agent_handoff/v5` |
@@ -697,8 +697,8 @@ For the short, current statement of "which fields to read", see [`docs/agent-con
697697
| Host-grants baseline schema | [`docs/host-grants-baseline-schema.v0.2.json`](docs/host-grants-baseline-schema.v0.2.json) | `0.2` |
698698
| Host-grants drift schema | [`docs/host-grants-drift-schema.v0.2.json`](docs/host-grants-drift-schema.v0.2.json) | `0.2` |
699699
| Capability standard | [`docs/capability-standard.md`](docs/capability-standard.md) | `0.5` |
700-
| Capability lock schema | [`docs/capability-lock-schema.v0.6.json`](docs/capability-lock-schema.v0.6.json) | `0.6` |
701-
| Capability lock diff schema | [`docs/capability-lock-diff-schema.v0.7.json`](docs/capability-lock-diff-schema.v0.7.json) | `0.7` |
700+
| Capability lock schema | [`docs/capability-lock-schema.v0.7.json`](docs/capability-lock-schema.v0.7.json) | `0.7` |
701+
| Capability lock diff schema | [`docs/capability-lock-diff-schema.v0.8.json`](docs/capability-lock-diff-schema.v0.8.json) | `0.8` |
702702
| Governance benchmark catalog schema | [`docs/governance-benchmark-catalog-schema.v0.2.json`](docs/governance-benchmark-catalog-schema.v0.2.json) | `0.2` |
703703
| Governance benchmark result schema | [`docs/governance-benchmark-result-schema.v0.2.json`](docs/governance-benchmark-result-schema.v0.2.json) | `0.2` |
704704
| Check catalog | [`docs/checks.json`](docs/checks.json) | regenerated each release |
@@ -758,7 +758,7 @@ NOT prove. Use `--no-packet` / `--packet-format` on `scan`, and
758758
`agents-shipgate evidence-packet --from <packet.json|report.json>` to
759759
re-render. The full packet contract (fixed sections, disclaimers,
760760
`evidence_matrix` rules) lives in
761-
[STABILITY.md §Release Evidence Packet](STABILITY.md#release-evidence-packet-v012)
761+
[STABILITY.md §Release Evidence Packet](STABILITY.md#release-evidence-packet-v013)
762762
and [`docs/agent-contract-current.md`](docs/agent-contract-current.md#read-these-for-release-review).
763763

764764
Exit codes (stable):

‎CHANGELOG.md‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,24 @@
3737
evidence, or evidence that has since disappeared, re-opens the question
3838
instead of passing under a year-old answer.
3939

40+
Coverage is **category-aware**, not a total order. `financial_write` outranks
41+
`external_communication` while requiring approval, audit, and idempotency but
42+
not confirmation — so declaring the higher-risk effect does not discharge the
43+
other category's obligation, and an observation the declaration does not cover
44+
is reported whichever way it ranks. Coverage reads the whole reviewed surface:
45+
a `risk_tags: [financial_action]` entry accounts for an inferred
46+
`financial_write` exactly as the built-in controls already treat it.
47+
4048
One comparator (`core/effect_override.py`) serves the resolver, the gap
41-
scaffold, and the finding, so the row a coding agent works and the row a
42-
reviewer reads cannot disagree about whether a declaration is below its
43-
evidence, or by how much.
49+
scaffold, the finding, the PR-comment rows, and the
50+
`builtin-effect-control-applicability` policy path, so the row a coding agent
51+
works and the row a reviewer reads cannot disagree about whether a declaration
52+
accounts for an observation.
53+
54+
Schema versions move together because they all project the evidence-gap union:
55+
report `0.35 → 0.36`, packet `0.12 → 0.13`, verifier `0.9 → 0.10`,
56+
capability lock `0.6 → 0.7`, capability-lock diff `0.7 → 0.8`. Every prior
57+
document keeps its bytes, and artifacts declaring a prior version still read.
4458

4559
- **Every evidence gap now labels a tool the way a reader can use, in every gap
4660
kind.** `EvidenceGap.subject` is a display label — identity lives in

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -669,7 +669,7 @@ artifacts — in read order:
669669
- **`agents-shipgate-reports/capabilities.lock.json`** + **`agents-shipgate-reports/base.capabilities.lock.json`** + **`agents-shipgate-reports/capability-lock-diff.{json,md}`** — the **capability review primitive**. Verify always emits the head lock after a successful scan; it emits the base lock and diff when the base scan can be materialized, falling back to the reviewed committed lock at `.agents-shipgate/capabilities.lock.json` if needed.
670670
- **Gate source of truth** — `report.json.release_decision.decision` (`passed | review_required | insufficient_evidence | blocked`). `merge_verdict` is a deterministic projection of it; the report stays the one decision engine.
671671
- **Tool-Use Readiness Report** (supporting) — `agents-shipgate-reports/report.{md,json,sarif}`. Markdown for human release review, JSON for tools and coding agents, SARIF for GitHub code-scanning workflows. This is the underlying check domain the verdict summarizes.
672-
- **Release Evidence Packet** (supporting) — `agents-shipgate-reports/packet.{md,json,html}` (and `packet.pdf` with the `[pdf]` extras). Reviewer-shaped synthesis with fixed sections, including binding and semantic evidence coverage, the compact evidence matrix, and tool/action diffs when available. Packet outputs are locally redacted by default; see [STABILITY.md §Release Evidence Packet](STABILITY.md#release-evidence-packet-v012).
672+
- **Release Evidence Packet** (supporting) — `agents-shipgate-reports/packet.{md,json,html}` (and `packet.pdf` with the `[pdf]` extras). Reviewer-shaped synthesis with fixed sections, including binding and semantic evidence coverage, the compact evidence matrix, and tool/action diffs when available. Packet outputs are locally redacted by default; see [STABILITY.md §Release Evidence Packet](STABILITY.md#release-evidence-packet-v013).
673673

674674
## Exit codes
675675

‎STABILITY.md‎

Lines changed: 41 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,16 @@ for reproducible CI.
1717

1818
## Migration Note: unreleased — a declaration below the evidence is recorded
1919

20-
`report_schema_version` moves `0.35` → `0.36`; `docs/report-schema.v0.35.json`
21-
is frozen. `contract_version` and `minimum_control_contract_version` are
22-
unchanged, and the manifest stays `version: "0.1"`. Both changes are additive:
23-
one new value in a frozen union, one new optional manifest field.
20+
Five published schemas move, because five of them project the evidence-gap
21+
union: `report_schema_version` `0.35` → `0.36`, `packet_schema_version` `0.12`
22+
→ `0.13`, `verifier_schema_version` `0.9` → `0.10`, the capability lock `0.6` →
23+
`0.7`, and the capability-lock diff `0.7` → `0.8`. Every prior document keeps
24+
its bytes: a consumer pinned to `packet-schema.v0.12.json` still validates every
25+
artifact that declares `0.12`, and artifacts declaring a prior version are read
26+
and advanced on load rather than rejected. `contract_version` and
27+
`minimum_control_contract_version` are unchanged, and the manifest stays
28+
`version: "0.1"`. Every change is additive: one new value in a frozen union, one
29+
new optional manifest field.
2430

2531
**New evidence-gap kind: `declaration_below_inferred_evidence.`** It is raised
2632
on an action whose declared `action_surface.actions[].effect` is weaker than an
@@ -51,6 +57,22 @@ finding. `evidence` must name exactly the inferred effects observed above the
5157
declared effect, so an override that has drifted from its evidence re-opens the
5258
question rather than passing under an old answer.
5359

60+
**Coverage is category-aware.** A declaration accounts for an inferred
61+
observation only when it ranks at or above it under both published effect-rank
62+
orders *and* obliges at least that observation's built-in controls.
63+
`financial_write` therefore does not discharge an inferred
64+
`external_communication`: it outranks it, but requires no confirmation, which is
65+
what communicating outward requires. Coverage reads every policy-eligible claim
66+
on the action, so a `risk_tags: [financial_action]` entry accounts for an
67+
inferred `financial_write` exactly as the built-in control evaluator already
68+
treats it.
69+
70+
**`override` is not a policy path.** It is a manifest input, absent from
71+
`action_surface_facts.actions[]`, so `action_surface.policies[].require` cannot
72+
read `override.reason` or `override.evidence` — writing one resolves to
73+
`unknown_path` and fails the rule. Read a recorded override from the
74+
`SHIP-ACTION-EFFECT-OVERRIDES-EVIDENCE` finding's `evidence`.
75+
5476
**What may change for an existing repository.** Any manifest that already
5577
declares an effect below an inferred observation gains one gap, one finding,
5678
and one non-pass-eligible action, which can move a `passed` or `review_required`
@@ -2013,13 +2035,15 @@ release decision. That action may be `detect`/`initialize` for
20132035
relevant unconfigured repos, or `verify` for configured repos. Use it as the
20142036
first touch on a repo or PR before committing to a full scan.
20152037

2016-
`verifier.json` is governed by [`docs/verifier-schema.v0.9.json`](docs/verifier-schema.v0.9.json).
2017-
Verifier v0.1 through v0.8 remain frozen references — a published schema
2018-
identifier never gains an emitted field, so `0.9` carries
2019-
`capability_review.policy_weakening_proven` and `0.8` keeps the bytes every
2020-
consumer pinned to it already validates against. Artifacts declaring `0.8`
2021-
and earlier still read: the field defaults to `false`, which is exactly what
2022-
"this artifact recorded no base-vs-head comparison" means. It remains an orchestration artifact: `release_decision.decision` in
2038+
`verifier.json` is governed by [`docs/verifier-schema.v0.10.json`](docs/verifier-schema.v0.10.json).
2039+
Verifier v0.1 through v0.9 remain frozen references — a published schema
2040+
identifier never gains an emitted value, so `0.10` carries
2041+
`declaration_below_inferred_evidence` in the projected evidence-gap union and
2042+
`0.9` keeps the bytes every consumer pinned to it already validates against,
2043+
just as `0.9` carried `capability_review.policy_weakening_proven` over a frozen
2044+
`0.8`. Artifacts declaring `0.9` and earlier still read and are advanced to the
2045+
current identifier on load: both deltas are additive, so nothing in an older
2046+
artifact is reinterpreted. It remains an orchestration artifact: `release_decision.decision` in
20232047
`report.json` is still the only release gate. Release and merge fields remain
20242048
mirrors or deterministic projections of report data; the v0.6 authorization
20252049
evaluation and the v0.7 `diff_status` block are operational overlays that
@@ -2213,12 +2237,14 @@ infer runtime routing, or execute tools. Action Surface Diff policy findings
22132237
can affect release gating through `findings[].blocks_release`; Tool Surface
22142238
Diff remains explanatory only.
22152239

2216-
### Release Evidence Packet (v0.12)
2240+
### Release Evidence Packet (v0.13)
22172241

22182242
`agents-shipgate-reports/packet.json` is a supporting/provisional reviewer
2219-
artifact governed by [`docs/packet-schema.v0.12.json`](docs/packet-schema.v0.12.json).
2220-
v0.12 adds request, subject, input-set, engine-requirement, and decision IDs
2221-
while preserving the report release decision as the only gate. v0.11 and
2243+
artifact governed by [`docs/packet-schema.v0.13.json`](docs/packet-schema.v0.13.json).
2244+
v0.13 adds `declaration_below_inferred_evidence` to the projected evidence-gap
2245+
union and changes nothing else. v0.12 added request, subject, input-set,
2246+
engine-requirement, and decision IDs
2247+
while preserving the report release decision as the only gate. v0.12 and
22222248
earlier packets validate against their matching frozen schemas. v0.11 added
22232249
typed policy support; v0.9 added provider-scoped tool identities; v0.8 added
22242250
report v0.29 semantic

‎adoption-kits/claude-code-skill/.agents-shipgate-kit-metadata.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,8 @@
4343
"ce417b1351675cf8c96d5846c7bc73842b481b07fbf1720f6352d431135f5a52",
4444
"cc468647434d12be45819de4db6ece8a860010b44c31ba3586038067eb3357af",
4545
"3e10877198eead17338d54810ef7dc32317fa1b1013b54acccc1c2e9a94f53e4",
46-
"86a0e4a2b3711d3684305f5d9b2e61a9968e44472dcc67c3e7d3b2a84b965568"
46+
"86a0e4a2b3711d3684305f5d9b2e61a9968e44472dcc67c3e7d3b2a84b965568",
47+
"00400f9515ceb32b3ef52e42af82f14404bf471941f0d628a9d9abf5514ec037"
4748
],
4849
"prompts/add-shipgate-to-repo.md": [
4950
"ea3c37cfbbd42c40d164abfe21d468a3a5550d5384125f94a53c947dea6b4b2a",

0 commit comments

Comments
 (0)