Repository navigation
Commit 0c5f40f
authored
* fix(evidence): a declaration weaker than its evidence is never silent (#409)
Increment 1 of the evidence-first declaration RFC (#410): the monotone
declaration rule.
Declaring `effect: read` on a tool this scanner itself tagged
`external_write` was accepted with zero findings. The pre-existing
`inferred_effect_only` gap was closed by the very declaration that
contradicted the heuristic which raised it, the action went pass-eligible
(0/12 -> 1/12), and the contradicting `risk_tags` stayed in the same report
with nothing joining them.
The contradiction check already existed and was correct for the claims it
could see: `_assess_effect` admits a claim into `contradictory` only when
`policy_eligible`, which `domain.py` grants only to typed, high-confidence
bases. Heuristic risk hints are deliberately excluded -- a heuristic must
never *drive* policy (#357). But one flag governed two different powers.
Driving a verdict heuristics rightly cannot. Challenging a human assertion
they should: a declaration sitting *below* an observation is not the
heuristic gating anything, it is a human statement contradicting something
the scan saw.
Effect declarations are now monotone. Escalation stays silent.
De-escalating past a non-policy-eligible inference raises
`declaration_below_inferred_evidence` (report schema 0.35 -> 0.36), a
review-level evidence gap naming the declared value, the inferred value,
and the hint that produced it. The declaration stays operative and the row
never blocks, but the action is not evidence-backed-pass until it is
answered -- by raising `effect`, or by the new
`action_surface.actions[].override` block naming the `evidence` checked and
the `reason` it does not apply. An acknowledged override is accepted (the
action is pass-eligible again) and reported as one semantic review concern,
so a run carrying one can never read `passed`.
Two guards keep the rule from firing where nothing is being asserted. A
declaration the source itself corroborates is not challenged
(`support.search_kb` declares `read` beside `readOnlyHint: true`, and a
keyword reading `financial_write` out of "refund" in its description is the
weaker signal). Corroboration never counts the manifest row's own
restatements of itself, so a declaration cannot close its own gap. And an
override can never silence `conflicting_effect_evidence`: where
policy-eligible evidence outranks the declaration, the blocking conflict is
unchanged and no acknowledgement attaches.
Closes #409.
Co-Authored-By: Claude Opus 5 <[email protected]>
* review: corroborating source evidence names the row, it does not exempt it
Self-review round 1 on #411.
The first draft exempted a declaration the source itself corroborates:
`support.search_kb` declares `read` and carries `readOnlyHint: true`, so why
make a reviewer defend a protocol annotation against a keyword? Because this
resolver already refuses to pass on that annotation alone -- with no
declaration the same tool is `inferred_effect_only` and not pass-eligible,
precisely because a hint outranks it. A declaration that merely restates the
annotation must not buy what the annotation could not, or #409's hole moves
rather than closes. Worse, the corroboration was drawn from content the tool
source supplies about itself, which is not conditioned on
`tool_sources[].trust`: an MCP server could assert `readOnlyHint: true` about a
destructive tool and a `read` declaration would sail through with zero rows.
Corroboration is now named in the row instead -- "source evidence agrees with
the declaration (mcp_annotation)" -- which is what makes the override one line
to write rather than an investigation to open. Measured cost of dropping the
exemption across the whole sample and fixture corpus: two rows, both true, both
now carried as overrides in `samples/support_refund_agent` and
`samples/ai_generated_refund_pr`, so the shipped samples are the worked
example.
Also from the review:
- An `override` written where the conflict is with policy-eligible evidence was
silently discarded. That is the field a blocked reviewer reaches for first,
and they got a byte-identical message on re-run. `conflicting_effect_evidence`
now says the override does not reach it and why.
- `_decision_reason` composed its sentence from three known `reason_counts`
keys, so a future review-concern reason would vanish from the sentence
whenever an authority or override phrase was present. The unnamed remainder
is now counted explicitly.
- `SemanticCoverageDecision`'s docstring -- published verbatim as the
`description` in the generated report schema -- and the agent contract still
described `review_concern_count` as authority-only.
- The comparator block now lives entirely inside the guard that gives its
locals meaning, dropping a type-checker-only conjunct that read as a real
possibility.
- The #409 rule, the trust boundary, the ignored override, and the deliberately
accepted stale override are each pinned by a test.
The `test_weaker_write_declaration_with_inferred_financial_effect_is_ie` canary
now asserts the stronger property: `create_refund` declares `write` against a
`financial_write` heuristic, so the discarded escalation is on the record
instead of being carried only by the policy-evidence gap.
Co-Authored-By: Claude Opus 5 <[email protected]>
* review: both conflicting branches name an ignored override
Self-review round 2 on #411.
The read-versus-side-effect conflict is the same user error as the ranked one —
a reviewer reaches for `override`, the resolver discards it — so both branches
now route through one message builder instead of one of them going quiet.
Reworded so the sentence is accurate for both: the conflict is *in*
policy-eligible source evidence, which is true whether it outranks the
declaration or splits read from side-effect.
`docs/passed-verdict-contract.md` enumerates what keeps human review mandatory
and did not mention that an acknowledged override joins `unscoped`/`ambient`
authority on that list.
Swept every sample and fixture manifest for rows the stricter rule newly
raises: zero, after the two overrides added in the previous commit.
Co-Authored-By: Claude Opus 5 <[email protected]>
* review: the instruction names the effect it asks for
Self-review round 3 on #411.
The `Improve evidence:` line read "Raise action_surface.actions[].effect to the
inferred effect this row names" -- and named it nowhere the user could see.
Every human surface renders the short per-kind headline
(`evidence_gap_headline`), not the row's `why`, so the one artifact carrying
`external_communication` was report.json. A published next step that cannot be
executed from what it publishes is the #362 failure mode.
`expects` now names the value: "Raise action_surface.actions[].effect to
'external_communication', or acknowledge the difference with an override ...".
That also settles what `accepted_values` is the vocabulary for -- the field
`expects` names -- which was the other half of the complaint.
Deriving the value twice is this codebase's recurring defect class, so the
comparison lives in one function. `claims_above_declared_effect` is called by
the resolver (which needs the claims, to name their sources) and by
`strongest_effect_above_declaration`, which the release-decision projection
calls for the value alone. A test asserts the instruction and the reason name
the same effect. `action_surface_declaration` was spelled eight times across
two modules and is now load-bearing for the projection, so it is
`DECLARED_EFFECT_SOURCE`.
Carrying the value on `SemanticIssue` instead would have added a published
field to every semantic issue in the report schema; the CLAUDE.md
surface-discipline gate says to default against that when one shared function
does the job.
Co-Authored-By: Claude Opus 5 <[email protected]>
* review: consume the acknowledgement, and project each exception
Addresses the four findings on #411.
[P1] Consume the acknowledgement in policy applicability. Policy applicability
asks exactly what the override answers -- "does the higher heuristic effect
apply here?" -- so leaving the acknowledged claim unresolved there traded
`declaration_below_inferred_evidence` for `mixed_policy_evidence`: the reviewer
followed the row's own instruction and landed on a differently-named
`insufficient_evidence`. The override claim now carries `overridden_claim_ids`,
and `_non_authoritative_effect_escalation_support`, the action-policy
predicates, and capability-policy matching all read that one authored list
rather than re-deriving the comparison. The acknowledged fixture reaches
`review_required` with zero policy gaps, asserted end to end.
[P1] Preserve each override in the reviewer projection. A count is not a review
surface. `semantic_coverage.acknowledged_overrides[]` names the action, both
readings, the hint source, any source evidence that agrees, and the human's
evidence and reason. The packet's section 1 and the PR comment
(`SHIP-ACTION-EFFECT-OVERRIDE-ACKNOWLEDGED`) render one row per override.
Packet schema 0.12 -> 0.13 and verifier 0.9 -> 0.10, both with forward-reading
paths for the frozen shapes: the field is absent there and an empty list is the
honest reading.
[P2] Reject visually blank override evidence. `strip()` leaves U+200B and
U+2060 intact, so an override that renders as nothing to the reviewer it exists
for validated, suppressed the mismatch, and restored pass-eligibility. Both
fields now require visible content, using the repository's own
`has_visible_content` semantics -- moved to `schemas/text.py` so the schema
layer can use it without importing `core`, with `core.evidence_actions`
re-exporting it.
[P2] Keep the public manifest schema aligned with runtime validation.
`docs/manifest-v0.1.json` is advertised for live editor validation and accepted
both an `override` with no `effect` and blank `evidence`/`reason`. The
dependency is published as an `if`/`then`, the visible-content rule as a
`pattern` generated from the same code-point table the runtime reads (BMP as
`\uXXXX` and astral as literals -- `\u{...}` is ECMA-262-with-u-flag only and
Python's `re` rejects it outright), and `tests/test_manifest_schema_parity.py`
runs twelve payloads through both validators and requires them to agree.
`samples/support_refund_agent` and `samples/ai_generated_refund_pr` already
carried the two overrides this rule asks for, so the packet golden now shows
the rendered rows.
Co-Authored-By: Claude Opus 5 <[email protected]>
* test: pin the property that makes consuming acknowledgements safe
Policy applicability, action policies, and capability policies all drop the
claims an override names. That is only safe because an override can never name
a policy-eligible claim -- the resolver refuses to attach one while
policy-eligible evidence outranks the declaration, and the recorded set is
filtered on `not policy_eligible`. Asserted once at the producer rather than
three times at the consumers.
1 parent 109c586 commit 0c5f40f
85 files changed
Lines changed: 15693 additions & 403 deletions
File tree
- .well-known
- adoption-kits/claude-code-skill
- benchmark/safety-qualification
- docs
- plugins/claude-code/skills/agents-shipgate
- samples
- ai_generated_refund_pr
- conductor_agent/expected
- simple_crewai_agent/expected
- simple_langchain_agent/expected
- simple_openai_api_agent/expected
- support_refund_agent
- expected
- skills/agents-shipgate
- src/agents_shipgate
- ci
- cli/scan
- core
- lenses
- packet
- report
- schemas
- manifest
- tests
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
188 | 188 | | |
189 | 189 | | |
190 | 190 | | |
191 | | - | |
192 | | - | |
193 | | - | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
| |||
480 | 480 | | |
481 | 481 | | |
482 | 482 | | |
483 | | - | |
| 483 | + | |
484 | 484 | | |
485 | 485 | | |
486 | 486 | | |
487 | | - | |
| 487 | + | |
488 | 488 | | |
489 | 489 | | |
490 | 490 | | |
491 | 491 | | |
492 | 492 | | |
493 | 493 | | |
494 | 494 | | |
495 | | - | |
| 495 | + | |
496 | 496 | | |
497 | 497 | | |
498 | 498 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
532 | 532 | | |
533 | 533 | | |
534 | 534 | | |
535 | | - | |
| 535 | + | |
536 | 536 | | |
537 | 537 | | |
538 | 538 | | |
| |||
612 | 612 | | |
613 | 613 | | |
614 | 614 | | |
615 | | - | |
| 615 | + | |
616 | 616 | | |
617 | 617 | | |
618 | 618 | | |
| |||
650 | 650 | | |
651 | 651 | | |
652 | 652 | | |
653 | | - | |
| 653 | + | |
654 | 654 | | |
655 | 655 | | |
656 | 656 | | |
| |||
688 | 688 | | |
689 | 689 | | |
690 | 690 | | |
691 | | - | |
| 691 | + | |
692 | 692 | | |
693 | 693 | | |
694 | 694 | | |
| |||
746 | 746 | | |
747 | 747 | | |
748 | 748 | | |
749 | | - | |
| 749 | + | |
750 | 750 | | |
751 | 751 | | |
752 | 752 | | |
| |||
758 | 758 | | |
759 | 759 | | |
760 | 760 | | |
761 | | - | |
| 761 | + | |
762 | 762 | | |
763 | 763 | | |
764 | 764 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
5 | 116 | | |
6 | 117 | | |
7 | 118 | | |
| |||
0 commit comments