Skip to content

Commit 17753cd

Browse files
fix(evidence): one spelling for every gap that names a catalog tool
Review of the exclusion ledger found the failure it exists to prevent, reproduced one layer up. `partial_binding_evidence` and the binding graph-issue rows spelled their subject as the raw canonical tool id, while `_unreached_tool_gaps` and `_semantic_gap` rendered `name [provider]`. The ledger joined on the second spelling, missed, and recorded a tool the decision had gapped as `not_claimed` — `binding_coverage.gap_count: 1` beside `surface_exclusions.gated: 0`, and the whole suite stayed green. Every tool-scoped gap now renders its subject through the shared `catalog_subject`, and the conservation invariant gained the two claims that would have caught it: no joinable gap may name a catalog tool by raw id, and an excluded tool the decision gapped may never be recorded `not_claimed`. The spelling rule is scoped to the gap kinds the ledger actually joins, so it does not force unrelated surfaces to change for a join that does not exist. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 660f6fb commit 17753cd

5 files changed

Lines changed: 157 additions & 14 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,19 @@
3838

3939
A conservation invariant is enforced at emission: `observed == analysed ∪
4040
excluded`, every excluded subject appears in the ledger, every `evidence_gap`
41-
record is backed by a gap row carrying the same subject, and a subject *this
42-
change* newly excluded can never be `not_claimed`. The
41+
record is backed by a gap row carrying the same subject, an excluded tool the
42+
decision *did* gap is never recorded `not_claimed`, and a subject *this
43+
change* newly excluded can never be `not_claimed` either. The
4344
`unbound_tools: 1 / gap_count: 0` state is now unrepresentable rather than
4445
something each call site has to remember.
4546

47+
Every evidence gap that names a catalog tool now names it the same way.
48+
`partial_binding_evidence` and the binding graph-issue rows used to carry the
49+
raw canonical tool id (`tool_v2_2c9ee6…`) while every other emitter rendered
50+
`name [provider]`, which both read badly in `Improve evidence:` and made a
51+
tool unjoinable with itself — the ledger looked up one spelling and found the
52+
other. The invariant rejects a raw id reaching a joinable gap subject.
53+
4654
The gate itself moved only where a diff proves it should. `binding_surface_diff`
4755
gained `added_unbound_tool_ids` — head exclusions minus base exclusions — and
4856
a tool in that set raises a `missing_binding_evidence` gap naming it. A

‎src/agents_shipgate/ci/release_decision.py‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -643,6 +643,21 @@ def _binding_coverage(
643643
graph = report.binding_surface_facts
644644
reason_counts: dict[str, int] = {}
645645
gaps: list[EvidenceGap] = []
646+
# Every gap that names a catalog tool names it the same way. Two of the
647+
# emitters below used to spell the subject as the raw canonical id while
648+
# the others rendered ``name [provider]``, which made the same tool
649+
# unjoinable with itself: the exclusion ledger looked up one spelling,
650+
# found the other, and recorded a gated exclusion as ``not_claimed``.
651+
# ``_gap_subject`` is the single spelling, and
652+
# ``validate_semantic_consistency`` rejects a raw id reaching this field.
653+
catalog_rows = {
654+
str(row["tool_id"]): row for row in report.tool_catalog if row.get("tool_id")
655+
}
656+
657+
def _gap_subject(tool_id: str | None, fallback: str) -> str:
658+
if not tool_id:
659+
return fallback
660+
return catalog_subject(catalog_rows.get(tool_id) or {"tool_id": tool_id})
646661
for issue in graph.issues:
647662
_increment(reason_counts, issue.kind)
648663
if issue.kind == "ambiguous_root_agent":
@@ -679,7 +694,10 @@ def _binding_coverage(
679694
gaps.append(
680695
EvidenceGap(
681696
kind=issue.kind,
682-
subject=issue.tool_id or issue.agent_id or graph.root_agent_id or "agent_binding_graph",
697+
subject=_gap_subject(
698+
issue.tool_id,
699+
issue.agent_id or graph.root_agent_id or "agent_binding_graph",
700+
),
683701
source_ref=issue.source_pointer or issue.source,
684702
why=issue.message,
685703
next_action=EvidenceGapAction(
@@ -703,7 +721,7 @@ def _binding_coverage(
703721
gaps.append(
704722
EvidenceGap(
705723
kind="partial_binding_evidence",
706-
subject=tool_id,
724+
subject=_gap_subject(tool_id, tool_id),
707725
why="A possibly reachable tool lacks a complete static binding edge.",
708726
next_action=EvidenceGapAction(
709727
kind="provide_complete_binding_graph",

‎src/agents_shipgate/core/semantic_consistency.py‎

Lines changed: 45 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,11 @@
22

33
from agents_shipgate.core.disclaimers import STATIC_VERDICT_DISCLAIMER
44
from agents_shipgate.core.domain import Tool, ToolSemanticAssessment
5-
from agents_shipgate.core.surface_exclusions import catalog_subject
5+
from agents_shipgate.core.surface_exclusions import (
6+
BINDING_GAP_KINDS,
7+
LEDGER_JOINED_GAP_KINDS,
8+
catalog_subject,
9+
)
610
from agents_shipgate.schemas.report import ReadinessReport
711
from agents_shipgate.schemas.semantic import ToolSemanticEvidence
812

@@ -171,7 +175,7 @@ def _validate_exclusion_ledger(report: ReadinessReport) -> None:
171175
the partition perfectly; what it violated is that a subject the diff
172176
removed from analysis reached no gap (#403).
173177
174-
Three claims, each one a way that state could come back:
178+
Five claims, each one a way that state could come back:
175179
176180
1. every excluded subject is in the ledger — a stage cannot narrow
177181
silently;
@@ -180,7 +184,12 @@ def _validate_exclusion_ledger(report: ReadinessReport) -> None:
180184
have;
181185
3. a subject *this change* newly excluded is always ``evidence_gap`` — the
182186
pre-existing/newly-arrived distinction is the whole basis on which a
183-
``not_claimed`` record is allowed at all.
187+
``not_claimed`` record is allowed at all;
188+
4. no joinable gap names a catalog tool by its raw canonical id, so the
189+
join in (2) is exact rather than approximately right;
190+
5. an excluded tool the decision *did* gap is never recorded
191+
``not_claimed`` — (2) with the sign flipped, which is the direction a
192+
second spelling breaks.
184193
"""
185194

186195
decision = report.release_decision
@@ -212,7 +221,8 @@ def _validate_exclusion_ledger(report: ReadinessReport) -> None:
212221
f"{sorted(expected_subjects - binding_subjects)}"
213222
)
214223

215-
gap_subjects = {gap.subject for gap in decision.evidence_coverage.evidence_gaps}
224+
gaps = decision.evidence_coverage.evidence_gaps
225+
gap_subjects = {gap.subject for gap in gaps}
216226
for entry in ledger.entries:
217227
if entry.accounting == "evidence_gap" and entry.subject not in gap_subjects:
218228
raise SemanticConsistencyError(
@@ -222,9 +232,38 @@ def _validate_exclusion_ledger(report: ReadinessReport) -> None:
222232
raise SemanticConsistencyError(
223233
f"exclusion {entry.subject!r} was introduced by this change and is not gated"
224234
)
225-
if ledger.gated and not decision.evidence_coverage.evidence_gaps:
235+
236+
# The join above only catches the ledger over-claiming. Under-claiming — a
237+
# gap exists and the ledger says `not_claimed` — is the same failure with
238+
# the sign flipped, and it is what a subject spelled two ways produces: a
239+
# `partial_binding_evidence` gap carrying the raw canonical id could not be
240+
# matched against a ledger row carrying `name [provider]`, so a gated tool
241+
# read as unclaimed and the whole suite stayed green.
242+
#
243+
# Checked at the source rather than by re-running the same join, which
244+
# would just repeat any shared mistake: a gap may never name a catalog tool
245+
# by its raw id. With one spelling enforced here, the join above is exact.
246+
raw_ids = set(by_id)
247+
for gap in gaps:
248+
if gap.kind in LEDGER_JOINED_GAP_KINDS and gap.subject in raw_ids:
249+
raise SemanticConsistencyError(
250+
f"evidence gap names tool {gap.subject!r} by raw id rather than "
251+
"its catalog subject; the exclusion ledger cannot join it"
252+
)
253+
gated_binding_subjects = {
254+
entry.subject
255+
for entry in ledger.entries
256+
if entry.stage == "binding" and entry.accounting == "evidence_gap"
257+
}
258+
unrecorded = {
259+
gap.subject
260+
for gap in gaps
261+
if gap.kind in BINDING_GAP_KINDS and gap.subject in expected_subjects
262+
} - gated_binding_subjects
263+
if not ledger.truncated and unrecorded:
226264
raise SemanticConsistencyError(
227-
"exclusion ledger reports gated exclusions with no evidence gaps"
265+
"excluded tools carry a binding evidence gap the ledger does not "
266+
f"account for: {sorted(unrecorded)}"
228267
)
229268

230269

‎src/agents_shipgate/core/surface_exclusions.py‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -54,19 +54,33 @@ def _catalog_by_id(report: ReadinessReport) -> dict[str, Mapping[str, Any]]:
5454
}
5555

5656

57-
def _gap_subjects(gaps: Sequence[EvidenceGap], kinds: set[str]) -> set[str]:
57+
def _gap_subjects(gaps: Sequence[EvidenceGap], kinds: frozenset[str]) -> set[str]:
5858
return {gap.subject for gap in gaps if gap.kind in kinds}
5959

6060

61-
_BINDING_GAP_KINDS = {
61+
#: Gap kinds a binding-stage exclusion can be accounted for by. Public so
62+
#: ``core.semantic_consistency`` checks the same set the ledger joins on — a
63+
#: private copy there would drift and quietly stop checking anything.
64+
BINDING_GAP_KINDS = frozenset({
6265
"missing_binding_evidence",
6366
"partial_binding_evidence",
6467
"unresolved_bound_tool",
6568
"unresolved_agent_binding",
6669
"conflicting_binding_evidence",
6770
"incomplete_handoff_graph",
6871
"invalid_tool_binding",
69-
}
72+
})
73+
74+
#: Every gap kind the ledger joins a subject against. The spelling rule
75+
#: ``core.semantic_consistency`` enforces is scoped to exactly these: a gap the
76+
#: ledger never reads may name its subject however it likes, and widening the
77+
#: rule past them would force unrelated surfaces to change for a join that does
78+
#: not exist. (``inferred_policy_applicability`` does name tools by raw
79+
#: canonical id, which reads badly in ``Improve evidence:`` — a separate
80+
#: actionability question, not a conservation one.)
81+
LEDGER_JOINED_GAP_KINDS = BINDING_GAP_KINDS | frozenset(
82+
{"incomplete_surface", "source_warning"}
83+
)
7084

7185

7286
def build_surface_exclusions(report: ReadinessReport) -> SurfaceExclusionLedger:
@@ -80,7 +94,7 @@ def build_surface_exclusions(report: ReadinessReport) -> SurfaceExclusionLedger:
8094
gaps: Sequence[EvidenceGap] = (
8195
decision.evidence_coverage.evidence_gaps if decision is not None else ()
8296
)
83-
binding_gap_subjects = _gap_subjects(gaps, _BINDING_GAP_KINDS)
97+
binding_gap_subjects = _gap_subjects(gaps, BINDING_GAP_KINDS)
8498

8599
entries: list[SurfaceExclusion] = []
86100
entries.extend(_binding_exclusions(report, binding_gap_subjects))
@@ -298,6 +312,8 @@ def build_detect_exclusions(result: DetectResult) -> SurfaceExclusionLedger:
298312

299313

300314
__all__ = [
315+
"BINDING_GAP_KINDS",
316+
"LEDGER_JOINED_GAP_KINDS",
301317
"build_detect_exclusions",
302318
"build_surface_exclusions",
303319
"catalog_subject",

‎tests/test_surface_exclusions.py‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -338,3 +338,65 @@ def _rehydrated_tools(report):
338338
if str(row.get("tool_id")) in reachable
339339
]
340340
return attach_semantic_assessments(tools, {}, copy_tools=False)
341+
342+
343+
def test_a_possibly_reachable_tool_is_recorded_as_gated():
344+
"""One spelling, or the ledger cannot join a tool with itself (review 1).
345+
346+
`partial_binding_evidence` used to name its subject by the raw canonical
347+
tool id while every other emitter rendered `name [provider]`. The ledger
348+
looked up one spelling, found the other, and wrote `not_claimed` for a tool
349+
the decision had gapped — `binding_coverage.gap_count: 1` beside
350+
`surface_exclusions.gated: 0`, which is the reported failure re-expressed
351+
one layer up.
352+
"""
353+
354+
from agents_shipgate.ci.release_decision import build_release_decision
355+
from agents_shipgate.core.surface_exclusions import build_surface_exclusions
356+
from agents_shipgate.schemas.bindings import AgentBindingGraphAssessment
357+
from agents_shipgate.schemas.report import (
358+
ReadinessReport,
359+
ReportSummary,
360+
ToolSurfaceSummary,
361+
)
362+
363+
report = ReadinessReport(
364+
run_id="run-1",
365+
project={},
366+
agent={},
367+
environment={},
368+
summary=ReportSummary(status="review_required"),
369+
tool_surface=ToolSurfaceSummary(total_tools=1, high_risk_tools=0),
370+
binding_surface_facts=AgentBindingGraphAssessment(
371+
root_agent_id="agent",
372+
status="partial",
373+
pass_eligible=False,
374+
possible_tool_ids=["tool_v1:abc"],
375+
),
376+
tool_catalog=[
377+
{
378+
"tool_id": "tool_v1:abc",
379+
"name": "charge_card",
380+
"provider": "billing",
381+
"source_type": "mcp",
382+
"source_ref": "mcp/tools.json",
383+
}
384+
],
385+
)
386+
report.release_decision = build_release_decision(
387+
report=report,
388+
tools=[],
389+
tool_catalog=[],
390+
ci_mode="advisory",
391+
fail_on=None,
392+
new_findings_only=False,
393+
)
394+
coverage = report.release_decision.evidence_coverage
395+
assert coverage.binding_coverage.gap_count == 1
396+
# No gap may name a catalog tool by its raw id — that is the spelling that
397+
# broke the join, and `validate_semantic_consistency` now refuses it.
398+
assert [gap.subject for gap in coverage.evidence_gaps] == ["charge_card [billing]"]
399+
400+
ledger = build_surface_exclusions(report)
401+
assert [entry.accounting for entry in ledger.entries] == ["evidence_gap"]
402+
assert ledger.gated == 1

0 commit comments

Comments
 (0)