Skip to content

Commit 8852e9d

Browse files
Address PR #90 P1 + P2 review findings
P1 — Coerced-truthy bypass closed (plugin_validation.py). The raw-metadata gate used ``value.get("dynamic_default") is True``, a strict identity check. Pydantic's bool coercion accepts ``"true"``, ``"True"``, ``"yes"``, ``"on"``, and ``1`` as ``True``, so a plugin declaring ``{"dynamic_default": "true", "floor_severity": "medium"}`` would pass the raw gate, get coerced to ``dynamic_default=True`` by ``_coerce_metadata``, and land as ``valid`` — reopening the silent bypass this PR closes. Fix: ``_raw_metadata_declares_dynamic_default`` now uses ``TypeAdapter(bool).validate_python(...)`` to mirror Pydantic's coercion exactly. Anything Pydantic stores as ``dynamic_default=True`` on the constructed ``CheckMetadata`` is now rejected at the gate. Values Pydantic refuses outright (arbitrary objects) fall through to ``_coerce_metadata`` as ``bad_metadata``. Tests: 12 new parametrized cases in test_plugin_validation.py covering 6 truthy raw values (``"true"``, ``"True"``, ``"TRUE"``, ``"yes"``, ``"on"``, ``1``) — each must land in ``dynamic_default_not_supported`` — and 7 falsey controls (``False``, ``"false"``, ``"False"``, ``"no"``, ``"off"``, ``0``, absent key) — each must NOT trip the gate. P2 — Aggregator-completeness test was vacuous (severity_override_floor.py). The original test only asserted that every ``dynamic_default=True`` catalog ID appeared as a key in ``_dynamic_check_defaults`` output. But the v0.18 seed loop (step 1) populates EVERY such key with the catalog static default unconditionally — so a future catalog entry that sets ``dynamic_default=True`` without adding the matching overlay branch in ``cli/scan.py:_dynamic_check_defaults`` would still pass the test. The resolver guard would not fire either (the key is present). This silently widens the tier-crossing bypass the PR exists to prevent. Fix: replace the weak test with a per-check overlay-verification registry ``_DYNAMIC_DEFAULT_OVERLAY_CASES`` mapping each ``dynamic_default=True`` catalog ID to ``(manifest_overlay, expected_dynamic_severity)`` where ``expected_dynamic`` is provably STRONGER than the catalog static. Two tests now pin the contract: - ``test_dynamic_default_overlay_registry_is_complete``: every catalog ``dynamic_default=True`` ID must have a registry entry. Fails the moment someone adds a new dynamic-default check without thinking through (and documenting) the manifest path that strengthens its default. - ``test_dynamic_default_aggregator_overlay_fires``: parametrized per registry entry. Builds the manifest with the strengthened section, calls ``_dynamic_check_defaults``, asserts the aggregator returns the manifest-effective severity (not the catalog static). Fails when an overlay branch is missing. Together the two new tests catch the regression class the seed loop alone cannot. P3 — Already addressed in prior follow-up commit 3fbb1e3 (docs/checks.md: "Plugin validation (v0.17+; six gates v0.18+)" now documents ``dynamic_default_not_supported`` between metadata and ID-collision gates). Verification: 1198 tests pass (1192 + 12 new + −6 net from P2 restructuring); ruff clean; ``python scripts/generate_schemas.py --check`` exits 0; coverage 87.23% (above 75% floor). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 3fbb1e3 commit 8852e9d

3 files changed

Lines changed: 229 additions & 38 deletions

File tree

‎src/agents_shipgate/checks/plugin_validation.py‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@
5151
from dataclasses import dataclass, field
5252
from typing import Any
5353

54-
from pydantic import ValidationError
54+
from pydantic import TypeAdapter, ValidationError
5555

5656
from agents_shipgate.core.context import ScanContext
5757
from agents_shipgate.core.models import CheckMetadata, Finding
@@ -363,25 +363,51 @@ def _is_floor_error(exc: BaseException) -> bool:
363363
return "floor_severity" in str(exc)
364364

365365

366+
_DYNAMIC_DEFAULT_BOOL = TypeAdapter(bool)
367+
368+
366369
def _raw_metadata_declares_dynamic_default(value: Any) -> bool:
367370
"""v0.18 (PR #1): inspect raw plugin metadata for the
368371
``dynamic_default`` flag BEFORE Pydantic validation runs.
369372
370373
Plugin metadata can arrive as either a dict (most plugins) or a
371374
pre-constructed ``CheckMetadata`` instance (test fixtures, advanced
372-
plugins). For dict form, read the field directly. For instance
375+
plugins). For dict form, read the field via Pydantic's own bool
376+
coercion so any value that ``CheckMetadata`` would store as
377+
``dynamic_default=True`` (``True``, ``1``, ``"true"``, ``"yes"``,
378+
``"on"``) lands here under ``dynamic_default_not_supported``
379+
instead of slipping past as ``valid`` after coercion. For instance
373380
form, the field is already typed — read it via attribute. Returns
374381
``False`` on any other shape (the downstream ``_coerce_metadata``
375-
will then surface a ``bad_metadata`` error for the shape itself).
382+
will then surface a ``bad_metadata`` error for the shape itself)
383+
and on values that Pydantic rejects outright (the downstream
384+
validator will produce the appropriate error).
376385
377386
Placement of the gate that calls this helper matters: it MUST run
378387
before ``_coerce_metadata``, because a plugin declaring
379388
``dynamic_default=True`` without ``floor_severity`` would otherwise
380389
trigger the new ``CheckMetadata`` model validator and be
381390
mis-classified as ``bad_floor``.
391+
392+
Parity invariant: this function returns ``True`` iff
393+
``CheckMetadata.model_validate(value).dynamic_default`` would be
394+
``True``. Mismatch would re-open the silent bypass — a regression
395+
test pins it.
382396
"""
383397
if isinstance(value, dict):
384-
return value.get("dynamic_default") is True
398+
if "dynamic_default" not in value:
399+
return False
400+
try:
401+
return _DYNAMIC_DEFAULT_BOOL.validate_python(
402+
value["dynamic_default"]
403+
)
404+
except ValidationError:
405+
# Value Pydantic refuses outright (e.g. an arbitrary
406+
# object). Leave it for _coerce_metadata to surface as
407+
# bad_metadata; we cannot have caused a silent bypass
408+
# because Pydantic's CheckMetadata.model_validate will
409+
# also reject it.
410+
return False
385411
if isinstance(value, CheckMetadata):
386412
return value.dynamic_default
387413
return False

‎tests/test_plugin_validation.py‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -410,6 +410,95 @@ def plugin(context):
410410
)
411411

412412

413+
@pytest.mark.parametrize(
414+
"raw_value",
415+
["true", "True", "TRUE", "yes", "on", 1],
416+
ids=["str_true", "str_True", "str_TRUE", "str_yes", "str_on", "int_1"],
417+
)
418+
def test_plugin_dynamic_default_coerced_truthy_is_rejected(
419+
monkeypatch, tmp_path, raw_value
420+
):
421+
"""v0.18 (PR #1) post-merge fix: the raw-metadata pre-check must
422+
catch values Pydantic would coerce to ``True``, not just the
423+
literal Python ``True``. Otherwise a plugin with
424+
``{"dynamic_default": "true", "floor_severity": "medium"}`` would
425+
pass the raw gate (``"true" is True`` → False), then Pydantic
426+
coerces it to ``True``, and the plugin lands as ``valid`` with
427+
``metadata.dynamic_default == True`` — reopening the silent
428+
bypass this PR closes.
429+
430+
Parity invariant: anything Pydantic stores as ``dynamic_default=True``
431+
on the constructed ``CheckMetadata`` MUST be rejected by the gate.
432+
"""
433+
434+
def plugin(context):
435+
return []
436+
437+
plugin.AGENTS_SHIPGATE_METADATA = {
438+
"id": "ACME-COERCED-TRUTHY",
439+
"category": "custom",
440+
"default_severity": "high",
441+
"description": "Plugin with truthy non-True dynamic_default.",
442+
"floor_severity": "medium",
443+
"dynamic_default": raw_value,
444+
}
445+
_patch_entries(monkeypatch, [_entry_point(lambda: plugin)])
446+
447+
report, _ = run_scan(
448+
config_path=CLEAN_FIXTURE,
449+
output_dir=tmp_path,
450+
formats=["json"],
451+
ci_mode="advisory",
452+
)
453+
record = report.loaded_plugins[0]
454+
assert record["validation_status"] == DYNAMIC_DEFAULT_NOT_SUPPORTED, (
455+
f"Coerced-truthy raw value {raw_value!r} bypassed the gate; "
456+
f"validation_status={record['validation_status']!r}. The raw "
457+
"gate must match Pydantic's bool coercion."
458+
)
459+
460+
461+
@pytest.mark.parametrize(
462+
"raw_value",
463+
[False, "false", "False", "no", "off", 0, None],
464+
ids=["bool_false", "str_false", "str_False", "str_no", "str_off", "int_0", "none"],
465+
)
466+
def test_plugin_dynamic_default_coerced_falsey_is_accepted(
467+
monkeypatch, tmp_path, raw_value
468+
):
469+
"""Parity invariant, falsey direction: values Pydantic would store
470+
as ``dynamic_default=False`` (or that omit the key entirely) must
471+
NOT be flagged as dynamic_default_not_supported. Pydantic accepts
472+
None as 'use default' for bool fields, so the gate must too.
473+
"""
474+
475+
def plugin(context):
476+
return []
477+
478+
metadata = {
479+
"id": "ACME-FALSEY-DEFAULT",
480+
"category": "custom",
481+
"default_severity": "medium",
482+
"description": "Plugin with falsey dynamic_default.",
483+
}
484+
if raw_value is not None:
485+
metadata["dynamic_default"] = raw_value
486+
plugin.AGENTS_SHIPGATE_METADATA = metadata
487+
_patch_entries(monkeypatch, [_entry_point(lambda: plugin)])
488+
489+
report, _ = run_scan(
490+
config_path=CLEAN_FIXTURE,
491+
output_dir=tmp_path,
492+
formats=["json"],
493+
ci_mode="advisory",
494+
)
495+
record = report.loaded_plugins[0]
496+
assert record["validation_status"] != DYNAMIC_DEFAULT_NOT_SUPPORTED, (
497+
f"Falsey raw value {raw_value!r} was wrongly flagged as "
498+
f"dynamic_default_not_supported. The gate is over-aggressive."
499+
)
500+
501+
413502
def test_plugin_with_dynamic_default_as_checkmetadata_instance_is_rejected(
414503
monkeypatch, tmp_path
415504
):

‎tests/test_severity_override_floor.py‎

Lines changed: 110 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -781,26 +781,66 @@ def test_dynamic_default_with_floor_is_accepted() -> None:
781781
assert meta.floor_severity == "medium"
782782

783783

784-
def test_dynamic_default_aggregator_completeness() -> None:
784+
def _base_manifest_dict() -> dict:
785+
return {
786+
"version": "0.1",
787+
"project": {"name": "p"},
788+
"agent": {"name": "a", "declared_purpose": ["t"]},
789+
"environment": {"target": "local"},
790+
"tool_sources": [{"id": "mcp", "type": "mcp", "path": "tools.json"}],
791+
}
792+
793+
794+
# v0.18 (PR #1) — per-check overlay-verification registry. Each entry maps a
795+
# ``dynamic_default=True`` catalog check ID to:
796+
# (manifest_overlay, expected_dynamic_severity)
797+
# where ``manifest_overlay`` is a partial-manifest dict that, when merged
798+
# into a minimal manifest, populates the section the aggregator MUST read
799+
# to produce a manifest-effective default *stronger* than the catalog
800+
# static. The expected severity must beat the catalog default for the
801+
# check so the overlay is observable in the aggregator output.
802+
#
803+
# When a new built-in dynamic-default check is added:
804+
# 1. CHECK_METADATA marks dynamic_default=True (forces floor_severity).
805+
# 2. cli/scan.py:_dynamic_check_defaults gets an overlay branch.
806+
# 3. _DYNAMIC_DEFAULT_OVERLAY_CASES gets an entry below.
807+
#
808+
# The completeness test below fails if (1) is done without (3), and the
809+
# per-case test fails if (2) is missing — together they pin the contract
810+
# the seed loop alone cannot enforce.
811+
_DYNAMIC_DEFAULT_OVERLAY_CASES: dict[str, tuple[dict, str]] = {
812+
"SHIP-ACTION-POLICY-VIOLATION": (
813+
{
814+
"action_surface": {
815+
"policies": [
816+
{
817+
"id": "p1",
818+
"severity": "critical",
819+
"match": {"effects": ["write"]},
820+
"require": {"approval.required": True},
821+
}
822+
]
823+
}
824+
},
825+
"critical",
826+
),
827+
}
828+
829+
830+
def test_dynamic_default_overlay_registry_is_complete() -> None:
785831
"""v0.18 (PR #1) contract: every catalog check carrying
786-
``dynamic_default=True`` MUST appear in the result of
787-
``cli/scan.py:_dynamic_check_defaults``. This is the structural
788-
safety net for the resolver's internal-consistency guard: the
789-
aggregator seeds every dynamic-default catalog check with its
790-
static default, so a manifest can override any dynamic-default
791-
check without the resolver raising ``RuntimeError`` for a
792-
legitimate user input.
793-
794-
Fails the moment someone adds ``dynamic_default=True`` to a
795-
catalog entry without ensuring the aggregator covers it. The seed
796-
loop in ``_dynamic_check_defaults`` is what guarantees coverage,
797-
so any new dynamic-default check is automatically included once
798-
its catalog entry validates.
832+
``dynamic_default=True`` MUST have an entry in
833+
``_DYNAMIC_DEFAULT_OVERLAY_CASES`` above. The entry documents
834+
which manifest section drives the dynamic default and pins the
835+
overlay-verification test below.
836+
837+
A missing entry means a contributor added a new dynamic-default
838+
check without thinking through the manifest path that
839+
strengthens its default — exactly the wiring gap M1 closed for
840+
SHIP-ACTION-POLICY-VIOLATION. Add an entry, then ensure the
841+
aggregator overlay test passes.
799842
"""
800843
from agents_shipgate.checks.registry import CHECK_METADATA
801-
from agents_shipgate.cli.scan import _dynamic_check_defaults
802-
from agents_shipgate.config.schema import AgentsShipgateManifest
803-
from agents_shipgate.inputs.policy_packs import LoadedPolicyPacks
804844

805845
dynamic_check_ids = {
806846
entry.id for entry in CHECK_METADATA if entry.dynamic_default
@@ -809,30 +849,66 @@ def test_dynamic_default_aggregator_completeness() -> None:
809849
"expected at least one dynamic_default=True check in catalog "
810850
"(SHIP-ACTION-POLICY-VIOLATION at minimum)"
811851
)
852+
missing = dynamic_check_ids - _DYNAMIC_DEFAULT_OVERLAY_CASES.keys()
853+
assert not missing, (
854+
f"dynamic_default=True check IDs without entries in "
855+
f"_DYNAMIC_DEFAULT_OVERLAY_CASES: {sorted(missing)}. "
856+
"Add an overlay case showing which manifest section drives the "
857+
"dynamic default for each new check, then ensure "
858+
"cli/scan.py:_dynamic_check_defaults has the matching overlay "
859+
"branch (the per-case test will fail until it does)."
860+
)
812861

813-
manifest = AgentsShipgateManifest.model_validate(
814-
{
815-
"version": "0.1",
816-
"project": {"name": "p"},
817-
"agent": {"name": "a", "declared_purpose": ["t"]},
818-
"environment": {"target": "local"},
819-
"tool_sources": [
820-
{"id": "mcp", "type": "mcp", "path": "tools.json"}
821-
],
822-
}
862+
863+
@pytest.mark.parametrize(
864+
"check_id",
865+
sorted(_DYNAMIC_DEFAULT_OVERLAY_CASES),
866+
ids=lambda cid: cid,
867+
)
868+
def test_dynamic_default_aggregator_overlay_fires(check_id: str) -> None:
869+
"""v0.18 (PR #1) contract: for each dynamic-default check, the
870+
aggregator MUST overlay the manifest-effective default onto the
871+
seeded catalog default. Verified by populating the relevant manifest
872+
section with a known stronger severity and asserting the aggregator
873+
returns that stronger value (not the catalog static).
874+
875+
This catches the regression class the seed loop alone cannot: a
876+
new ``dynamic_default=True`` catalog entry without a corresponding
877+
overlay branch in ``cli/scan.py:_dynamic_check_defaults`` would
878+
return only the catalog static and the resolver would never see
879+
the stronger manifest-effective default, silently widening the
880+
tier-crossing bypass.
881+
"""
882+
from agents_shipgate.checks.registry import CHECK_METADATA
883+
from agents_shipgate.cli.scan import _dynamic_check_defaults
884+
from agents_shipgate.config.schema import AgentsShipgateManifest
885+
from agents_shipgate.inputs.policy_packs import LoadedPolicyPacks
886+
887+
overlay, expected_dynamic = _DYNAMIC_DEFAULT_OVERLAY_CASES[check_id]
888+
manifest_dict = {**_base_manifest_dict(), **overlay}
889+
manifest = AgentsShipgateManifest.model_validate(manifest_dict)
890+
catalog = list(CHECK_METADATA)
891+
catalog_default = next(e.default_severity for e in catalog if e.id == check_id)
892+
assert catalog_default != expected_dynamic, (
893+
f"test rigged wrong: expected_dynamic={expected_dynamic!r} matches "
894+
f"the catalog static default for {check_id}; the overlay must be "
895+
"STRONGER than the catalog so the test can observe it firing."
823896
)
897+
824898
aggregated = _dynamic_check_defaults(
825899
manifest,
826900
LoadedPolicyPacks(loaded=[], rules=[], warnings=[]),
827-
catalog=list(CHECK_METADATA),
901+
catalog=catalog,
828902
)
829903

830-
missing = dynamic_check_ids - aggregated.keys()
831-
assert not missing, (
832-
f"dynamic_default check IDs missing from "
833-
f"_dynamic_check_defaults seed: {sorted(missing)}. "
834-
"Either the seed loop regressed or a new dynamic_default check "
835-
"needs an aggregator overlay branch."
904+
assert aggregated.get(check_id) == expected_dynamic, (
905+
f"For {check_id}, expected dynamic default {expected_dynamic!r} after "
906+
f"populating the manifest section, but _dynamic_check_defaults "
907+
f"returned {aggregated.get(check_id)!r}. The aggregator likely "
908+
f"lacks an overlay branch — the seed loop alone returns the catalog "
909+
f"static ({catalog_default!r}). Add a branch in "
910+
f"cli/scan.py:_dynamic_check_defaults that reads the relevant "
911+
f"manifest section for {check_id}."
836912
)
837913

838914

0 commit comments

Comments
 (0)