Repository navigation
Commit 8852e9d
Address PR #90 P1 + P2 review findings
P1 — Coerced-truthy bypass closed (plugin_validation.py).
The raw-metadata gate used ``value.get("dynamic_default") is True``, a
strict identity check. Pydantic's bool coercion accepts ``"true"``,
``"True"``, ``"yes"``, ``"on"``, and ``1`` as ``True``, so a plugin
declaring ``{"dynamic_default": "true", "floor_severity": "medium"}``
would pass the raw gate, get coerced to ``dynamic_default=True`` by
``_coerce_metadata``, and land as ``valid`` — reopening the silent
bypass this PR closes.
Fix: ``_raw_metadata_declares_dynamic_default`` now uses
``TypeAdapter(bool).validate_python(...)`` to mirror Pydantic's
coercion exactly. Anything Pydantic stores as ``dynamic_default=True``
on the constructed ``CheckMetadata`` is now rejected at the gate.
Values Pydantic refuses outright (arbitrary objects) fall through to
``_coerce_metadata`` as ``bad_metadata``.
Tests: 12 new parametrized cases in test_plugin_validation.py
covering 6 truthy raw values (``"true"``, ``"True"``, ``"TRUE"``,
``"yes"``, ``"on"``, ``1``) — each must land in
``dynamic_default_not_supported`` — and 7 falsey controls (``False``,
``"false"``, ``"False"``, ``"no"``, ``"off"``, ``0``, absent key) —
each must NOT trip the gate.
P2 — Aggregator-completeness test was vacuous (severity_override_floor.py).
The original test only asserted that every ``dynamic_default=True``
catalog ID appeared as a key in ``_dynamic_check_defaults`` output.
But the v0.18 seed loop (step 1) populates EVERY such key with the
catalog static default unconditionally — so a future catalog entry
that sets ``dynamic_default=True`` without adding the matching
overlay branch in ``cli/scan.py:_dynamic_check_defaults`` would
still pass the test. The resolver guard would not fire either
(the key is present). This silently widens the tier-crossing
bypass the PR exists to prevent.
Fix: replace the weak test with a per-check overlay-verification
registry ``_DYNAMIC_DEFAULT_OVERLAY_CASES`` mapping each
``dynamic_default=True`` catalog ID to
``(manifest_overlay, expected_dynamic_severity)`` where
``expected_dynamic`` is provably STRONGER than the catalog static.
Two tests now pin the contract:
- ``test_dynamic_default_overlay_registry_is_complete``: every
catalog ``dynamic_default=True`` ID must have a registry entry.
Fails the moment someone adds a new dynamic-default check
without thinking through (and documenting) the manifest path
that strengthens its default.
- ``test_dynamic_default_aggregator_overlay_fires``: parametrized
per registry entry. Builds the manifest with the strengthened
section, calls ``_dynamic_check_defaults``, asserts the
aggregator returns the manifest-effective severity (not the
catalog static). Fails when an overlay branch is missing.
Together the two new tests catch the regression class the seed
loop alone cannot.
P3 — Already addressed in prior follow-up commit 3fbb1e3
(docs/checks.md: "Plugin validation (v0.17+; six gates v0.18+)" now
documents ``dynamic_default_not_supported`` between metadata and
ID-collision gates).
Verification: 1198 tests pass (1192 + 12 new + −6 net from P2
restructuring); ruff clean; ``python scripts/generate_schemas.py
--check`` exits 0; coverage 87.23% (above 75% floor).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 3fbb1e3 commit 8852e9d
3 files changed
Lines changed: 229 additions & 38 deletions
File tree
- src/agents_shipgate/checks
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| |||
363 | 363 | | |
364 | 364 | | |
365 | 365 | | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
366 | 369 | | |
367 | 370 | | |
368 | 371 | | |
369 | 372 | | |
370 | 373 | | |
371 | 374 | | |
372 | | - | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
373 | 380 | | |
374 | 381 | | |
375 | | - | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
376 | 385 | | |
377 | 386 | | |
378 | 387 | | |
379 | 388 | | |
380 | 389 | | |
381 | 390 | | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
382 | 396 | | |
383 | 397 | | |
384 | | - | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
385 | 411 | | |
386 | 412 | | |
387 | 413 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
410 | 410 | | |
411 | 411 | | |
412 | 412 | | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
413 | 502 | | |
414 | 503 | | |
415 | 504 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
781 | 781 | | |
782 | 782 | | |
783 | 783 | | |
784 | | - | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
785 | 831 | | |
786 | | - | |
787 | | - | |
788 | | - | |
789 | | - | |
790 | | - | |
791 | | - | |
792 | | - | |
793 | | - | |
794 | | - | |
795 | | - | |
796 | | - | |
797 | | - | |
798 | | - | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
799 | 842 | | |
800 | 843 | | |
801 | | - | |
802 | | - | |
803 | | - | |
804 | 844 | | |
805 | 845 | | |
806 | 846 | | |
| |||
809 | 849 | | |
810 | 850 | | |
811 | 851 | | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
812 | 861 | | |
813 | | - | |
814 | | - | |
815 | | - | |
816 | | - | |
817 | | - | |
818 | | - | |
819 | | - | |
820 | | - | |
821 | | - | |
822 | | - | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
823 | 896 | | |
| 897 | + | |
824 | 898 | | |
825 | 899 | | |
826 | 900 | | |
827 | | - | |
| 901 | + | |
828 | 902 | | |
829 | 903 | | |
830 | | - | |
831 | | - | |
832 | | - | |
833 | | - | |
834 | | - | |
835 | | - | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
836 | 912 | | |
837 | 913 | | |
838 | 914 | | |
| |||
0 commit comments