fix(acq-kits): give verify scripts a "could not check" verdict state by wz-gsa · Pull Request #460 · GSA-TTS/agentic-coding-patterns

wz-gsa · 2026-10-02T15:00:31Z

All seven kit verify scripts ended with the same two-state verdict:

[ "$fail" -eq 0 ]

That answers "did anything fail?" and cannot answer "did we actually check anything?". Two consequences, both measured rather than inferred.

1. A run where nothing executed reported success

Reproduced in isolation:

pass=0 fail=0  ->  "All checks passed."   exit 0

2. A skipped check was indistinguishable from a passing one

Across the seven scripts there are 29 skip/NOTE sites and 9 warn sites that increment no counter.

Worst case: git-ssh-sign skips its end-to-end signed-commit check — the kit's entire purpose — whenever no key is in the forwarded SSH agent, and still reported "All checks passed."

And openchamber/paseo do define skip() and warn(), but they're cosmetic printf with no counter:

skip() { printf '  \033[33mSKIP\033[0m %s\n' "$1"; }

So paseo can print 19 yellow SKIPs and still claim the offline gate passed.

This is the ninth instance of this defect class across this family of repos (ENV-1, SEC-6, DX-7/8/9, SEC-12/13/17, DEP-4 being the prior ones), which is why the fix is a shared contract rather than seven local patches.

The contract

integrations/isolation/acq-kits/verify-report.sh — five states, and a verdict that is a function of a coverage record rather than a failure counter:

Reporter	Meaning	Degrades verdict?
`ok`	ran and passed	—
`bad`	ran and failed	yes → `1`
`unver`	could not run (missing tool, absent credential, no network)	yes → `3`
`skip`	not applicable in this configuration	no
`warn`	advisory; a documented fallback was taken	no

exit 0   every check that ran passed, and at least one ran
exit 1   at least one check FAILED
exit 3   nothing failed, but the run is not a pass

3 is distinct from 1 so a caller can tell "we looked and found problems" from "we could not look." The unver/skip split is the load-bearing part — collapsing them is how a verify script ends up claiming coverage it does not have.

Staged on purpose: two kits here

The two smallest (git-ssh-sign, usai-provider), so the contract is reviewable in isolation before a bulk edit. Remaining: openchamber (641 lines, 3 gates), paseo (864, 3), pi-coding-agent (337, 3), agentic-coding-playbook (167), zscaler-ca-certificate (167) — listed in the test's UNCONVERTED set, which must reach empty.

Converted sites:

git-ssh-sign — no host SSH key → unver; no forwarded key → unver (the kit's only end-to-end assertion).
usai-provider — no USAi secret → unver; API probe empty → unver. That last one matters: in_sbx() swallows the exit status, so an empty result means either "no curl in the guest" or "sbx exec failed" — indistinguishable, and never a pass.

Enforcement, because nothing in CI runs these scripts

They need a live sandbox CI cannot create, and this repo has no shellcheck hook — so 2,400 lines of shell are currently linted by nothing. scripts/tests/test_verify_contract.py (12 tests) reads them as text and asserts:

the library keeps both degrading branches — deleting either silently restores the two-state verdict for every caller at once;
converted kits source it, call verify_reset, and don't shadow its reporters with no-op locals (which would pass every other test while behaving exactly as before);
no bare echo "SKIP" remains in a converted kit;
UNCONVERTED describes reality in both directions — a kit converted without being delisted fails, and so does a listed kit that reverted.

Every helper asserts its input set is non-empty first, so the suite cannot pass vacuously over a glob that stopped matching. That would be the same defect wearing a different hat.

A near-miss worth recording

The library was first written to acq-kits/lib/. .gitignore:13 carries the stock Python build-artifact block, whose lib/ rule silently excluded it:

$ git check-ignore -v integrations/isolation/acq-kits/lib/verify-report.sh
.gitignore:13:lib/	integrations/isolation/acq-kits/lib/verify-report.sh

Committing would have shipped two kits sourcing a file git never tracked — every verify run dying on "No such file or directory." Caught because git status listed the two modified kits and the new test but not the library. It now sits at the acq-kits root beside validate-kits.py (the existing precedent for shared assets there), and the reason is recorded in the test file so nobody re-adds a lib/.

Verification

all five verdict states exercised directly:
  nothing ran -> exit 3     clean -> exit 0      a failure -> exit 1
  an unver    -> exit 3     skip-only -> exit 0

degraded path of the TESTS: reverting usai-provider to origin/main
  -> FAILS "does not source lib/verify-report.sh"

pytest scripts/tests/                -> 413 passed (12 new)
validate_repo.py --root .            -> exit 0
validate-kits.py --root . --strict   -> all kits OK
scan_unsafe_shell.py .               -> 0 errors, 0 warnings
shellcheck -s sh verify-report.sh    -> exit 0
ruff check / ruff format --check     -> clean
markdownlint                         -> 0 issues
git diff --check                     -> clean

shellcheck on the converted kits:
  git-ssh-sign    10 findings before / 10 after
  usai-provider    7 findings before /  7 after    (pre-existing SC2015/SC2016
                                                    info-level; I added none)

I exercised the degraded paths of both the library and the tests, because a guard that only passes when things are fine is indistinguishable from a guard that always passes.

Not in this PR, deliberately

General shell linting. These scripts have never been shellcheck'd, and adding a hook would surface unrelated findings across 2,400 lines — that belongs in its own change. This one closes the verdict-honesty gap only.

acq-kits/README.md gains a short section documenting what each exit code means, since the exit contract is now something a caller can rely on.

AI-assisted (OpenCode), prepared on operator instruction. Human review and merge required.

All seven kit verify scripts ended with the same two-state verdict: [ "$fail" -eq 0 ] That answers "did anything fail?" and cannot answer "did we actually check anything?". Two consequences, both measured: 1. A run where NOTHING executed reported success. With pass=0 and fail=0 the test is true, so the script printed "All checks passed." and exited 0 having verified nothing. Reproduced in isolation: pass=0 fail=0 -> "All checks passed." exit 0 2. A skipped check was indistinguishable from a passing one. Across the seven scripts there are 29 skip/NOTE sites and 9 warn sites that increment no counter. Worst case: git-ssh-sign skips its end-to-end signed-commit check -- the kit's entire purpose -- whenever no key is in the forwarded SSH agent, and still reported "All checks passed." openchamber and paseo DO define skip() and warn(), but they are cosmetic printf with no counter, so a run can print 19 yellow SKIPs and still claim the offline gate passed. This is the ninth instance of this defect class in this family of repos (ENV-1, SEC-6, DX-7/8/9, SEC-12/13/17, DEP-4 are the prior ones), so the fix is a shared contract rather than seven local patches. WHAT THIS ADDS. integrations/isolation/acq-kits/verify-report.sh: five states and a verdict that is a function of a coverage record, not of a failure counter. ok ran and passed bad ran and failed unver COULD NOT RUN (missing tool, absent credential, no network) -- counted, and degrades the verdict skip not applicable in this configuration -- counted, does NOT degrade warn advisory; a documented fallback was taken -- does not degrade exit 0 every check that ran passed, and at least one ran exit 1 at least one check FAILED exit 3 nothing failed but the run is not a pass: a check could not be performed, or no check ran at all 3 is distinct from 1 so a caller can tell "we looked and found problems" from "we could not look". The unver/skip split is the load-bearing part: collapsing them is how a verify script claims coverage it does not have. STAGED CONVERSION. Two kits here -- git-ssh-sign and usai-provider, the two smallest -- so the contract is reviewable in isolation before the bulk edit. Remaining: openchamber (641 lines, 3 gates), paseo (864, 3), pi-coding-agent (337, 3), agentic-coding-playbook (167), zscaler-ca-certificate (167). They are listed in the test's UNCONVERTED set, which must reach empty. Converted sites: git-ssh-sign - no host SSH key -> unver (was a bare echo NOTE) - no forwarded key -> unver (was a bare echo SKIP; this is the kit's only end-to-end assertion) usai-provider - no USAi secret -> unver (was a bare echo NOTE) - API probe empty -> unver. in_sbx() swallows the exit status, so an empty result means EITHER "no curl in the guest" OR "sbx exec failed" -- indistinguishable, and never a pass. ENFORCEMENT. scripts/tests/test_verify_contract.py, 12 tests, because nothing in CI runs these shell scripts: they need a live sandbox this repo's CI cannot create, and the repo has no shellcheck hook, so 2,400 lines of shell are currently linted by nothing. The tests read the scripts as text and assert the library keeps BOTH degrading branches (a deleted branch silently restores the two-state verdict for every caller at once), that converted kits source it and do not shadow its reporters with no-op locals, that no bare `echo "SKIP"` remains, and that UNCONVERTED describes reality in BOTH directions -- a kit converted without being delisted fails, and so does a listed kit that reverted. Every helper asserts its input set is non-empty first, so the suite cannot pass vacuously over a glob that stopped matching. A near-miss worth recording: the library was first written to acq-kits/lib/, and .gitignore:13 carries the stock Python build-artifact block whose `lib/` rule silently excluded it. Committing would have shipped two kits sourcing a file git never tracked -- every verify run dying on "No such file or directory". Caught because `git status` listed the two modified kits and the new test but not the library. It now sits at the acq-kits root beside validate-kits.py, the existing precedent for shared assets there, and the reason is recorded in the test file so nobody re-adds a lib/. Verified: all five verdict states exercised directly: nothing ran -> exit 3 clean -> exit 0 a failure -> exit 1 an unver -> exit 3 skip-only -> exit 0 pytest scripts/tests/ -> 413 passed (12 new) degraded path of the TESTS: reverting usai-provider to origin/main -> FAILS "does not source lib/verify-report.sh" validate_repo.py --root . -> exit 0 validate-kits.py --root . --strict -> all kits OK scan_unsafe_shell.py . -> 0 errors, 0 warnings shellcheck -s sh verify-report.sh -> exit 0 shellcheck, converted kits -> git-ssh-sign 10 before / 10 after usai-provider 7 before / 7 after (pre-existing SC2015/SC2016 info findings; I added none) ruff check / ruff format --check -> clean markdownlint -> 0 issues git diff --check -> clean NOT IN THIS PR, deliberately: general shell linting. These scripts have never been shellcheck'd and adding a hook would surface unrelated findings across 2,400 lines; that belongs in its own change. This one closes the verdict-honesty gap only. AI assistance disclosed: prepared by an AI agent (OpenCode) on operator instruction. Human review and merge still required.

…o converted CI caught a real defect in the contract test, from a direction I had not anticipated. Rebased onto main (1.10.0) and fixed. WHAT HAPPENED. `oci-engine` (#437) landed on main while this change was in review — an EIGHTH verify script that did not exist when I branched. The test derived its converted set by subtraction: {k: p for k, p in _verify_scripts().items() if k not in UNCONVERTED} so a kit in neither set was treated as CONVERTED by default. The four contract tests then failed against it with "oci-engine/scripts/verify does not source lib/verify-report.sh", which is true but misleading: the real problem was that a kit appeared that nobody had classified. I had guarded both directions I thought of — a kit converted but still listed, and a listed kit that reverted — and missed the one that actually happened. The default was the bug: an unclassified kit silently inherited a contract it does not implement, which is the same shape as the defect this whole change exists to fix (something unmeasured treated as passing). THE FIX. Two explicit sets, CONVERTED and UNCONVERTED, plus `test_every_kit_is_classified` asserting that every verify script appears in exactly one of them. A kit in neither is now a hard error naming the real cause, and the sets cannot name a kit that has no verify script. Proved it by simulating a ninth unclassified kit: AssertionError: verify script(s) in no classification set: ['zz-phantom']. Add each to CONVERTED (if it sources verify-report.sh) or to UNCONVERTED (if it is still on the old two-state verdict). A kit is never converted by default. CORRECTED COUNTS, recounted on the rebased tree rather than carried forward: there are EIGHT verify scripts, not seven, and 35 uncounted `skip`/NOTE sites, not 29 (9 `warn` sites unchanged). oci-engine itself has 303 lines, 2 gates, and 2 `[ "$fail" -eq 0 ]` verdicts, so it has the primary defect too; it is listed as UNCONVERTED for the staged bulk conversion. Stale numbers in the library header and the test docstring are updated. UNCONVERTED is now: agentic-coding-playbook, oci-engine, openchamber, paseo, pi-coding-agent, zscaler-ca-certificate. It must reach empty. Unrelated to the code, but it is why this took a second pass: the CI log could not be read at first because `gh run view --log` failed with `x509: certificate signed by unknown authority`. Root cause is Go-specific and documented in the review workspace, not here — Go's x509.SystemCertPool() omits macOS admin-installed roots (121 of the host's 177), so any Go tool fails against a proxy-intercepted host while `curl` to the same host verifies fine. Reading the log required pointing SSL_CERT_FILE at a bundle including the admin roots. No repo change needed; noted so the next person who sees that error does not mistake it for a CI outage. Verified on the rebased tree: pytest scripts/tests/ -> 420 passed (13 contract tests) unclassified-kit probe -> FAILS with the naming assertion validate_repo.py --root . -> exit 0 validate-kits.py --root . --strict -> all kits OK scan_unsafe_shell.py . -> 0 errors, 0 warnings shellcheck -s sh verify-report.sh -> exit 0 ruff check / ruff format --check -> clean AI assistance disclosed: prepared by an AI agent (OpenCode) on operator instruction. Human review and merge still required.

wz-gsa · 2026-10-02T16:11:29Z

CI is green now (7/7). Rebased onto main (1.10.0) and fixed a real defect the failure exposed — in my own test, not the library.

What CI caught

oci-engine (#437) landed on main while this was in review — an eighth verify script that didn't exist when I branched. The test derived its converted set by subtraction:

{k: p for k, p in _verify_scripts().items() if k not in UNCONVERTED}

So a kit in neither set was treated as converted by default, and four contract tests failed against it with "oci-engine/scripts/verify does not source lib/verify-report.sh" — true, but misleading. The real problem was that a kit appeared nobody had classified.

I'd guarded both directions I thought of — a kit converted but still listed, and a listed kit that reverted — and missed the one that actually happened. The default was the bug: an unclassified kit silently inherited a contract it doesn't implement, which is the same shape as the defect this whole change exists to fix.

The fix

Two explicit sets, CONVERTED and UNCONVERTED, plus test_every_kit_is_classified asserting every verify script appears in exactly one. Proved it by simulating a ninth unclassified kit:

AssertionError: verify script(s) in no classification set: ['zz-phantom'].
Add each to CONVERTED (if it sources verify-report.sh) or to UNCONVERTED
(if it is still on the old two-state verdict). A kit is never converted by default.

Corrected counts

Recounted on the rebased tree rather than carried forward from the original description:

Eight verify scripts, not seven.
35 uncounted skip/NOTE sites, not 29. (9 warn sites, unchanged.)
oci-engine has 303 lines, 2 gates, and 2 [ "$fail" -eq 0 ] verdicts — it has the primary defect too, and is listed UNCONVERTED for the staged bulk conversion.

Stale numbers in the library header and test docstring are updated. UNCONVERTED is now agentic-coding-playbook, oci-engine, openchamber, paseo, pi-coding-agent, zscaler-ca-certificate — it must reach empty.

Verification on the rebased tree

pytest scripts/tests/                -> 420 passed (13 contract tests)
unclassified-kit probe               -> FAILS with the naming assertion
validate_repo.py --root .            -> exit 0
validate-kits.py --root . --strict   -> all kits OK
scan_unsafe_shell.py .               -> 0 errors, 0 warnings
shellcheck -s sh verify-report.sh    -> exit 0
ruff check / ruff format --check     -> clean
gh pr checks                         -> 7/7 pass

One note, not a repo issue

I initially couldn't read the CI log at all: gh run view --log failed with x509: certificate signed by unknown authority. Worth recording in case anyone else hits it, because it looks like a CI outage and isn't.

Go's crypto/x509.SystemCertPool() omits macOS admin-installed roots — on my host macOS trusts 177 roots (158 Apple + 19 admin-added, including the corporate proxy CA) while Go sees 121, and the proxy root is not among them. So any Go tool fails against a proxy-intercepted host while curl to that same host verifies fine. GitHub's API is on a bypass list, which is why gh api works and only the Actions-log redirect to *.blob.core.windows.net fails.

Reading the log required pointing SSL_CERT_FILE at a bundle including the admin roots. No change needed here — just not a trust-store problem, and not something to work around by disabling verification.

AI-assisted (OpenCode), on operator instruction. Human review and merge required.

mogul

Technically this is fine, but the level of bot-rumination in the comments distracts from understanding what it does, and memorializes context that won't matter going forward.

So I'm grudgingly requesting changes, even though that means another review round-trip... Sorry.

If you respond with improvements that only land in comments, then I approve your next push in advance, so feel free to check the box that skips the "require review" gate and merge it then!

…ath-local

…im comment narration per mogul's review Two fixes, both from mogul's CHANGES_REQUESTED review: 1. Rebase onto main pulled in the opencode kit (#466), which landed with a verify script in neither CONVERTED nor UNCONVERTED, failing test_every_kit_is_classified exactly as the test is designed to catch (reproduced locally before fixing: `AssertionError: verify script(s) in no classification set: ['opencode']`). opencode's verify script is on the old two-state verdict, so it goes in UNCONVERTED alongside the other not-yet-migrated kits. 2. Trimmed six comment/docstring blocks that narrated the OLD two-state verdict's history ("used to end with...", "the old reporters had no...", the oci-engine near-miss anecdote) instead of describing the current code. Per review: this kind of narration belongs in the commit message, not in the code, where it has no reader value and persists as noise. Trimmed in: - integrations/isolation/acq-kits/git-ssh-sign/scripts/verify - integrations/isolation/acq-kits/usai-provider/scripts/verify - integrations/isolation/acq-kits/README.md - integrations/isolation/acq-kits/verify-report.sh (module header) - scripts/tests/test_verify_contract.py (module docstring + 2 inline comments + 1 docstring) Comment-only / allowlist-entry changes: no test logic, no shell logic, no shared-library behavior changed. Per the review's own standing offer ("if you respond with improvements that only land in comments... feel free to check the box that skips the require-review gate"), this should qualify to merge without a second review round-trip. Verification: - PYTHONPATH=scripts python3 -m pytest scripts/tests/: 420 passed (was failing 1/420 before the opencode classification fix, reproduced first). - python3 scripts/validate_repo.py: all validators pass. Co-authored-by: OpenCode Agent <[email protected]>

wz-gsa · 2026-10-06T16:25:42Z

Addressed in `4f26eb4`. Two changes:

Comment-only trims (the 6 flagged blocks) — `git-ssh-sign/scripts/verify`, `usai-provider/scripts/verify`, `README.md`, `verify-report.sh`'s module header, and `test_verify_contract.py`'s module docstring + 2 inline comments. All narration of the old two-state verdict's history removed; each now just describes the current contract.
One allowlist data entry, not comment-only: `opencode` added to `UNCONVERTED` in `test_verify_contract.py`. This isn't from your review — `main` moved (opencode kit landed via feat(acq-kits): install OpenCode v2 via opencode kit #466) between your review and now, and `test_every_kit_is_classified` correctly caught it as unclassified (reproduced locally first: `AssertionError: verify script(s) in no classification set: ['opencode']`). opencode's verify script is still on the old two-state verdict, so `UNCONVERTED` is the correct bucket.

Flagging #2 explicitly because your standing offer was for comment-only fixes — this PR also has that one allowlist line, so I didn't want to assume it's covered and self-merge on a stretched reading. Your call whether that still qualifies or you'd rather take another look.

CI is 7/7 green. `python3 scripts/validate_repo.py`: all validators pass.

Replace text-presence assertions with a six-state executable matrix, require real library/verdict invocations, and trim the remaining investigation-history narration. Co-authored-by: OpenCode Agent <[email protected]>

Require the exact shared reporter source, exercise both missing-tool degraded paths, and keep contract-test comments in present-tense invariant form. Co-authored-by: OpenCode Agent <[email protected]>

Correct the copyable usage example and format the executable contract tests. Co-authored-by: OpenCode Agent <[email protected]>

mogul

One approval-blocking documentation issue remains.

integrations/isolation/acq-kits/README.md:87 says “Verify scripts share one verdict contract,” then presents exit codes 0, 1, and 3 as the meaning of scripts/verify. But scripts/tests/test_verify_contract.py explicitly classifies only git-ssh-sign and usai-provider as CONVERTED; seven kits remain UNCONVERTED and retain the old two-state verdict. A caller can therefore rely on this documented contract for a listed kit and receive an incorrect result.

Please scope this section to converted kits and make the staged migration status clear, or defer the general exit-code documentation until every kit uses the shared reporter. The implementation and contract tests otherwise look sound.

The exit-code table described a contract only two kits implement, so a caller could trust a 0 from an unconverted kit whose 0 does not distinguish every-check-passed from a check that could not run. Document the staged split explicitly, state what an unconverted 0 does and does not establish, and derive both lists from the test suite so a README-only edit, a conversion, or a reversion each fail. Co-authored-by: OpenCode Agent <[email protected]>

wz-gsa · 2026-10-14T16:36:39Z

Addressed in 2dc0af4. You were right, and the defect was worse than a wording problem: the section asserted one contract while the suite classified only two of nine kits as implementing it, so a caller could read the table, run an unconverted kit, and treat its 0 as coverage evidence.

What changed

integrations/isolation/acq-kits/README.md now states the staged split explicitly and scopes the table to the kits that implement it:

Converted — git-ssh-sign, usai-provider. The exit-code table and the five reporters apply to these "and only these".
Unconverted — the other seven. Stated plainly: nothing above applies, a non-zero exit does not distinguish a failed check from an unmet precondition, and a 0 does not establish that every check ran.

On that last point I went looking for the concrete mechanism rather than asserting it generically. These scripts report an unmet precondition with a bare message that increments no counter, then exit 0 on the strength of whatever did run — for example zscaler-ca-certificate/scripts/verify:130 (NOTE: could not read fingerprint … skipping match) and agentic-coding-playbook/scripts/verify:64 (NOTE: no 'github' secret), and the local no-op skip() definitions in opencode, openchamber, paseo, and pi-coding-agent. All of those paths reach [ "$fail" -eq 0 ] and exit 0. The README now says to read such a 0 as "nothing that ran failed", never as full coverage.

The split is now enforced, not just described

Prose that must track a migration will drift, so scripts/tests/test_verify_contract.py gained test_readme_classification_matches_this_suite, which parses both lists out of the README paragraph and compares them to CONVERTED / UNCONVERTED. Measured, each drift direction fails:

converted-list over-claim inside the paragraph   -> FAIL
unconverted kit removed from the README only     -> FAIL

Three details worth flagging, because a guard like this is easy to write so that it passes vacuously:

It scans the whole paragraph, not the first sentence. An earlier draft stopped at the first . , which meant a later sentence in the same paragraph claiming extra kits were converted would have passed.
Kit names are intersected against the set of real scripts/verify paths, so backticked prose in the paragraph (exit codes, filenames) cannot be mistaken for a kit.
The unconverted lookup treats an absent section as the empty set. The README says the carve-out disappears once the list empties, and an earlier draft would have failed at exactly that moment — blaming the README for being correct. I simulated the end state (section removed, UNCONVERTED emptied, all nine kits converted) and the test passes.

The converted lookup still hard-fails on a missing section, since that one is not supposed to disappear.

Verification

scripts/tests/ (excluding the unrelated pre-existing
  test_oci_engine_vfs_fallback.py path failures)   424 passed
test_verify_contract.py                             23 passed
validate_repo.py                                   OK
validate-kits.py --strict                          all 10 valid
ruff check / ruff format                           clean

The five test_oci_engine_vfs_fallback.py failures are pre-existing on main and unrelated: the test targets files/home/oci-engine-config/oci-engine-vfs-fallback.sh while the script lives at files/home/oci-engine-vfs-fallback.sh. This PR touches neither.

No implementation or contract-test behavior changed — only the documentation scope plus the new guard.

AI-assisted (OpenCode). Advisory; requires your review.

wz-gsa · 2026-10-15T17:40:14Z

@mogul — branch updated to current main (5ec3be6), 7/7 checks green, and the blocker from your 604b5ba review is addressed in 2dc0af4 (described in my 2026-10-14 comment above).

One thing I can't clear myself: six inline review threads from your 2026-10-02 pass are still open.

resolved=false outdated=true  integrations/isolation/acq-kits/git-ssh-sign/scripts/verify
resolved=false outdated=true  integrations/isolation/acq-kits/README.md
resolved=false outdated=true  integrations/isolation/acq-kits/verify-report.sh        (x2)
resolved=false outdated=true  scripts/tests/test_verify_contract.py                   (x2)

All six are outdated — the lines they anchored to have since changed — but outdated is not resolved, and the repository ruleset sets required_review_thread_resolution: true, so they block merge independently of the review verdict. They were addressed across 4f26eb4 and 2dc0af4.

I'd rather not resolve another reviewer's threads unilaterally: resolving them is you agreeing the point is closed, and me marking them closed would assert that on your behalf. Could you take a pass and resolve the ones you're satisfied with, or re-raise any that are not actually addressed?

Also worth flagging for whoever approves: this PR touches scripts/tests/test_verify_contract.py, and .github/CODEOWNERS routes /scripts/ to @GSA-TTS/agentic-coding-admins rather than the broad @GSA-TTS/agentic-coding-team default. With require_code_owner_review: true in the ruleset, that means an approval here has to come from an admin — currently @mogul, @bryanlalexander, @mark-aronson, or @rpalmer-gsa (I'm the author, so not me). CODEOWNERS describes that line as a deliberate additional gate, so this is working as designed, not a misconfiguration.

Note the ruleset also has dismiss_stale_reviews_on_push: true, so an approval is dismissed by any later push — worth doing review and merge close together.

Verification at 5ec3be6 (merge of main, clean — no conflicts):

python -m pytest scripts/tests/ -q        ->  471 passed
python -m pytest .../test_verify_contract.py  ->  23 passed
python scripts/validate_repo.py           ->  All validators passed
CI                                        ->  7/7 SUCCESS

main brought in two new kits that ship scripts/verify (examples/personal-kit, examples/team-kit). I re-ran the drift guard after merging specifically to check they don't make the scoped verdict-contract table under-claim; it passes, because the guard intersects the prose against kits that actually have a scripts/verify rather than against a hardcoded list.

AI-assisted (OpenCode), advisory — a human should confirm before this merges.

🤖 I have created a release *beep* *boop* --- ## [1.14.0](v1.13.0...v1.14.0) (2026-10-19) ### Features * **paseo:** bump pinned Paseo CLI to 0.12.0 ([#504](#504)) ([0c5258e](0c5258e)) ### Bug Fixes * **acq-kits:** bump opencode kit to OpenCode 2.0.30 ([#499](#499)) ([c9d2281](c9d2281)) * **acq-kits:** give verify scripts a "could not check" verdict state ([#460](#460)) ([306906a](306906a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

## Summary Moves `PATTERNS_KIT_REF` from v1.12.0 (`56d1f5f`) to agentic-coding-patterns v1.15.0 (`f3c6acb`), so the built-in kit bundle picks up: - **opencode kit → OpenCode 2.0.30** (GSA-TTS/agentic-coding-patterns#499). OpenCode 2.0.0–2.0.29 ignore config `instructions` (anomalyco/opencode#51341, fixed in 2.0.30), so a sandbox on an image without a baked OpenCode (`devenv-shell`, or the `devenv-shell` default proposed in #584) currently runs the bundled 2.0.22 and silently drops `instructions`-based conventions. - **paseo kit → Paseo 0.12.0** (GSA-TTS/agentic-coding-patterns#504): fixes for OpenCode v2 under Paseo (daemon OOM on long v2 streams, multi-select answers, HTTP 500s instead of tool errors) and the host-sleep daemon-status fix. - **Kit verify scripts gain a "could not check" verdict** (GSA-TTS/agentic-coding-patterns#460); it only affects kit verify output. Also v1.13.0: usai-provider's host-side catalog normalizer (#436), the paseo identity-gate script (#453), and the team-kit and personal-kit templates under `acq-kits/examples/`. None changes what acq applies by default beyond the two kits above. Also v1.15.0: refreshed usai-provider model catalog (GSA-TTS/agentic-coding-patterns#519, GSA-TTS/agentic-coding-patterns#484), the OpenDesign opt-in kit (GSA-TTS/agentic-coding-patterns#438), and the provider-neutral input contract (GSA-TTS/agentic-coding-patterns#518). acq's provider-facts loader reads `provider-facts/usai.env`, which v1.15.0 does not change. The doc examples and Bats assertions that spell the pin move with it. KNOWN_FAILURE_MODES §9 now names 2.0.30 as the version the kit installs and notes its 15-minute default `chunkTimeout` (the timeout message and retry limits are unchanged from 2.0.22). ## Verification On the v1.15.0 pin: - Focused Bats: `135-configure`, `95-kit-ref-persistence`, `36-agent-catalog`, `50-kit-list-completeness` (63/63) - markdownlint on the changed docs On the earlier v1.14.0 pin (not re-run for v1.15.0): - `npm run lint:md` - `RUN_ACQ=1 scripts/verify-backends --only msb` on macOS, msb 0.7.8: 16 passed, 1 skipped (playbook fetch, no `gh` login), run with `TMPDIR` set to its resolved `/private/var/...` path to get past an unrelated create failure that also hits `main` (#594) - Downstream, with this pin on msb 0.7.8 and `ACQ_IMAGE=devenv-shell:1.0.0`: a team kit's end-to-end verify passed 71/71 (OpenCode 2.0.30 runs and loads `instructions`); with the paseo kit added, Paseo 0.12.0 drove an OpenCode 2.0.30 agent that received the team `instructions`, and its sessions survived a backup, `acq rm`, recreate and restore ## Rollback Revert this PR to restore the v1.12.0 bundle.

wz-gsa requested a review from a team as a code owner October 2, 2026 15:00

wz-gsa added 2 commits October 2, 2026 12:05

wz-gsa force-pushed the fix/verify-degraded-path branch from 54563f0 to 49572de Compare October 2, 2026 16:08

This was referenced Oct 2, 2026

acq-kits: convert the remaining six verify scripts to the shared verdict contract #463

Open

prime-agent kit: scripts/verify (pins + live assertions) #374

Closed

mogul requested changes Oct 2, 2026

View reviewed changes

mogul and others added 3 commits October 2, 2026 17:03

Merge branch 'main' into fix/verify-degraded-path

f850fce

Merge remote-tracking branch 'origin/main' into fix/verify-degraded-p…

cd305e6

…ath-local

wz-gsa mentioned this pull request Oct 7, 2026

feat(isolation): add the scope-layers pattern and a team-kit template #441

Merged

22 of 26 tasks

mogul mentioned this pull request Oct 8, 2026

feat(provider-facts): declare constrained provider credential operations #486

Open

8 tasks

wz-gsa added 4 commits October 8, 2026 18:01

Merge remote-tracking branch 'origin/main' into HEAD

1ccb230

test(acq-kits): execute the verify verdict contract

b7050f8

Replace text-presence assertions with a six-state executable matrix, require real library/verdict invocations, and trim the remaining investigation-history narration. Co-authored-by: OpenCode Agent <[email protected]>

fix(acq-kits): classify missing sbx as unverified

98efa06

Require the exact shared reporter source, exercise both missing-tool degraded paths, and keep contract-test comments in present-tense invariant form. Co-authored-by: OpenCode Agent <[email protected]>

docs(acq-kits): show the canonical verify-report source path

604b5ba

Correct the copyable usage example and format the executable contract tests. Co-authored-by: OpenCode Agent <[email protected]>

wz-gsa self-assigned this Oct 12, 2026

mogul requested changes Oct 14, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into pr460local

5ec3be6

mogul approved these changes Oct 15, 2026

View reviewed changes

mogul merged commit 306906a into main Oct 15, 2026
7 checks passed

mogul deleted the fix/verify-degraded-path branch October 15, 2026 18:08

github-actions Bot mentioned this pull request Oct 15, 2026

chore(main): release 1.14.0 #497

Merged

basiliskus mentioned this pull request Oct 20, 2026

feat(kits): pin agentic-coding-patterns v1.15.0 GSA-TTS/agentic-coding-quickstart#595

Merged

Navigation Menu

Repository navigation

fix(acq-kits): give verify scripts a "could not check" verdict state - #460

fix(acq-kits): give verify scripts a "could not check" verdict state#460
mogul merged 11 commits into
mainfrom
fix/verify-degraded-path

wz-gsa commented Oct 2, 2026

Uh oh!

wz-gsa commented Oct 2, 2026

Uh oh!

mogul left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wz-gsa commented Oct 6, 2026

Uh oh!

mogul left a comment

Uh oh!

wz-gsa commented Oct 14, 2026

Uh oh!

wz-gsa commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wz-gsa commented Oct 2, 2026

1. A run where nothing executed reported success

2. A skipped check was indistinguishable from a passing one

The contract

Staged on purpose: two kits here

Enforcement, because nothing in CI runs these scripts

A near-miss worth recording

Verification

Not in this PR, deliberately

Uh oh!

wz-gsa commented Oct 2, 2026

What CI caught

The fix

Corrected counts

Verification on the rebased tree

One note, not a repo issue

Uh oh!

mogul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wz-gsa commented Oct 6, 2026

Uh oh!

mogul left a comment

Choose a reason for hiding this comment

Uh oh!

wz-gsa commented Oct 14, 2026

What changed

The split is now enforced, not just described

Verification

Uh oh!

wz-gsa commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants