Skip to content

Commit 306906a

Browse files
wz-gsamogulOpenCode Agent
authored
fix(acq-kits): give verify scripts a "could not check" verdict state (#460)
All seven kit verify scripts ended with the same two-state verdict: ```bash [ "$fail" -eq 0 ] ``` That answers *"did anything fail?"* and cannot answer *"did we actually check anything?"*. Two consequences, both measured rather than inferred. ## 1. A run where nothing executed reported success Reproduced in isolation: ``` pass=0 fail=0 -> "All checks passed." exit 0 ``` ## 2. A skipped check was indistinguishable from a passing one Across the seven scripts there are **29 skip/NOTE sites and 9 warn sites that increment no counter.** Worst case: `git-ssh-sign` skips its **end-to-end signed-commit check — the kit's entire purpose** — whenever no key is in the forwarded SSH agent, and still reported "All checks passed." And `openchamber`/`paseo` *do* define `skip()` and `warn()`, but they're cosmetic `printf` with no counter: ```bash skip() { printf ' \033[33mSKIP\033[0m %s\n' "$1"; } ``` So `paseo` can print **19 yellow SKIPs** and still claim the offline gate passed. This is the ninth instance of this defect class across this family of repos (ENV-1, SEC-6, DX-7/8/9, SEC-12/13/17, DEP-4 being the prior ones), which is why the fix is a shared contract rather than seven local patches. ## The contract `integrations/isolation/acq-kits/verify-report.sh` — five states, and a verdict that is a function of a coverage record rather than a failure counter: | Reporter | Meaning | Degrades verdict? | |---|---|---| | `ok` | ran and passed | — | | `bad` | ran and failed | yes → `1` | | `unver` | **could not run** (missing tool, absent credential, no network) | yes → `3` | | `skip` | not applicable in this configuration | no | | `warn` | advisory; a documented fallback was taken | no | ``` exit 0 every check that ran passed, and at least one ran exit 1 at least one check FAILED exit 3 nothing failed, but the run is not a pass ``` `3` is distinct from `1` so a caller can tell *"we looked and found problems"* from *"we could not look."* The `unver`/`skip` split is the load-bearing part — collapsing them is how a verify script ends up claiming coverage it does not have. ## Staged on purpose: two kits here The two smallest (`git-ssh-sign`, `usai-provider`), so the contract is reviewable in isolation before a bulk edit. Remaining: `openchamber` (641 lines, 3 gates), `paseo` (864, 3), `pi-coding-agent` (337, 3), `agentic-coding-playbook` (167), `zscaler-ca-certificate` (167) — listed in the test's `UNCONVERTED` set, which must reach empty. Converted sites: - **git-ssh-sign** — no host SSH key → `unver`; no forwarded key → `unver` (the kit's only end-to-end assertion). - **usai-provider** — no USAi secret → `unver`; API probe empty → `unver`. That last one matters: `in_sbx()` swallows the exit status, so an empty result means *either* "no curl in the guest" *or* "`sbx exec` failed" — indistinguishable, and never a pass. ## Enforcement, because nothing in CI runs these scripts They need a live sandbox CI cannot create, and this repo has **no shellcheck hook** — so 2,400 lines of shell are currently linted by nothing. `scripts/tests/test_verify_contract.py` (12 tests) reads them as text and asserts: - the library keeps **both** degrading branches — deleting either silently restores the two-state verdict for every caller at once; - converted kits source it, call `verify_reset`, and **don't shadow its reporters** with no-op locals (which would pass every other test while behaving exactly as before); - no bare `echo "SKIP"` remains in a converted kit; - `UNCONVERTED` describes reality **in both directions** — a kit converted without being delisted fails, and so does a listed kit that reverted. Every helper asserts its input set is non-empty first, so the suite cannot pass vacuously over a glob that stopped matching. That would be the same defect wearing a different hat. ## A near-miss worth recording The library was first written to `acq-kits/lib/`. `.gitignore:13` carries the stock Python build-artifact block, whose `lib/` rule **silently excluded it**: ``` $ git check-ignore -v integrations/isolation/acq-kits/lib/verify-report.sh .gitignore:13:lib/ integrations/isolation/acq-kits/lib/verify-report.sh ``` Committing would have shipped two kits sourcing a file git never tracked — every verify run dying on *"No such file or directory."* Caught because `git status` listed the two modified kits and the new test but **not** the library. It now sits at the acq-kits root beside `validate-kits.py` (the existing precedent for shared assets there), and the reason is recorded in the test file so nobody re-adds a `lib/`. ## Verification ``` all five verdict states exercised directly: nothing ran -> exit 3 clean -> exit 0 a failure -> exit 1 an unver -> exit 3 skip-only -> exit 0 degraded path of the TESTS: reverting usai-provider to origin/main -> FAILS "does not source lib/verify-report.sh" pytest scripts/tests/ -> 413 passed (12 new) validate_repo.py --root . -> exit 0 validate-kits.py --root . --strict -> all kits OK scan_unsafe_shell.py . -> 0 errors, 0 warnings shellcheck -s sh verify-report.sh -> exit 0 ruff check / ruff format --check -> clean markdownlint -> 0 issues git diff --check -> clean shellcheck on the converted kits: git-ssh-sign 10 findings before / 10 after usai-provider 7 findings before / 7 after (pre-existing SC2015/SC2016 info-level; I added none) ``` I exercised the degraded paths of both the library *and* the tests, because a guard that only passes when things are fine is indistinguishable from a guard that always passes. ## Not in this PR, deliberately General shell linting. These scripts have never been shellcheck'd, and adding a hook would surface unrelated findings across 2,400 lines — that belongs in its own change. This one closes the verdict-honesty gap only. `acq-kits/README.md` gains a short section documenting what each exit code means, since the exit contract is now something a caller can rely on. --- AI-assisted (OpenCode), prepared on operator instruction. **Human review and merge required.** --------- Co-authored-by: Bret Mogilefsky <[email protected]> Co-authored-by: OpenCode Agent <[email protected]>
1 parent 80a03ee commit 306906a

5 files changed

Lines changed: 494 additions & 31 deletions

File tree

‎integrations/isolation/acq-kits/README.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,3 +96,46 @@ make test-kits
9696
# Live, per-backend end-to-end check (needs a backend CLI + a sandbox-capable host):
9797
<kit>/scripts/verify
9898
```
99+
100+
### What a `scripts/verify` exit code means
101+
102+
The shared verdict contract lives in `verify-report.sh` in this directory, and
103+
**adoption is staged**: only the kits listed as converted below implement it
104+
today. A verdict is a function of a coverage record, not a failure counter, so a
105+
converted run that exercised nothing cannot report the same result as a run that
106+
passed.
107+
108+
**Converted kits** — `git-ssh-sign`, `usai-provider`. For these, and only these,
109+
the exit codes and reporters below apply.
110+
111+
| Exit | Meaning |
112+
|---|---|
113+
| `0` | every check that ran passed, and at least one check ran |
114+
| `1` | at least one check FAILED |
115+
| `3` | nothing failed, but the run is **not** a pass: a check could not be performed, or no check ran at all |
116+
117+
`3` is distinct from `1` so a caller can tell *"we looked and found problems"*
118+
from *"we could not look"*. Treat any non-zero exit as "do not ship".
119+
120+
The five reporters: `ok` / `bad` are the ran-and-passed / ran-and-failed cases;
121+
`unver` is **could not check** (missing tool, absent credential, no network) and
122+
degrades the verdict to `3`; `skip` is *not applicable in this configuration*
123+
(e.g. a backend-specific check on the other backend) and does **not** degrade
124+
it; `warn` is advisory.
125+
126+
**Unconverted kits** — `agentic-coding-playbook`, `oci-engine`, `opencode`,
127+
`openchamber`, `paseo`, `pi-coding-agent`, `zscaler-ca-certificate`. These still
128+
use the older two-state verdict, so nothing above applies to them. A non-zero
129+
exit means a check failed *or* a precondition could not be met — the two are not
130+
distinguished — and a `0` does **not** establish that every check ran: these
131+
scripts report an unmet precondition with a bare message that increments no
132+
counter, then exit `0` on the strength of the checks that did run. Treat a `0`
133+
here as "nothing that ran failed", never as full coverage.
134+
135+
The authoritative classification is `scripts/tests/test_verify_contract.py`. It
136+
executes the verdict contract for converted kits, asserts each unconverted kit
137+
still matches its description, and compares both lists above against its own, so
138+
the split cannot rot in either direction: converting a kit without moving it
139+
fails, a converted kit reverting fails, and a README edit alone fails. The
140+
unconverted list must reach empty, at which point this carve-out goes away and
141+
the contract above applies to every kit.

‎integrations/isolation/acq-kits/git-ssh-sign/scripts/verify‎

Lines changed: 14 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,11 @@ WORK="$(mktemp -d "${TMPDIR:-/tmp}/git-ssh-sign-verify.XXXXXX")"
3131
CREATE_LOG="$WORK/sbx-create.log"
3232
KEEP="${KEEP:-0}"
3333

34-
pass=0
35-
fail=0
36-
ok() { printf ' \033[32mPASS\033[0m %s\n' "$1"; pass=$((pass+1)); }
37-
bad() { printf ' \033[31mFAIL\033[0m %s\n' "$1"; fail=$((fail+1)); }
38-
info() { printf '\n\033[1m%s\033[0m\n' "$1"; }
34+
# Shared verdict contract (ok / bad / unver / skip / warn + verify_verdict).
35+
# Sourced rather than redefined here so a future fix to the contract cannot
36+
# leave this kit's copy behind.
37+
. "$KIT_DIR/../verify-report.sh"
38+
verify_reset
3939

4040
cleanup() {
4141
if [ "$KEEP" = "1" ]; then
@@ -51,16 +51,16 @@ in_sbx() { sbx exec "$SBX_NAME" -- sh -c "$1" </dev/null 2>/dev/null || true; }
5151

5252
info "0. Preconditions"
5353
if ! command -v sbx >/dev/null 2>&1; then
54-
echo " sbx not on PATH. Install sbx >= 0.34.0 and retry." >&2
55-
exit 1
54+
unver "sbx not on PATH — install sbx >= 0.34.0 and retry"
55+
verify_verdict "Summary"
56+
exit $?
5657
fi
5758
echo " sbx version: $(sbx version 2>/dev/null | grep -oE 'v?[0-9]+\.[0-9]+(\.[0-9]+)?' | head -n1)"
5859
[ -f "$KIT_DIR/spec.yaml" ] && ok "kit spec.yaml present" || { bad "spec.yaml missing"; exit 1; }
5960
if ssh-add -L >/dev/null 2>&1; then
6061
ok "host SSH agent has a key loaded (signing can succeed)"
6162
else
62-
echo " NOTE: no key in the host SSH agent. The kit installs fine, but signing" >&2
63-
echo " will fail until you run: ssh-add ~/.ssh/id_ed25519" >&2
63+
unver "host SSH agent has no key loaded — signing cannot be exercised (run: ssh-add ~/.ssh/id_ed25519)"
6464
fi
6565

6666
info "1. sbx kit validate"
@@ -102,12 +102,10 @@ if [ "$(in_sbx 'ssh-add -L >/dev/null 2>&1 && echo yes')" = "yes" ]; then
102102
')
103103
[ "$out" = "SIGNED" ] && ok "commit was signed and verified" || bad "commit not signed (got: ${out:-none})"
104104
else
105-
echo " SKIP: no key in the forwarded SSH agent inside the sandbox." >&2
106-
echo " Load one on the host (ssh-add) and re-run for the full check." >&2
105+
# This is the kit's whole purpose. Reporting it as anything other than
106+
# UNVERIFIED would let the script claim the signing path works when it was
107+
# never exercised.
108+
unver "end-to-end signed commit — no key in the forwarded SSH agent (load one with ssh-add, then re-run)"
107109
fi
108110

109-
info "Summary"
110-
printf ' %d passed, %d failed\n' "$pass" "$fail"
111-
[ "$fail" -eq 0 ] && printf ' \033[32mAll checks passed.\033[0m\n' || printf ' \033[31mSome checks failed — see notes above.\033[0m\n'
112-
113-
[ "$fail" -eq 0 ]
111+
verify_verdict "Summary"

‎integrations/isolation/acq-kits/usai-provider/scripts/verify‎

Lines changed: 13 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -39,11 +39,11 @@ CREATE_LOG="$WORK/sbx-create.log"
3939
KEEP="${KEEP:-0}"
4040
USAI_MODELS_URL="https://api.gsa.usai.gov/api/v1/models"
4141

42-
pass=0
43-
fail=0
44-
ok() { printf ' \033[32mPASS\033[0m %s\n' "$1"; pass=$((pass+1)); }
45-
bad() { printf ' \033[31mFAIL\033[0m %s\n' "$1"; fail=$((fail+1)); }
46-
info() { printf '\n\033[1m%s\033[0m\n' "$1"; }
42+
# Shared verdict contract (ok / bad / unver / skip / warn + verify_verdict).
43+
# Sourced rather than redefined here so a future fix to the contract cannot
44+
# leave this kit's copy behind.
45+
. "$KIT_DIR/../verify-report.sh"
46+
verify_reset
4747

4848
cleanup() {
4949
if [ "$KEEP" = "1" ]; then
@@ -59,16 +59,16 @@ in_sbx() { sbx exec "$SBX_NAME" -- sh -c "$1" </dev/null 2>/dev/null || true; }
5959

6060
info "0. Preconditions"
6161
if ! command -v sbx >/dev/null 2>&1; then
62-
echo " sbx not on PATH. Install sbx >= 0.34.0 and retry." >&2
63-
exit 1
62+
unver "sbx not on PATH — install sbx >= 0.34.0 and retry"
63+
verify_verdict "Summary"
64+
exit $?
6465
fi
6566
echo " sbx version: $(sbx version 2>/dev/null | grep -oE 'v?[0-9]+\.[0-9]+(\.[0-9]+)?' | head -n1)"
6667
[ -f "$KIT_DIR/spec.yaml" ] && ok "kit spec.yaml present" || { bad "spec.yaml missing"; exit 1; }
6768
if sbx secret ls 2>/dev/null | grep -q 'api.gsa.usai.gov'; then
6869
ok "USAi custom secret is set"
6970
else
70-
echo " NOTE: no USAi custom secret found. Set it with:" >&2
71-
echo " sbx secret set-custom -g --host api.gsa.usai.gov --env USAI_API_KEY" >&2
71+
unver "USAi custom secret is not set — the credential path cannot be exercised (sbx secret set-custom -g --host api.gsa.usai.gov --env USAI_API_KEY)"
7272
fi
7373

7474
info "1. sbx kit validate"
@@ -121,14 +121,12 @@ echo " HTTP status: ${status:-<none>}"
121121
if [ "$status" = "200" ]; then
122122
ok "USAi models API returned 200"
123123
elif [ -z "$status" ]; then
124-
echo " NOTE: could not probe (no curl, or no network); skipping." >&2
124+
# in_sbx swallows the exit status, so an empty result means either "no curl in
125+
# the guest" or "sbx exec failed" — both are "could not look", never a pass.
126+
unver "USAi models API — could not probe (no curl in the guest, or no network)"
125127
else
126128
bad "USAi API returned $status (key missing/expired, or egress blocked)"
127129
echo " Check: sbx policy log $SBX_NAME (api.gsa.usai.gov), and the USAi secret." >&2
128130
fi
129131

130-
info "Summary"
131-
printf ' %d passed, %d failed\n' "$pass" "$fail"
132-
[ "$fail" -eq 0 ] && printf ' \033[32mAll checks passed.\033[0m\n' || printf ' \033[31mSome checks failed — see notes above.\033[0m\n'
133-
134-
[ "$fail" -eq 0 ]
132+
verify_verdict "Summary"
Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
#!/usr/bin/env bash
2+
#
3+
# verify-report.sh — the shared verdict contract for acq-kit `scripts/verify`.
4+
#
5+
# A verdict is a function of a COVERAGE RECORD, not a failure counter: a run
6+
# that checked nothing, and a run where every check passed, must not produce
7+
# the same result. This library supplies that record and the five states a
8+
# verify script needs to express it.
9+
#
10+
# STATES
11+
#
12+
# ok "<what passed>" a check ran and passed
13+
# bad "<what failed>" a check ran and failed
14+
# unver "<what, and why not>" a check COULD NOT RUN — missing tool, absent
15+
# credential, no network. Not a pass. Counted,
16+
# and it degrades the verdict.
17+
# skip "<what, and why>" deliberately not applicable in this
18+
# configuration (e.g. a backend-specific check on
19+
# the other backend). Counted and reported, but
20+
# does NOT degrade the verdict.
21+
# warn "<note>" advisory only; a documented fallback was taken.
22+
# Counted and reported; does not degrade.
23+
#
24+
# `unver` degrades the verdict because a check could not run. `skip` records a
25+
# check that does not apply to the current configuration and does not degrade it.
26+
#
27+
# EXIT CODES (verify_verdict)
28+
#
29+
# 0 every check that ran passed, and at least one check ran
30+
# 1 at least one check FAILED
31+
# 3 nothing failed, but the run is not a clean pass:
32+
# - at least one check could not be performed (unverified > 0), or
33+
# - no check ran at all (pass == 0)
34+
#
35+
# 3 is separate from 1 so a caller can tell "we looked and found problems" from
36+
# "we could not look". A caller that only cares about red/green can test for
37+
# non-zero and will correctly refuse to treat a degraded run as success.
38+
#
39+
# USAGE
40+
#
41+
# KIT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
42+
# . "$KIT_DIR/../verify-report.sh"
43+
# verify_reset # initialize the counters
44+
# info "1. Something"
45+
# ok "it worked"
46+
# unver "API probe — no network"
47+
# verify_verdict "Offline gate" # prints the summary, returns 0/1/3
48+
#
49+
# `verify_verdict` takes an optional label so a script with more than one gate
50+
# (openchamber, paseo and pi-coding-agent each have three) can name each one.
51+
#
52+
# POSIX sh compatible on purpose: these scripts run on contributor machines with
53+
# whatever /bin/sh is, and the arithmetic and parameter expansion here stay
54+
# within that subset.
55+
56+
# Counters. Deliberately not `local` — they are the script-wide coverage record.
57+
verify_reset() {
58+
pass=0
59+
fail=0
60+
unverified=0
61+
skipped=0
62+
warned=0
63+
}
64+
65+
ok() {
66+
printf ' \033[32mPASS\033[0m %s\n' "$1"
67+
pass=$((pass + 1))
68+
}
69+
70+
bad() {
71+
printf ' \033[31mFAIL\033[0m %s\n' "$1"
72+
fail=$((fail + 1))
73+
}
74+
75+
# Could not check. The message SHOULD say what was not checked AND why, because
76+
# the reader's next question is always "why not".
77+
unver() {
78+
printf ' \033[35mUNVERIFIED\033[0m %s\n' "$1"
79+
unverified=$((unverified + 1))
80+
}
81+
82+
# Not applicable in this configuration. Does not degrade the verdict.
83+
skip() {
84+
printf ' \033[33mSKIP\033[0m %s\n' "$1"
85+
skipped=$((skipped + 1))
86+
}
87+
88+
# Advisory note; a documented fallback was taken. Does not degrade the verdict.
89+
warn() {
90+
printf ' \033[33mWARN\033[0m %s\n' "$1"
91+
warned=$((warned + 1))
92+
}
93+
94+
info() {
95+
printf '\n\033[1m%s\033[0m\n' "$1"
96+
}
97+
98+
# Print the coverage record and return the verdict.
99+
#
100+
# $1 — optional gate label (default "Summary").
101+
verify_verdict() {
102+
_vr_label="${1:-Summary}"
103+
104+
info "$_vr_label"
105+
printf ' %d passed, %d failed, %d unverified, %d skipped, %d warned\n' \
106+
"$pass" "$fail" "$unverified" "$skipped" "$warned"
107+
108+
if [ "$fail" -gt 0 ]; then
109+
printf ' \033[31m%s FAILED — see above.\033[0m\n' "$_vr_label"
110+
return 1
111+
fi
112+
113+
# Nothing ran. Arithmetic over zero must never produce a pass: pass=0 and
114+
# fail=0 is a report of nothing proven, not a report of success.
115+
if [ "$pass" -eq 0 ]; then
116+
printf ' \033[35m%s UNVERIFIED — no check ran, so nothing was proven.\033[0m\n' \
117+
"$_vr_label"
118+
return 3
119+
fi
120+
121+
if [ "$unverified" -gt 0 ]; then
122+
printf ' \033[35m%s UNVERIFIED — %d check(s) could not be performed; this is not a pass.\033[0m\n' \
123+
"$_vr_label" "$unverified"
124+
return 3
125+
fi
126+
127+
printf ' \033[32m%s passed.\033[0m\n' "$_vr_label"
128+
return 0
129+
}

0 commit comments

Comments
 (0)