Repository navigation
Commit 228cc4b
fix(usai-catalog): preserve and validate catalog role recommendations (#553)
🤖: AI agent (First Mate), AI-assisted change (OpenCode). Human review
required before merge.
> ## Merge status
> Merged main after #551; no longer
stacked. The diff against `main` is now only this PR's own changes:
`integrations/providers/usai/{README.md, scripts/build-catalog.mjs,
tests/recommendations.test.mjs}` and a docstring correction in
`scripts/tests/test_usai_catalog_schema.py`. Merged with a plain `git
merge` (no rebase, no force-push).
## Summary
Part of #419 (slice S1). Makes the USAi
catalog's `recommendations` (role defaults) a validated, preserved,
deterministic part of the catalog pipeline, so a catalog refresh can no
longer silently delete them. `catalog.json` is unchanged.
## Eligibility & Accountability
- [ ] Federal employee or contractor attestation: for the human author
of record to complete.
- [ ] Author understands the change and can explain it: for the human
author of record to complete.
- [ ] Right-to-release (CC0-1.0) attestation: for the human author of
record to complete.
## AI Assistance Disclosure
- [x] AI tools were used: OpenCode (First Mate agent) drafted the
change, tests and this description.
- [ ] Human verification of AI-generated content: pending reviewer.
## Type of Change
- [x] Bug fix
- [x] Infrastructure/tooling change
## Context
Found in the #419 audit and re-verified on main@7dd2ddc0:
- `build-catalog.mjs` neither produced nor preserved `recommendations`.
A bootstrap run to a temp file dropped the block, and a fixture refresh
did the same.
- `recommendations` was missing from `KEY_ORDER`, so
`serialize(JSON.parse(catalog.json))` did not reproduce the committed
file.
- `validateCatalog` did not check role names or that role ids exist in
`models[]`. A dangling id plus a made-up role produced 0 errors.
- `describeDrift` ignored `recommendations`.
- The schema omitted `summary` and `title`. That is fixed by #551, not
here.
## Plan (what changed)
- `scripts/build-catalog.mjs`
- `KEY_ORDER`: `recommendations` after `sources`, roles in fixed order.
- `validateCatalog`: role names come from
`schema.properties.recommendations.properties`; each value must be a
non-empty string naming a model in `models[]`; guards against non-object
blocks.
- Carry-forward: `readExistingRecommendations` reads the catalog at
`--out`; `carryForwardRecommendations` keeps the values and fails
closed. Applies to the live/fixture and bootstrap paths and to
`--check`.
- `describeDrift`: reports `recommendations.<role>: "old" -> "new"`.
- `tests/recommendations.test.mjs`: 31 offline tests (fixtures only,
temp output paths, no network).
- `README.md`: new "Role defaults" section.
- Review fix: the file at `--out` must be absent or a real catalog
(object, `schemaVersion` `usai-model-catalog/v1`, `models` array). `[]`,
`null`, `123`, `"text"`, `{}`, a wrong `schemaVersion`, or a catalog
without `models[]` now fail before any write and leave the file
untouched. Previously valid JSON without a `recommendations` key was
read as "no catalog yet" and overwritten. A real catalog with no
`recommendations` yet is still accepted (no roles to lose; roles are
only added by a person); the shipped catalog always has them.
- `scripts/tests/test_usai_catalog_schema.py`: docstring corrected,
because the hand-written validator now enforces role names and
referential ids.
Blast radius: 4 files.
### Decisions recorded (owner, binding for later slices)
- **D4:** role defaults carry forward. When a recommended model
disappears the refresh **fails** with an actionable message naming each
role and id, writes nothing, and a human re-picks. There is no
priority-list auto-selection (the idea in the stale draft PR 495 is
deliberately not carried).
- D1: the lifecycle ADR (slice S2) will state that repo-time authoring
refresh is out of scope for ADR 0002 and is not a runtime component.
- D2: no USAi API key in this repo's Actions secrets; the later drift
workflow is offline-only and reports BLOCKED for the live check.
- D3: no Prime Agent work now. Nothing Prime Agent was touched; its
drift test is unaffected because `catalog.json` is unchanged.
- D5 to D7 apply to later slices.
From PR 495 this carries only fail-closed referential validation of role
ids against `models[]`. Deterministic key order was already mostly in
`serialize`; this finishes it for `recommendations`.
## Verification
Local run on this branch (head 6ffada1):
| Check | Result |
|---|---|
| New tests against the base script (old fails) | 17 of 22 failed on the
first push; for the review fix, 8 of 9 new non-catalog-output tests
failed on the pre-fix code (the 9th, a valid catalog with no roles,
documents behavior that already held) |
| New tests with the change (new passes) | 31 passed |
| `catalog.json` re-serialization byte-identical | yes (test
"re-serializing the committed catalog is byte-identical") |
| `catalog.json` modified in this PR | no |
| `node scripts/build-catalog.mjs --check` | "catalog.json is up to date
(24 models)" |
| providers/usai suite | 153 passed |
| `make validate` | pass |
| `make test-kits` (opencode 62, usai-provider 18, paseo 65,
providers/usai 153) | all pass |
| `PYTHONPATH=scripts pytest scripts/tests` | 740 passed |
| `ruff check` / `ruff format --check` on `scripts/` | pass |
| `make validate-kits`, `make validate-provider-input-contract`, `make
generate-check` | pass |
Independent read-only AI reviews (blockers only): NO BLOCKERS on the
original change, NO BLOCKERS on the merge of main plus the review fix.
Non-blocking notes: with `--check` the role lines in `describeDrift` are
reachable only through a direct call, because carry-forward copies the
existing roles into the fresh catalog; the non-catalog tests exercise
write mode, and `--check` shares the same code path.
CI results: see the checks on this PR.
## Rollback
Revert the commit. `catalog.json` is unchanged, so no data needs
restoring.
## Security Impact
None to authentication, authorization or attack surface. No credential
handling changed and no secret value appears in the diff. The
carry-forward only reads and writes the catalog path the build already
uses. The fail-closed paths reduce the chance of shipping a catalog with
missing defaults.
Related: #357.
---------
Co-authored-by: OpenCode Agent <[email protected]>
Co-authored-by: OpenCode Agent <[email protected]>1 parent 95be9b9 commit 228cc4b
4 files changed
Lines changed: 409 additions & 3 deletions
File tree
- integrations/providers/usai
- scripts
- tests
- scripts/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
37 | 64 | | |
38 | 65 | | |
39 | 66 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
991 | 991 | | |
992 | 992 | | |
993 | 993 | | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
994 | 1015 | | |
995 | 1016 | | |
996 | 1017 | | |
| |||
1064 | 1085 | | |
1065 | 1086 | | |
1066 | 1087 | | |
1067 | | - | |
| 1088 | + | |
| 1089 | + | |
1068 | 1090 | | |
1069 | 1091 | | |
1070 | 1092 | | |
| |||
1081 | 1103 | | |
1082 | 1104 | | |
1083 | 1105 | | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
1084 | 1109 | | |
1085 | 1110 | | |
1086 | 1111 | | |
| |||
1131 | 1156 | | |
1132 | 1157 | | |
1133 | 1158 | | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
1134 | 1225 | | |
1135 | 1226 | | |
1136 | 1227 | | |
| |||
1191 | 1282 | | |
1192 | 1283 | | |
1193 | 1284 | | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
1194 | 1292 | | |
1195 | 1293 | | |
1196 | 1294 | | |
| |||
1220 | 1318 | | |
1221 | 1319 | | |
1222 | 1320 | | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
1223 | 1330 | | |
1224 | 1331 | | |
1225 | 1332 | | |
| |||
1284 | 1391 | | |
1285 | 1392 | | |
1286 | 1393 | | |
| 1394 | + | |
1287 | 1395 | | |
1288 | 1396 | | |
1289 | 1397 | | |
| |||
0 commit comments