Skip to content

Commit 228cc4b

Browse files
mogulOpenCode AgentOpenCode Agent
authored
fix(usai-catalog): preserve and validate catalog role recommendations (#553)
🤖: AI agent (First Mate), AI-assisted change (OpenCode). Human review required before merge. > ## Merge status > Merged main after #551; no longer stacked. The diff against `main` is now only this PR's own changes: `integrations/providers/usai/{README.md, scripts/build-catalog.mjs, tests/recommendations.test.mjs}` and a docstring correction in `scripts/tests/test_usai_catalog_schema.py`. Merged with a plain `git merge` (no rebase, no force-push). ## Summary Part of #419 (slice S1). Makes the USAi catalog's `recommendations` (role defaults) a validated, preserved, deterministic part of the catalog pipeline, so a catalog refresh can no longer silently delete them. `catalog.json` is unchanged. ## Eligibility & Accountability - [ ] Federal employee or contractor attestation: for the human author of record to complete. - [ ] Author understands the change and can explain it: for the human author of record to complete. - [ ] Right-to-release (CC0-1.0) attestation: for the human author of record to complete. ## AI Assistance Disclosure - [x] AI tools were used: OpenCode (First Mate agent) drafted the change, tests and this description. - [ ] Human verification of AI-generated content: pending reviewer. ## Type of Change - [x] Bug fix - [x] Infrastructure/tooling change ## Context Found in the #419 audit and re-verified on main@7dd2ddc0: - `build-catalog.mjs` neither produced nor preserved `recommendations`. A bootstrap run to a temp file dropped the block, and a fixture refresh did the same. - `recommendations` was missing from `KEY_ORDER`, so `serialize(JSON.parse(catalog.json))` did not reproduce the committed file. - `validateCatalog` did not check role names or that role ids exist in `models[]`. A dangling id plus a made-up role produced 0 errors. - `describeDrift` ignored `recommendations`. - The schema omitted `summary` and `title`. That is fixed by #551, not here. ## Plan (what changed) - `scripts/build-catalog.mjs` - `KEY_ORDER`: `recommendations` after `sources`, roles in fixed order. - `validateCatalog`: role names come from `schema.properties.recommendations.properties`; each value must be a non-empty string naming a model in `models[]`; guards against non-object blocks. - Carry-forward: `readExistingRecommendations` reads the catalog at `--out`; `carryForwardRecommendations` keeps the values and fails closed. Applies to the live/fixture and bootstrap paths and to `--check`. - `describeDrift`: reports `recommendations.<role>: "old" -> "new"`. - `tests/recommendations.test.mjs`: 31 offline tests (fixtures only, temp output paths, no network). - `README.md`: new "Role defaults" section. - Review fix: the file at `--out` must be absent or a real catalog (object, `schemaVersion` `usai-model-catalog/v1`, `models` array). `[]`, `null`, `123`, `"text"`, `{}`, a wrong `schemaVersion`, or a catalog without `models[]` now fail before any write and leave the file untouched. Previously valid JSON without a `recommendations` key was read as "no catalog yet" and overwritten. A real catalog with no `recommendations` yet is still accepted (no roles to lose; roles are only added by a person); the shipped catalog always has them. - `scripts/tests/test_usai_catalog_schema.py`: docstring corrected, because the hand-written validator now enforces role names and referential ids. Blast radius: 4 files. ### Decisions recorded (owner, binding for later slices) - **D4:** role defaults carry forward. When a recommended model disappears the refresh **fails** with an actionable message naming each role and id, writes nothing, and a human re-picks. There is no priority-list auto-selection (the idea in the stale draft PR 495 is deliberately not carried). - D1: the lifecycle ADR (slice S2) will state that repo-time authoring refresh is out of scope for ADR 0002 and is not a runtime component. - D2: no USAi API key in this repo's Actions secrets; the later drift workflow is offline-only and reports BLOCKED for the live check. - D3: no Prime Agent work now. Nothing Prime Agent was touched; its drift test is unaffected because `catalog.json` is unchanged. - D5 to D7 apply to later slices. From PR 495 this carries only fail-closed referential validation of role ids against `models[]`. Deterministic key order was already mostly in `serialize`; this finishes it for `recommendations`. ## Verification Local run on this branch (head 6ffada1): | Check | Result | |---|---| | New tests against the base script (old fails) | 17 of 22 failed on the first push; for the review fix, 8 of 9 new non-catalog-output tests failed on the pre-fix code (the 9th, a valid catalog with no roles, documents behavior that already held) | | New tests with the change (new passes) | 31 passed | | `catalog.json` re-serialization byte-identical | yes (test "re-serializing the committed catalog is byte-identical") | | `catalog.json` modified in this PR | no | | `node scripts/build-catalog.mjs --check` | "catalog.json is up to date (24 models)" | | providers/usai suite | 153 passed | | `make validate` | pass | | `make test-kits` (opencode 62, usai-provider 18, paseo 65, providers/usai 153) | all pass | | `PYTHONPATH=scripts pytest scripts/tests` | 740 passed | | `ruff check` / `ruff format --check` on `scripts/` | pass | | `make validate-kits`, `make validate-provider-input-contract`, `make generate-check` | pass | Independent read-only AI reviews (blockers only): NO BLOCKERS on the original change, NO BLOCKERS on the merge of main plus the review fix. Non-blocking notes: with `--check` the role lines in `describeDrift` are reachable only through a direct call, because carry-forward copies the existing roles into the fresh catalog; the non-catalog tests exercise write mode, and `--check` shares the same code path. CI results: see the checks on this PR. ## Rollback Revert the commit. `catalog.json` is unchanged, so no data needs restoring. ## Security Impact None to authentication, authorization or attack surface. No credential handling changed and no secret value appears in the diff. The carry-forward only reads and writes the catalog path the build already uses. The fail-closed paths reduce the chance of shipping a catalog with missing defaults. Related: #357. --------- Co-authored-by: OpenCode Agent <[email protected]> Co-authored-by: OpenCode Agent <[email protected]>
1 parent 95be9b9 commit 228cc4b

4 files changed

Lines changed: 409 additions & 3 deletions

File tree

‎integrations/providers/usai/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,33 @@ comparing it would fail every offline run regardless of real drift — and a
3434
detector that always fires is one everybody learns to ignore, leaving genuine
3535
drift indistinguishable from the noise.
3636

37+
## Role defaults (`recommendations`)
38+
39+
`recommendations` maps a role (`primary`, `small`, `compaction`, `summary`,
40+
`title`) to a model id. A person chooses these; the feeds do not. So a
41+
regeneration carries the existing values forward from the catalog at `--out`
42+
rather than recomputing them.
43+
44+
- Every value must name a model in `models[]`. A role name outside the schema,
45+
or an id that is not in `models[]`, fails validation.
46+
- If a recommended model is no longer in the regenerated list, the build **fails**
47+
and names each missing role and id. Nothing is written and nothing is
48+
re-picked automatically. Choose a replacement, edit `recommendations` in
49+
`catalog.json`, and re-run.
50+
- The file at `--out` must be absent or a real catalog (a JSON object with
51+
`schemaVersion` `usai-model-catalog/v1` and a `models` array). Anything else,
52+
including invalid JSON, `[]`, `null`, a number, `{}`, or a catalog with no
53+
`models`, fails the build and leaves the file untouched, instead of
54+
overwriting it with a catalog that has no roles.
55+
- A real catalog that has no `recommendations` yet is accepted: there are no
56+
roles to lose, so the new catalog is written without any. Roles are only ever
57+
added by a person editing `recommendations`; the build never invents them.
58+
The shipped catalog always carries them, so deleting the block from it is a
59+
reviewable diff, not a silent build outcome.
60+
- `--check` applies the same rules: a recommended model that left the feed makes
61+
it fail with the message above, and `describeDrift` also names any role whose
62+
value differs between the two catalogs it compares.
63+
3764
## Files
3865

3966
- `catalog.schema.json` — JSON Schema (draft 2020-12), id

‎integrations/providers/usai/scripts/build-catalog.mjs‎

Lines changed: 109 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -991,6 +991,27 @@ function validateCatalog(catalog, schema) {
991991

992992
assert(catalog.schemaVersion === SCHEMA_VERSION, `schemaVersion must be "${SCHEMA_VERSION}"`, errors)
993993

994+
// recommendations (optional): role names come from the schema, and every
995+
// value must name a model that is actually in models[].
996+
if ("recommendations" in catalog) {
997+
const recs = catalog.recommendations
998+
if (!recs || typeof recs !== "object" || Array.isArray(recs)) {
999+
errors.push("recommendations must be an object")
1000+
} else {
1001+
const roles = new Set(Object.keys(schema.properties.recommendations.properties))
1002+
const ids = new Set(Array.isArray(catalog.models) ? catalog.models.map((m) => m?.id) : [])
1003+
for (const [role, modelId] of Object.entries(recs)) {
1004+
if (!roles.has(role)) {
1005+
errors.push(`recommendations: disallowed role "${role}" (allowed: ${[...roles].join(", ")})`)
1006+
} else if (typeof modelId !== "string" || modelId.length === 0) {
1007+
errors.push(`recommendations.${role} must be a non-empty string`)
1008+
} else if (!ids.has(modelId)) {
1009+
errors.push(`recommendations.${role} references "${modelId}", which is not in models[]`)
1010+
}
1011+
}
1012+
}
1013+
}
1014+
9941015
// gateway
9951016
const gw = catalog.gateway
9961017
if (gw && typeof gw === "object") {
@@ -1064,7 +1085,8 @@ function validateCatalog(catalog, schema) {
10641085
// newline. Order is stable so the committed catalog.json is reproducible.
10651086
// -----------------------------------------------------------------------------
10661087
const KEY_ORDER = {
1067-
root: ["schemaVersion", "generatedBy", "sources", "gateway", "vendors", "models"],
1088+
root: ["schemaVersion", "generatedBy", "sources", "recommendations", "gateway", "vendors", "models"],
1089+
recommendations: ["primary", "small", "compaction", "summary", "title"],
10681090
gateway: ["host", "baseUrl", "api", "apiKeyEnv"],
10691091
vendor: ["key", "label", "order", "usaiBackend"],
10701092
model: ["id", "vendor", "name", "contextWindow", "maxOutputTokens", "cost", "costAbove200kContext", "reasoning", "inputModalities"],
@@ -1081,6 +1103,9 @@ function orderObject(obj, order) {
10811103

10821104
function orderCatalog(catalog) {
10831105
const out = orderObject(catalog, KEY_ORDER.root)
1106+
if (out.recommendations && typeof out.recommendations === "object" && !Array.isArray(out.recommendations)) {
1107+
out.recommendations = orderObject(out.recommendations, KEY_ORDER.recommendations)
1108+
}
10841109
if (out.gateway) out.gateway = orderObject(out.gateway, KEY_ORDER.gateway)
10851110
if (Array.isArray(out.vendors)) out.vendors = out.vendors.map((v) => orderObject(v, KEY_ORDER.vendor))
10861111
if (Array.isArray(out.models)) {
@@ -1131,6 +1156,72 @@ function parseArgs(argv) {
11311156
return args
11321157
}
11331158

1159+
/**
1160+
* Role defaults are curated by a human, not derived from the feeds, so a
1161+
* regeneration must carry them forward. Returns the existing catalog's
1162+
* `recommendations` object, or undefined when there is nothing to carry:
1163+
* - the file does not exist (first generation), or
1164+
* - the file is a catalog (right schemaVersion, models[] array) that has no
1165+
* `recommendations` yet; there are no roles to lose.
1166+
* Anything else that exists at the output path is an error. In particular a
1167+
* file that parses as JSON but is not a catalog (`[]`, `null`, `123`, `{}`, a
1168+
* catalog with no models[]) must never be silently overwritten with a catalog
1169+
* that has no roles, because that is the failure this guards against.
1170+
*/
1171+
async function readExistingRecommendations(catalogPath) {
1172+
let raw
1173+
try {
1174+
raw = await readFile(catalogPath, "utf8")
1175+
} catch (error) {
1176+
if (error?.code === "ENOENT") return undefined
1177+
throw error
1178+
}
1179+
const refuse = (why) =>
1180+
new Error(
1181+
`cannot carry recommendations forward: ${catalogPath} ${why}.\n` +
1182+
` Repair or remove it, then re-run. The catalog was NOT written.`,
1183+
)
1184+
let existing
1185+
try {
1186+
existing = JSON.parse(raw)
1187+
} catch {
1188+
throw refuse("exists but is not valid JSON")
1189+
}
1190+
if (!existing || typeof existing !== "object" || Array.isArray(existing)) {
1191+
throw refuse("is valid JSON but not a catalog object")
1192+
}
1193+
if (existing.schemaVersion !== SCHEMA_VERSION) {
1194+
throw refuse(`is not a ${SCHEMA_VERSION} catalog (schemaVersion is ${JSON.stringify(existing.schemaVersion)})`)
1195+
}
1196+
if (!Array.isArray(existing.models)) {
1197+
throw refuse("is a catalog without a models[] array")
1198+
}
1199+
return existing.recommendations
1200+
}
1201+
1202+
/**
1203+
* Keep the existing role defaults. If any recommended model is no longer in
1204+
* the regenerated model list, fail closed: a human must pick the replacement.
1205+
* No role is ever re-chosen automatically.
1206+
*/
1207+
function carryForwardRecommendations(recommendations, models, where = "catalog.json") {
1208+
if (recommendations === undefined) return undefined
1209+
if (!recommendations || typeof recommendations !== "object" || Array.isArray(recommendations)) {
1210+
throw new Error(`recommendations in ${where} must be an object. The catalog was NOT written.`)
1211+
}
1212+
const ids = new Set(models.map((m) => m.id))
1213+
const gone = Object.entries(recommendations).filter(([, id]) => !ids.has(id))
1214+
if (gone.length) {
1215+
throw new Error(
1216+
`recommended model(s) are no longer in the model list:\n` +
1217+
gone.map(([role, id]) => ` - ${role}: ${id}`).join("\n") +
1218+
`\n Choose a replacement for each role, set it under "recommendations" in ${where}, ` +
1219+
`and re-run. The catalog was NOT written.`,
1220+
)
1221+
}
1222+
return { ...recommendations }
1223+
}
1224+
11341225
async function buildCatalog(args) {
11351226
const schemaPath = args.opts["--schema"] || DEFAULT_SCHEMA
11361227
const schema = JSON.parse(await readFile(schemaPath, "utf8"))
@@ -1191,6 +1282,13 @@ async function buildCatalog(args) {
11911282
}
11921283
}
11931284

1285+
const carried = carryForwardRecommendations(
1286+
await readExistingRecommendations(args.opts["--out"] || DEFAULT_OUT),
1287+
catalog.models,
1288+
args.opts["--out"] || DEFAULT_OUT,
1289+
)
1290+
if (carried !== undefined) catalog.recommendations = carried
1291+
11941292
const errors = validateCatalog(catalog, schema)
11951293
if (errors.length) {
11961294
throw new Error(`catalog failed schema validation:\n - ${errors.join("\n - ")}`)
@@ -1220,6 +1318,15 @@ function describeDrift(existing, fresh) {
12201318
lines.push(`${key}: ${JSON.stringify(existing?.[key])} -> ${JSON.stringify(fresh?.[key])}`)
12211319
}
12221320
}
1321+
const rolesBefore = existing?.recommendations ?? {}
1322+
const rolesAfter = fresh?.recommendations ?? {}
1323+
for (const role of [...new Set([...Object.keys(rolesBefore), ...Object.keys(rolesAfter)])].sort()) {
1324+
if (rolesBefore[role] !== rolesAfter[role]) {
1325+
lines.push(
1326+
`recommendations.${role}: ${JSON.stringify(rolesBefore[role] ?? "(none)")} -> ${JSON.stringify(rolesAfter[role] ?? "(none)")}`,
1327+
)
1328+
}
1329+
}
12231330
if (JSON.stringify(existing?.gateway) !== JSON.stringify(fresh?.gateway)) {
12241331
lines.push("gateway block changed")
12251332
}
@@ -1284,6 +1391,7 @@ export {
12841391
serialize,
12851392
substantive,
12861393
describeDrift,
1394+
carryForwardRecommendations,
12871395
parseBootstrapModels,
12881396
bootstrapModelsFromBlock,
12891397
shapeFromFeeds,

0 commit comments

Comments
 (0)