Skip to content

Commit fcd94e5

Browse files
mogulOpenCode Agent
andauthored
feat(usai-catalog): carry verified capability metadata through a catalog refresh (#578)
🤖: AI agent (First Mate), AI-assisted change (OpenCode). Human review required before merge. ## Summary A catalog refresh no longer erases verified `reasoning` / `inputModalities` values. - Verified capability values are curated overlay data keyed by model id, read from the committed catalog at `--out` through the existing shape-validated reader, exactly like `recommendations`. - A model still in the feed keeps its values. A model that leaves the feed takes them with it; the build names model and fields on stderr and `--check` reports `capability values dropped with removed model: <id> (<fields>)`. - `sources.capabilitiesVerified` (the release reminder, not evidence) is carried unchanged while any value is carried, never invented or advanced, and dropped when no value remains (tested). - Values are only trusted as verified when the existing catalog has a well-formed reminder. Without one they are not carried (warning) and `--check` reports them as dropped. - `--check` / `describeDrift` now report `capability added|changed|dropped: <id>.<field>` per model and field, for the catalog and the neutral snapshot. A clean tree with carried values reports "up to date". - The neutral snapshot already carried the fields through (normalizer, after the earlier schema work); this PR adds the test. - README gets a new section; ADR 0003 gets an additive clarification (status unchanged). ### Decision: `--include-capabilities` (requirement 2) Chosen, and the SAFER option: **while any verified value is carried, the flag is refused (before any write, non-zero exit) if its upstream values would (a) differ from a verified value, or (b) add a value for a model/field that has no verified value.** Identical values are fine. The message lists model, field, verified value and upstream value. With no verified value present, the flag behaves exactly as today (unverified, labelled `sources.capabilities`). Why not "allow it for models without a verified value": the catalog has one catalog-wide `sources.capabilitiesVerified` string. Unverified upstream values beside verified ones under that string would make the reminder vouch for data it does not cover, which is the hazard ADR 0006 warns about. `sources.capabilities` (unverified label) and `capabilitiesVerified` never appear together. To inspect upstream values, build to a scratch `--out` without verified values. ## Issues #576, #419, #575 ## Before / After | | Before | After | |---|---|---| | Plain refresh over a catalog with verified values | values dropped; `--check` "model changed" | values and reminder carried; `--check` up to date | | `--include-capabilities` over verified values | silently replaced by unverified upstream | refused, nothing written, conflict listed | | Model leaves feed | n/a | values dropped, named, reported as drift | | Capability drift lines | `model changed: <id>` | `capability added/changed/dropped: <id>.<field>` (+ `model changed` only for non-capability fields) | Note: `describeDrift` output for a capability-only change changed from `model changed: <id>` to the more specific capability lines; one existing assertion in `capabilities.test.mjs` was updated accordingly. ## Verification - New `tests/capability-carry.test.mjs`: 16 tests, offline fixtures. Run against `origin/main`'s `build-catalog.mjs`: 13 of the 16 fail (the 3 that pass on main are invariants: idempotence, capability-free stays capability-free, and the flag unchanged when no verified value exists). With the change: all pass. - `node --test` in providers/usai: 210 pass, 0 fail. `make test-kits`: 62 + 18 + 87 + 210 pass. `pytest`: 827 passed. `ruff check` / `ruff format --check scripts/`: clean. `make validate`, `make generate-check`: pass. markdownlint (CI runner): clean for tracked files. - Adversarial read-only review (sub-agent): NO BLOCKERS; mutation checks on 5 of 6 mutants caught by tests. Its two non-blocking notes were addressed (malformed-reminder test added; ADR wording corrected). - **Cross-check against #575's branch** (`data/541-usai-image-modalities`, scratch worktree, this PR's diff applied on top): - `node integrations/providers/usai/scripts/refresh.mjs --from integrations/providers/usai/tests/fixtures/refresh --check` → `catalog: up to date`, `neutral-snapshot: up to date`, exit 0 (on 575 alone, without this PR, it exits 1 with `model changed` for the 12 models). - Write-mode refresh in the scratch copy: exit 0; all 12 `inputModalities` values, the full `models[]` array, and `sources.capabilitiesVerified` ("2026-11-04; opencode 2.0.30") byte-identical; the neutral snapshot "unchanged". Only `sources` provenance changed (`bootstrappedFrom` → the fixture paths the run read), which is expected and not compared by `--check`. ## Merge Guard Independent of #575 for merge order; the cross-check above assumes it. No change to `catalog.json` or the snapshot bytes here. README wording is written to merge cleanly either way, but if 575 lands first, adjust its sentence "Rebuilding the catalog from the feeds does not carry them forward, so a refresh that rewrites `catalog.json` must keep them or re-verify them" (it becomes false) to say a refresh carries verified values forward; this PR's new README section describes the behavior. No workflow files, no Prime Agent changes, no secrets (offline only). ## Rollback Revert this PR. The catalog and snapshot are unchanged by it, so nothing needs regenerating. ## Security Impact No new network access, credentials, or workflow changes. The change makes a refresh fail closed instead of overwriting verified data with unverified upstream data, and treats values without a valid release reminder as unverified. Human review is required for the drift/refusal logic. --------- Co-authored-by: OpenCode Agent <[email protected]>
1 parent 89a66b2 commit fcd94e5

5 files changed

Lines changed: 537 additions & 8 deletions

File tree

‎docs/decisions/0003-repo-time-provider-catalog-refresh.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,13 @@ documented command, and never produce or read harness-native configuration.
6969
(`recommendations`) and model order come from the existing catalog. If a
7070
recommended model has left the feed, the refresh fails and names each role; a
7171
person chooses the replacement.
72+
Verified `reasoning` and `inputModalities` values are curated in the same way:
73+
they are carried forward by model id (while the catalog has a valid
74+
`sources.capabilitiesVerified`), dropped only when their model leaves the
75+
feed (reported as drift), and never overwritten by the unverified
76+
`--include-capabilities` copy, which is refused while a verified value is
77+
carried. `sources.capabilitiesVerified` is carried, never advanced. Details:
78+
`integrations/providers/usai/README.md`.
7279

7380
## Scope
7481

‎integrations/providers/usai/README.md‎

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,15 +42,16 @@ What happens:
4242

4343
1. Both feeds are fetched **once** (https only, bounded time and size) into
4444
`tmp/usai-source/`, which is git-ignored.
45-
2. `catalog.json` is built from that pull. Role defaults and model order are
46-
carried forward from the existing catalog.
45+
2. `catalog.json` is built from that pull. Role defaults, model order and
46+
verified capability values are carried forward from the existing catalog.
4747
3. Every artifact in [`refresh-registry.json`](refresh-registry.json) is
4848
regenerated from the new catalog by its file-to-file generator.
4949
4. Only if every step succeeded are the files written. Nothing is committed.
5050
Review `git diff`, then commit as a normal change.
5151

5252
Check mode compares in a temporary directory and exits `1` on drift, naming the
53-
artifact and the models that were added, removed or changed. `sources` is not
53+
artifact and the models that were added, removed or changed (and, for capability
54+
values, the model and field that were added, changed or dropped). `sources` is not
5455
compared, because it records where a run read from, not what the catalog says.
5556

5657
### Offline and deterministic
@@ -127,6 +128,37 @@ rather than recomputing them.
127128
it fail with the message above, and `describeDrift` also names any role whose
128129
value differs between the two catalogs it compares.
129130

131+
## Verified capability values across a refresh
132+
133+
Verified `reasoning` and `inputModalities` values are curated data, like
134+
`recommendations`: the feeds never supply them, a person commits them with
135+
per-model evidence, and a regeneration carries them forward from the catalog at
136+
`--out`, keyed by model id. The neutral snapshot is regenerated from the result,
137+
so it carries them too.
138+
139+
- A model that is still in the feed keeps its values. A model that leaves the
140+
feed takes its values with it: the build names the model and fields on
141+
stderr, and `--check` reports them as drift (`capability values dropped with
142+
removed model: <id> (<fields>)`).
143+
- `sources.capabilitiesVerified` is the release reminder, not evidence. A refresh
144+
carries it unchanged while any verified value is carried, never writes or
145+
advances it, and drops it when no verified value remains.
146+
- Values are treated as verified only when the existing catalog has a valid
147+
`sources.capabilitiesVerified`. Without one they are not carried (a warning
148+
names how many) and `--check` reports them as dropped.
149+
- `--include-capabilities` is refused, before any write, whenever a verified
150+
value is carried and the upstream copy would either differ from it or add a
151+
value for a field that has no verified value. The message lists each model and
152+
field with the verified and upstream values. Mixing unverified values under a
153+
string that vouches for verified ones is the hazard; to compare upstream values,
154+
build to a scratch `--out` that has no verified values. With no verified value
155+
present the flag behaves as before and labels its output unverified.
156+
- `--check` reports capability values per model and field as `capability added`,
157+
`capability changed` or `capability dropped`, for the catalog and the snapshot.
158+
159+
To change a verified value, re-verify it and edit `catalog.json` by hand with its
160+
evidence file; the refresh never edits one.
161+
130162
## Files
131163

132164
- `catalog.schema.json` — JSON Schema (draft 2020-12), id

‎integrations/providers/usai/scripts/build-catalog.mjs‎

Lines changed: 120 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@
3535
// reminder, not that evidence. With the flag, --check
3636
// against a capability-free catalog reports drift by
3737
// design.
38+
// Verified capability values already in the catalog at
39+
// --out are carried forward, and the flag is refused
40+
// (nothing written) if its upstream values would
41+
// conflict with them or sit beside them unverified.
3842
//
3943
// There is no bootstrap mode. The catalog used to be seeded from a frozen
4044
// copy of an old OpenCode config; that made a harness-native file a source of
@@ -1128,6 +1132,100 @@ function carryForwardRecommendations(recommendations, models, where = "catalog.j
11281132
return { ...recommendations }
11291133
}
11301134

1135+
const UNVERIFIED_LABEL = "models.dev upstream provider data; not verified against the USAi gateway"
1136+
const CAPABILITY_FIELDS = ["reasoning", "inputModalities"]
1137+
1138+
// Shape of sources.capabilitiesVerified: a release reminder, not evidence (ADR
1139+
// 0006). A carried-forward value is only trusted as verified when it is present.
1140+
const RELEASE_REMINDER = /^\d{4}-\d{2}-\d{2}; opencode v?\d+\.\d+\.\d+\S*$/
1141+
1142+
const sameValue = (a, b) => JSON.stringify(a) === JSON.stringify(b)
1143+
1144+
/**
1145+
* Verified capability values, keyed by model id, read from the already
1146+
* shape-validated existing catalog. They are curated overlay data exactly like
1147+
* `recommendations`: the feeds never supply them, a person commits them with
1148+
* per-model evidence, and a regeneration carries them forward.
1149+
*
1150+
* Values with no valid sources.capabilitiesVerified release reminder are not
1151+
* treated as verified (for example the output of an earlier
1152+
* --include-capabilities run): they are not carried, and a warning names how
1153+
* many were left behind. --check then reports them as dropped.
1154+
*/
1155+
function readVerifiedOverlay(existing, where) {
1156+
const reminder = existing?.sources?.capabilitiesVerified
1157+
const reminderOk = typeof reminder === "string" && RELEASE_REMINDER.test(reminder)
1158+
const overlay = new Map()
1159+
for (const model of existing?.models ?? []) {
1160+
const values = {}
1161+
for (const field of CAPABILITY_FIELDS) if (field in model) values[field] = model[field]
1162+
if (Object.keys(values).length > 0) overlay.set(model.id, values)
1163+
}
1164+
if (!reminderOk) {
1165+
if (overlay.size > 0) {
1166+
process.stderr.write(
1167+
` WARNING ${where}: capability values for ${overlay.size} model(s) have no valid ` +
1168+
`sources.capabilitiesVerified ("<YYYY-MM-DD>; opencode <version>"); not treated as verified, not carried\n`,
1169+
)
1170+
}
1171+
return { overlay: new Map(), reminder: undefined }
1172+
}
1173+
return { overlay, reminder }
1174+
}
1175+
1176+
const show = (value) => (value === undefined ? "(none)" : JSON.stringify(value))
1177+
1178+
/**
1179+
* Carry verified capability values onto the regenerated models. A model that
1180+
* left the feed takes its values with it (reported on stderr; --check reports it
1181+
* as drift). Returns the reminder to keep, or undefined when no value remains.
1182+
*
1183+
* With --include-capabilities the unverified upstream values are already on
1184+
* `models`. While any verified value is carried, an upstream value is accepted
1185+
* only if it equals the verified value for the same model and field. A
1186+
* different value is a conflict, and a value for a field that has no verified
1187+
* value would sit unverified under a sources.capabilitiesVerified string that
1188+
* vouches for verified ones; both are refused before anything is written.
1189+
*/
1190+
function applyVerifiedOverlay(models, { overlay, reminder }, { includeCapabilities, where }) {
1191+
const byId = new Map(models.map((m) => [m.id, m]))
1192+
const gone = [...overlay].filter(([id]) => !byId.has(id))
1193+
if (gone.length > 0) {
1194+
process.stderr.write(
1195+
`capability values dropped for model(s) no longer in the feed:\n` +
1196+
gone.map(([id, values]) => ` - ${id}: ${Object.keys(values).join(", ")}`).join("\n") + "\n",
1197+
)
1198+
}
1199+
const carried = [...overlay].filter(([id]) => byId.has(id))
1200+
if (includeCapabilities && carried.length > 0) {
1201+
const lines = []
1202+
for (const model of models) {
1203+
for (const field of CAPABILITY_FIELDS) {
1204+
if (!(field in model)) continue
1205+
const verified = overlay.get(model.id)?.[field]
1206+
if (verified === undefined) {
1207+
lines.push(` - ${model.id} ${field}: no verified value, upstream ${show(model[field])} (would be unverified)`)
1208+
} else if (!sameValue(verified, model[field])) {
1209+
lines.push(` - ${model.id} ${field}: verified ${show(verified)}, upstream ${show(model[field])} (CONFLICT)`)
1210+
}
1211+
}
1212+
}
1213+
if (lines.length > 0) {
1214+
throw new Error(
1215+
`--include-capabilities refused: ${where} carries verified capability values, and the upstream copy ` +
1216+
`would conflict with them or sit beside them unverified:\n${lines.join("\n")}\n` +
1217+
` To inspect upstream values, run against a scratch --out with no verified values; to change a ` +
1218+
`verified value, re-verify it and edit the catalog by hand. The catalog was NOT written.`,
1219+
)
1220+
}
1221+
}
1222+
for (const [id, values] of carried) Object.assign(byId.get(id), values)
1223+
if (carried.length === 0 && reminder !== undefined) {
1224+
process.stderr.write("no verified capability values remain; sources.capabilitiesVerified is dropped\n")
1225+
}
1226+
return carried.length > 0 ? reminder : undefined
1227+
}
1228+
11311229
const NO_SOURCE_MESSAGE =
11321230
"no model source given: pass --models-url <path|https url>.\n" +
11331231
" This builder has no default source and no bootstrap mode, so it cannot silently\n" +
@@ -1179,7 +1277,6 @@ async function buildCatalog(args) {
11791277
sources: {
11801278
modelsList: labels.modelsList ?? modelsUrl,
11811279
enrichment: noEnrichment ? "(none)" : (labels.enrichment ?? modelsDevRef),
1182-
...(includeCapabilities ? { capabilities: "models.dev upstream provider data; not verified against the USAi gateway" } : {}),
11831280
},
11841281
gateway: buildGateway(),
11851282
vendors: buildVendors(vendorOrder),
@@ -1189,6 +1286,13 @@ async function buildCatalog(args) {
11891286
const carried = carryForwardRecommendations(existing?.recommendations, catalog.models, carryPath)
11901287
if (carried !== undefined) catalog.recommendations = carried
11911288

1289+
const reminder = applyVerifiedOverlay(catalog.models, readVerifiedOverlay(existing, carryPath), {
1290+
includeCapabilities,
1291+
where: carryPath,
1292+
})
1293+
if (reminder !== undefined) catalog.sources.capabilitiesVerified = reminder
1294+
else if (includeCapabilities) catalog.sources.capabilities = UNVERIFIED_LABEL
1295+
11921296
const errors = validateCatalog(catalog, schema)
11931297
if (errors.length) {
11941298
throw new Error(`catalog failed schema validation:\n - ${errors.join("\n - ")}`)
@@ -1244,10 +1348,23 @@ function describeDrift(existing, fresh) {
12441348
const removed = [...before.keys()].filter((id) => !after.has(id)).sort()
12451349
if (added.length) lines.push(`models added: ${added.join(", ")}`)
12461350
if (removed.length) lines.push(`models removed: ${removed.join(", ")}`)
1351+
for (const id of removed) {
1352+
const fields = CAPABILITY_FIELDS.filter((field) => field in before.get(id))
1353+
if (fields.length) lines.push(`capability values dropped with removed model: ${id} (${fields.join(", ")})`)
1354+
}
12471355
for (const id of [...after.keys()].filter((i) => before.has(i)).sort()) {
1248-
if (JSON.stringify(before.get(id)) !== JSON.stringify(after.get(id))) {
1249-
lines.push(`model changed: ${id}`)
1356+
const was = before.get(id)
1357+
const now = after.get(id)
1358+
const capabilityLines = []
1359+
for (const field of CAPABILITY_FIELDS) {
1360+
if (sameValue(was[field], now[field])) continue
1361+
if (!(field in was)) capabilityLines.push(`capability added: ${id}.${field} = ${show(now[field])}`)
1362+
else if (!(field in now)) capabilityLines.push(`capability dropped: ${id}.${field} (was ${show(was[field])})`)
1363+
else capabilityLines.push(`capability changed: ${id}.${field}: ${show(was[field])} -> ${show(now[field])}`)
12501364
}
1365+
const strip = (model) => Object.fromEntries(Object.entries(model).filter(([key]) => !CAPABILITY_FIELDS.includes(key)))
1366+
if (JSON.stringify(strip(was)) !== JSON.stringify(strip(now))) lines.push(`model changed: ${id}`)
1367+
lines.push(...capabilityLines)
12511368
}
12521369
return lines
12531370
}

‎integrations/providers/usai/tests/capabilities.test.mjs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -222,8 +222,11 @@ test("a model fed through models.dev carries reasoning and modalities in canonic
222222
test("capability changes are reported as drift", () => {
223223
const before = { models: [{ id: "m", name: "M" }] }
224224
const known = { models: [{ id: "m", name: "M", inputModalities: ["text", "image"] }] }
225-
assert.deepEqual(describeDrift(before, known), ["model changed: m"])
226-
assert.deepEqual(describeDrift(known, { models: [{ id: "m", name: "M", inputModalities: ["text"] }] }), ["model changed: m"])
225+
assert.deepEqual(describeDrift(before, known), ['capability added: m.inputModalities = ["text","image"]'])
226+
assert.deepEqual(describeDrift(known, { models: [{ id: "m", name: "M", inputModalities: ["text"] }] }), [
227+
'capability changed: m.inputModalities: ["text","image"] -> ["text"]',
228+
])
229+
assert.deepEqual(describeDrift(known, before), ['capability dropped: m.inputModalities (was ["text","image"])'])
227230
})
228231

229232
test("the modality allowlist is identical in both schemas and the builder", () => {

0 commit comments

Comments
 (0)