You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fcd94e5
Browse filesBrowse the repository at this point in the historyBrowse files
feat(usai-catalog): carry verified capability metadata through a catalog refresh (#578)
🤖: AI agent (First Mate), AI-assisted change (OpenCode). Human review
required before merge.
## Summary
A catalog refresh no longer erases verified `reasoning` /
`inputModalities` values.
- Verified capability values are curated overlay data keyed by model id,
read from the committed catalog at `--out` through the existing
shape-validated reader, exactly like `recommendations`.
- A model still in the feed keeps its values. A model that leaves the
feed takes them with it; the build names model and fields on stderr and
`--check` reports `capability values dropped with removed model: <id>
(<fields>)`.
- `sources.capabilitiesVerified` (the release reminder, not evidence) is
carried unchanged while any value is carried, never invented or
advanced, and dropped when no value remains (tested).
- Values are only trusted as verified when the existing catalog has a
well-formed reminder. Without one they are not carried (warning) and
`--check` reports them as dropped.
- `--check` / `describeDrift` now report `capability
added|changed|dropped: <id>.<field>` per model and field, for the
catalog and the neutral snapshot. A clean tree with carried values
reports "up to date".
- The neutral snapshot already carried the fields through (normalizer,
after the earlier schema work); this PR adds the test.
- README gets a new section; ADR 0003 gets an additive clarification
(status unchanged).
### Decision: `--include-capabilities` (requirement 2)
Chosen, and the SAFER option: **while any verified value is carried, the
flag is refused (before any write, non-zero exit) if its upstream values
would (a) differ from a verified value, or (b) add a value for a
model/field that has no verified value.** Identical values are fine. The
message lists model, field, verified value and upstream value. With no
verified value present, the flag behaves exactly as today (unverified,
labelled `sources.capabilities`).
Why not "allow it for models without a verified value": the catalog has
one catalog-wide `sources.capabilitiesVerified` string. Unverified
upstream values beside verified ones under that string would make the
reminder vouch for data it does not cover, which is the hazard ADR 0006
warns about. `sources.capabilities` (unverified label) and
`capabilitiesVerified` never appear together. To inspect upstream
values, build to a scratch `--out` without verified values.
## Issues
#576,
#419, #575
## Before / After
| | Before | After |
|---|---|---|
| Plain refresh over a catalog with verified values | values dropped;
`--check` "model changed" | values and reminder carried; `--check` up to
date |
| `--include-capabilities` over verified values | silently replaced by
unverified upstream | refused, nothing written, conflict listed |
| Model leaves feed | n/a | values dropped, named, reported as drift |
| Capability drift lines | `model changed: <id>` | `capability
added/changed/dropped: <id>.<field>` (+ `model changed` only for
non-capability fields) |
Note: `describeDrift` output for a capability-only change changed from
`model changed: <id>` to the more specific capability lines; one
existing assertion in `capabilities.test.mjs` was updated accordingly.
## Verification
- New `tests/capability-carry.test.mjs`: 16 tests, offline fixtures. Run
against `origin/main`'s `build-catalog.mjs`: 13 of the 16 fail (the 3
that pass on main are invariants: idempotence, capability-free stays
capability-free, and the flag unchanged when no verified value exists).
With the change: all pass.
- `node --test` in providers/usai: 210 pass, 0 fail. `make test-kits`:
62 + 18 + 87 + 210 pass. `pytest`: 827 passed. `ruff check` / `ruff
format --check scripts/`: clean. `make validate`, `make generate-check`:
pass. markdownlint (CI runner): clean for tracked files.
- Adversarial read-only review (sub-agent): NO BLOCKERS; mutation checks
on 5 of 6 mutants caught by tests. Its two non-blocking notes were
addressed (malformed-reminder test added; ADR wording corrected).
- **Cross-check against #575's branch**
(`data/541-usai-image-modalities`, scratch worktree, this PR's diff
applied on top):
- `node integrations/providers/usai/scripts/refresh.mjs --from
integrations/providers/usai/tests/fixtures/refresh --check` → `catalog:
up to date`, `neutral-snapshot: up to date`, exit 0 (on 575 alone,
without this PR, it exits 1 with `model changed` for the 12 models).
- Write-mode refresh in the scratch copy: exit 0; all 12
`inputModalities` values, the full `models[]` array, and
`sources.capabilitiesVerified` ("2026-11-04; opencode 2.0.30")
byte-identical; the neutral snapshot "unchanged". Only `sources`
provenance changed (`bootstrappedFrom` → the fixture paths the run
read), which is expected and not compared by `--check`.
## Merge Guard
Independent of #575 for merge order; the
cross-check above assumes it. No change to `catalog.json` or the
snapshot bytes here. README wording is written to merge cleanly either
way, but if 575 lands first, adjust its sentence "Rebuilding the catalog
from the feeds does not carry them forward, so a refresh that rewrites
`catalog.json` must keep them or re-verify them" (it becomes false) to
say a refresh carries verified values forward; this PR's new README
section describes the behavior. No workflow files, no Prime Agent
changes, no secrets (offline only).
## Rollback
Revert this PR. The catalog and snapshot are unchanged by it, so nothing
needs regenerating.
## Security Impact
No new network access, credentials, or workflow changes. The change
makes a refresh fail closed instead of overwriting verified data with
unverified upstream data, and treats values without a valid release
reminder as unverified. Human review is required for the drift/refusal
logic.
---------
Co-authored-by: OpenCode Agent <[email protected]>
0 commit comments