Repository navigation
Commit 907d486
feat(usai-catalog): publish verified image input modality for 12 USAi models (#575)
🤖: AI agent (First Mate), AI-assisted change (OpenCode). Human review
required before merge.
> PR #578 merged; verified values are
preserved by refresh (checked on the merged tree on 2026-11-04, head
9a1c5f6).
## Summary
Publishes the verified image input modality for 12 USAi models:
`inputModalities: ["text","image"]` in
`integrations/providers/usai/catalog.json`, the regenerated neutral
snapshot, and `sources.capabilitiesVerified: "2026-11-04; opencode
2.0.30"`. Adds the per-model evidence file required by ADR 0006
(isolation). Nothing else is published: no `reasoning` for any model,
and the other 12 models stay absent (unknown).
Base: origin/main at fcd94e5 (includes the verified-capability carry PR
#578, and also the verification script PR
#572 and the refresh entrypoint PR
#573, both merged; this branch has been
merged with main).
## Issues
- #541 (this change)
- #366 (parent)
- #540 (the verification procedure and
script)
## What is published (every model, field, value)
Field `inputModalities`, value `["text","image"]`, for exactly these 12
models:
`claude-sonnet-5.5`, `claude-sonnet-5`, `claude_4_8_opus`,
`claude_4_7_opus`, `claude_4_6_sonnet`,
`gpt-6.1-sol-latest-guardrails-defaultv2`, `gpt-6-astra`,
`gpt_5_4_default_v2`, `gpt_5_2_default_v2`, `gemini-3.8-flash`,
`gemini-3-7-flash`, `gemini-3.5-flash`.
(The list was checked against `summary.qualifying` in the raw
`results.json`: identical.)
Evidence (full table and verbatim token replies in
`integrations/isolation/acq-kits/opencode/docs/capability-evidence/2026-11-04-opencode-2.0.30.md`):
- **Procedure:** synthetic-image procedure, live run through OpenCode
2.0.30 against the USAi gateway, on 2026-11-04.
- **Samples:** 3 independently generated random-token images per model,
per configuration. Configurations: `modalities.input` unset; explicit
`["text","image"]`; explicit `["text"]` (negative control).
- **Result for each of the 12:** exact token on 3 of 3 images with
explicit `["text","image"]`. Text control passed.
- **Control:** under explicit `["text"]`, 63 of 63 samples (21 models x
3) refused or did not return the token; 0 token leaks.
- **`claude_4_7_opus`:** read 2 of 3 with the setting unset, 3 of 3
explicit. The decision rule counts the explicit configuration, so it
qualifies. Called out in the evidence file.
- **Script:** `verify_image_capability.py` v1.0.0. The evidence was
produced by the pre-merge head `fa813f63` (with its live-run fixes), in
the maintainer's sandbox. The script has since merged as `f0a951f3`; the
merged version differs from `fa813f63`, so the evidence was **not**
produced by the merged script. The evidence file says so.
- Models tested but not published (9, misreads in at least one explicit
text+image sample) and the 3 unavailable models (HTTP 403
`model_blocked_globally` x2; HTTP 400 no tool use in streaming mode) are
listed with reasons in the evidence file, so a transient outage is not
read as "unsupported".
## Narrowing (reviewer must accept)
ADR 0006: a present `inputModalities` array asserts the listed
modalities only, and the OpenCode mapping treats an unlisted modality as
disabled. Publishing `["text","image"]` for these 12 models **disables
`audio`, `video`, and `pdf` input for them in OpenCode**, though those
were never probed and may work at the gateway. **Please explicitly
accept this narrowing for each of the 12 models**, or ask for the field
to stay absent for any of them.
## Before / After
| | Before | After |
|---|---|---|
| `catalog.json` models with `inputModalities` | 0 | 12 |
| `catalog.json` `sources.capabilitiesVerified` | absent | `2026-11-04;
opencode 2.0.30` |
| Neutral snapshot models with `inputModalities` | 0 | 12 (regenerated
by `normalize-catalog.mjs`, equal to the normalizer output) |
| `reasoning` on any model | none | none |
| Rendered OpenCode config | no `modalities` | `modalities.input:
["text","image"]` for the 12 (checked with `renderProviderConfig`) |
Files: `catalog.json` (+12 models, +1 source key), the snapshot (same),
the evidence file (new), `tests/catalog-check.test.mjs` (one added
assertion, see below), and `integrations/providers/usai/README.md` (a
link to the evidence file; the earlier sentence that a refresh does not
carry values forward was removed because #578 documents the
carry-forward).
Evidence files added beside the evidence file (all in
`capability-evidence/`): `2026-11-04-opencode-2.0.30-replies.md` (step-1
config facts for all 24 models, text-control result, and the verbatim
reply for every one of the 189 samples, including all 63 text-only
control replies, capped at the stored 500 characters), and the run's
sanitized `results.json` and `results.tsv` copied unchanged, with
SHA-256 recorded. Run id `70f8e8da5746`, script commit `fa813f63`
(merged version `f0a951f3`), OpenCode 2.0.30. The run console log is
omitted because it adds nothing beyond `results.json`.
**Provenance:** the positive-token table is author-reported evidence
from a run the author executed; it is not independently verified. A
human must accept the audio/video/pdf narrowing per model.
### Test change after merging main
The earlier edit to `default --check uses the committed 24-model
production catalog` is dropped. The refresh entrypoint PR removed the
bootstrap fixture and bootstrap mode, so that test no longer exists on
main; `--check` without a source now refuses. The conflict was resolved
by taking main's version of `catalog-check.test.mjs` and adding one
assertion to `the committed catalog is a valid catalog with at least one
model`: the committed catalog still has exactly 24 models. Nothing else
in that file differs from main.
## Verification
| Check | Result |
|---|---|
| After merging main (`9a1c5f6`) | `make test-kits`: opencode 69,
usai-provider 18, paseo 87, providers/usai 210: all pass |
| `make validate`, `make validate-kits`, `make
validate-provider-input-contract`, `make generate-check` | pass |
| `PYTHONPATH=scripts pytest scripts/tests` | 827 passed |
| `ruff check scripts`, `ruff format --check scripts` | pass |
| `.github/linters/run-markdownlint.mjs` | pass |
| Guard test for `sources.capabilitiesVerified` shape | pass (in
providers/usai suite) |
| Byte identity | `serialize(JSON.parse(catalog.json))` equals the
committed bytes; snapshot equals fresh `normalize-catalog.mjs` output
(`cmp`) |
| Older consumers | the OpenCode renderer accepts the snapshot with
capabilities present and renders `modalities.input` for exactly 12
models; renderer tests pass |
| Prime Agent | `models-json-regen.test.mjs` still passes; the Prime
Agent emitter does not carry these fields; **no Prime Agent file
changed** |
| Independent read-only AI review (blockers only) | NO BLOCKERS. Checked
the 12 ids, every evidence row against `results.json`, canonical form,
no Prime Agent change. One non-blocking comment wording fix applied in
the second commit. |
The data was written with the repo's own `serialize` from
`build-catalog.mjs` (not hand edits and not `--include-capabilities`,
which copies unverified upstream values).
## Before merge
Confirm per ADR 0006 rollout order that consumers validating with a
vendored copy of the neutral schema (Quickstart fixtures, any Prime
Agent consumer) have adopted the schema that allows these fields.
**Refresh interaction (verified on the merged tree, 2026-11-04):**
#578 has merged, so a refresh carries
these values forward (#576). On this
branch merged with main (`fcd94e5e`):
- `refresh.mjs --from tests/fixtures/refresh --check`: catalog and
neutral snapshot "up to date", exit 0.
- Write-mode refresh in a scratch copy: exit 0; all 12
`inputModalities`, the whole `models[]` array and
`sources.capabilitiesVerified` byte-identical (only `sources` provenance
paths differ, as expected for a fixture source); snapshot unchanged.
- `build-catalog.mjs ... --include-capabilities` against the verified
catalog: refused with exit 1, catalog not written.
This PR changes no refresh behavior. `catalog.json` and the snapshot
bytes are unchanged by the merge.
## Rollback
Revert the commit. Models return to "unknown"; OpenCode 2.0.30 then
defaults to text and image for them, so behavior on the pinned version
is unchanged either way.
## Security Impact
No change to authentication, authorization, data handling, or network
surface. Data only. The evidence file holds synthetic random tokens and
verbatim model replies; no credentials, no raw transcripts. No secret
values were used in any request recorded here.
Related: #357.
---------
Co-authored-by: OpenCode Agent <[email protected]>1 parent 1b24d7e commit 907d486
8 files changed
Lines changed: 3076 additions & 27 deletions
File tree
- integrations
- isolation/acq-kits
- opencode/docs/capability-evidence
- usai-provider/files/home/usai-config
- providers/usai
- tests
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
16 | 20 | | |
17 | 21 | | |
18 | 22 | | |
| |||
36 | 40 | | |
37 | 41 | | |
38 | 42 | | |
39 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
40 | 48 | | |
41 | 49 | | |
42 | 50 | | |
| |||
48 | 56 | | |
49 | 57 | | |
50 | 58 | | |
51 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
52 | 64 | | |
53 | 65 | | |
54 | 66 | | |
| |||
60 | 72 | | |
61 | 73 | | |
62 | 74 | | |
63 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
64 | 80 | | |
65 | 81 | | |
66 | 82 | | |
| |||
72 | 88 | | |
73 | 89 | | |
74 | 90 | | |
75 | | - | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
76 | 96 | | |
77 | 97 | | |
78 | 98 | | |
| |||
114 | 134 | | |
115 | 135 | | |
116 | 136 | | |
117 | | - | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
118 | 142 | | |
119 | 143 | | |
120 | 144 | | |
| |||
132 | 156 | | |
133 | 157 | | |
134 | 158 | | |
135 | | - | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
136 | 164 | | |
137 | 165 | | |
138 | 166 | | |
| |||
254 | 282 | | |
255 | 283 | | |
256 | 284 | | |
257 | | - | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
258 | 290 | | |
259 | 291 | | |
260 | 292 | | |
| |||
265 | 297 | | |
266 | 298 | | |
267 | 299 | | |
268 | | - | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
269 | 305 | | |
270 | 306 | | |
271 | 307 | | |
| |||
276 | 312 | | |
277 | 313 | | |
278 | 314 | | |
279 | | - | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
280 | 320 | | |
281 | 321 | | |
282 | 322 | | |
| |||
287 | 327 | | |
288 | 328 | | |
289 | 329 | | |
290 | | - | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
291 | 335 | | |
292 | 336 | | |
293 | 337 | | |
| |||
298 | 342 | | |
299 | 343 | | |
300 | 344 | | |
301 | | - | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
302 | 350 | | |
303 | 351 | | |
304 | 352 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
183 | | - | |
184 | | - | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
185 | 187 | | |
186 | 188 | | |
187 | 189 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
55 | 56 | | |
56 | 57 | | |
57 | 58 | | |
58 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
59 | 64 | | |
60 | 65 | | |
61 | 66 | | |
| |||
81 | 86 | | |
82 | 87 | | |
83 | 88 | | |
84 | | - | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
85 | 94 | | |
86 | 95 | | |
87 | 96 | | |
| |||
94 | 103 | | |
95 | 104 | | |
96 | 105 | | |
97 | | - | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
98 | 111 | | |
99 | 112 | | |
100 | 113 | | |
| |||
107 | 120 | | |
108 | 121 | | |
109 | 122 | | |
110 | | - | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
111 | 128 | | |
112 | 129 | | |
113 | 130 | | |
| |||
120 | 137 | | |
121 | 138 | | |
122 | 139 | | |
123 | | - | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
124 | 145 | | |
125 | 146 | | |
126 | 147 | | |
| |||
165 | 186 | | |
166 | 187 | | |
167 | 188 | | |
168 | | - | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
169 | 194 | | |
170 | 195 | | |
171 | 196 | | |
| |||
184 | 209 | | |
185 | 210 | | |
186 | 211 | | |
187 | | - | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
188 | 217 | | |
189 | 218 | | |
190 | 219 | | |
| |||
313 | 342 | | |
314 | 343 | | |
315 | 344 | | |
316 | | - | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
317 | 350 | | |
318 | 351 | | |
319 | 352 | | |
| |||
325 | 358 | | |
326 | 359 | | |
327 | 360 | | |
328 | | - | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
329 | 366 | | |
330 | 367 | | |
331 | 368 | | |
| |||
337 | 374 | | |
338 | 375 | | |
339 | 376 | | |
340 | | - | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
341 | 382 | | |
342 | 383 | | |
343 | 384 | | |
| |||
349 | 390 | | |
350 | 391 | | |
351 | 392 | | |
352 | | - | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
353 | 398 | | |
354 | 399 | | |
355 | 400 | | |
| |||
361 | 406 | | |
362 | 407 | | |
363 | 408 | | |
364 | | - | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
365 | 414 | | |
366 | 415 | | |
367 | 416 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
0 commit comments