You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c3a6dac
Browse filesBrowse the repository at this point in the historyBrowse files
authored
fix(usai-catalog): refuse a near-miss instead of enriching from the wrong model (#484)
Fixes#387.
## The defect
`integrations/providers/usai/scripts/build-catalog.mjs`'s models.dev
matcher scored candidates but never **rejected** one. A gateway model
absent from the enrichment source was enriched with the nearest
same-family model's data.
Reproduced offline against the committed fixtures (`--models-url
tests/fixtures/models-list.json --models-dev
tests/fixtures/models-dev.json`):
```
findModelsDevMatch("claude_4_5_haiku") -> anthropic.claude-opus-5
ctx out cost in/out
reviewed/shipped 200000 64000 1 / 5
enriched 1000000 128000 5 / 25 <- Opus's numbers
```
A **5x price overstatement**, presented as fact. The fixture contains no
haiku entry at all.
Scoring alone cannot fix this — it ranks whatever field it is handed,
and the best of a bad field still wins. There was no floor, so ids that
exist nowhere in the source still matched:
```
claude_9_9_nonexistent -> anthropic.claude-opus-5
claude_totally_made_up -> anthropic.claude-opus-5
llama_99_fake -> meta.llama4-maverick-17b-instruct-v1:0
```
Mechanism: a variant **match** scored `+50`, but a variant **mismatch**
scored nothing. Only `lite` carried a mismatch penalty. Base score `10`
survived every disagreement.
This is worse than missing data: the fields come out populated and
plausible, so a completeness check ("every model has a price")
**passes** while the values are wrong.
## The fix
An **eligibility guard** ahead of the score (`isPreciseMatch`). A
candidate must agree on all of:
| Axis | Because |
|---|---|
| family | claude is not gpt |
| tier / variant | haiku is not opus is not sonnet; flash is not pro |
| major.minor | 4.5 is not 5, and is not 4.8 |
| `lite` | flash-lite is not flash |
No match now means the model is emitted **visibly unenriched** rather
than silently mis-enriched.
The scoring terms the guard subsumes are removed — a hard tier check
plus a `-20` penalty for `pro` would be two mechanisms for one rule.
What remains discriminates only between duplicate listings of the
**same** model: dated SKUs, `us.`/`eu.` regional prefixes, modality
siblings (`-tts`, `-vision`). `lite` detection moves from a substring
test to a token test.
The verified normalizations are unchanged: gateway deployment suffixes
still strip (`gpt_5_5_default_v2` matches `gpt-5.5`) and the `llama4` ->
`llama-4` glue-split still matches the Bedrock form.
### Unenrichment is reported, not silent
The issue asked for absent fields. Limits cannot be absent here —
`shapeFromFeeds` applies `FALLBACK_LIMITS` (128000/8192), and making
them absent would make the opencode emitter write literal `undefined`
into the JSON (a separate defect, not fixed in this PR). Cost has no
fallback, so it genuinely goes absent.
So the fallback is made **loud** instead:
```
UNENRICHED (4): no models.dev entry agreed on family, tier and major.minor;
limits fall back to 128000/8192 and cost is ABSENT:
claude_4_5_haiku
claude_4_5_sonnet
claude_4_7_opus
claude_4_8_opus
```
Suppressed under `--no-enrichment`, so "did not look" cannot masquerade
as "looked and failed".
Note `claude_4_7_opus` / `claude_4_8_opus`: these previously matched
`claude-opus-5` — a **different model** — and their numbers merely
*coincided* with the reviewed values. The fixture has no 4.7/4.8 entry.
They were right by luck, and now report honestly.
## Tests
23 new hermetic tests in `tests/matcher-precision.test.mjs`, covering
the four negative classes the issue names plus the positive controls it
requires.
Fixtures are **inline**, not the committed `models-dev.json` subset — a
future refresh of that subset must not be able to quietly turn a
negative case positive.
**Proved failable:** neutering the guard with an early `return true`
fails **11 of 23**, while all 12 positive controls stay green. The tests
exercise the guard rather than merely passing alongside it.
## Verification
```
providers/usai 63/63 (was 40)
usai-provider kit 59/59
paseo 13/13
make validate clean
make validate-kits clean (10 kits)
```
`catalog.json` is **unchanged** — it is bootstrapped from the shipped
`opencode.jsonc`, and `--bootstrap` output is byte-identical to the
committed file. The byte-exact round-trip guards for both shipped
`opencode.jsonc` copies still pass.
This is therefore a **latent** fix: it changes nothing shipped today,
and makes the next live regeneration trustworthy. That is the condition
the issue describes as the reason the guard exists.
---
AI-assisted (OpenCode). Advisory — requires human review and sign-off.
---------
Co-authored-by: OpenCode Agent <[email protected]>
Co-authored-by: Bret Mogilefsky <[email protected]>
0 commit comments