fix(usai-catalog): refuse a near-miss instead of enriching from the wrong model - #484

AI-assisted follow-up (OpenCode), advisory — needs a human to confirm before it drives a change. Pushed 3cf43f0.

@basiliskus's finding was correct, and I reproduced it before changing anything. Extracted the guard from this PR's build-catalog.mjs at head e1346addd and drove it directly:

gpt-5.2-luna                 vs gpt-5.2              -> MATCH
claude-opus-5-thinking       vs claude-opus-5        -> MATCH
gemini-2.5-flash-preview     vs gemini-2.5-flash     -> MATCH
claude-opus-5-20260101       vs claude-opus-5        -> reject   (control)
gpt-5.2-mini                 vs gpt-5.2              -> reject   (control)
gemini-2.5-flash-lite        vs gemini-2.5-flash     -> reject   (control)

All three reported false matches reproduce. The controls confirm the guard was not broken generally — it failed open only for tokens absent from its hardcoded list, which is precisely the diagnosis in the review.

The fix inverts the comparison instead of extending the list

I did not add luna/thinking/preview to PROFILE_TOKENS. That would fix these three strings and leave the method intact, and the method is the defect: the qualifier set is open and owned by the vendors, so any allowlist is permanently one release behind. A fourth qualifier would have arrived and the same comment would have been written again.

identityResidue collects every token that is not a family, variant, version, or known packaging token, and a candidate must agree on that whole remainder:

function identityResidue(normalizedId) {
  const residue = new Set()
  for (const part of normalizedId.split(/[.\-/:]/)) {
    if (!part) continue
    if (FAMILY_TOKEN.test(part) || VARIANT_TOKEN.test(part)) continue
    if (/^\d+$/.test(part)) continue
    if (IDENTITY_NOISE.some((re) => re.test(part))) continue
    residue.add(part)
  }
  return residue
}

Unknown input now fails closed by construction rather than by enumeration.

Why enumerating noise is safe where enumerating qualifiers was not. The noise set is small, observable in the catalogs actually read (provider prefixes, region scopes, marketplace revisions like v1:0, parameter counts like 17b, tuning descriptors like instruct), and asymmetric in consequence: being wrong about a noise token costs a missed enrichment, which the pipeline already reports as unenriched. Being wrong about a qualifier cost a silent mis-enrichment. The two error directions are not equally bad, and the guard should fail in the cheap one.

I also hoisted FAMILY_TOKEN / VARIANT_TOKEN to module scope so the tokenizer and the residue cannot drift apart — a token consumed as a family or variant must not also count as identity-bearing remainder. They were previously inline literals in extractModelTokens.

Tests use invented qualifiers on purpose

Five new tests, with deliberately fictional tokens (luna, thinking, preview, solstice):

an unrecognized qualifier does not match the bare base model
a bare base model does not match an unrecognized-qualifier entry
two different unrecognized qualifiers do not match each other
an unrecognized qualifier still matches its own exact entry
an unrecognized qualifier is not rescued by a provider prefix or region

A test using a real qualifier would start passing the moment someone adds that token to a list — which is the failure mode being guarded against. An invented token can only pass if identity comparison is genuinely closed over unknown input.

The review asked for coverage of "an unknown same-version qualifier"; I added the reverse direction too, since a request for the base model is the more common case and a base→qualified match is equally damaging. And one positive control, because failing closed must not mean failing always: a qualified id still matches its own exact entry.

Verification

Confirmed these are regression tests, not restatements — the new tests against the previous guard:

not ok 13 - an unrecognized qualifier does not match the bare base model
not ok 14 - a bare base model does not match an unrecognized-qualifier entry
not ok 15 - two different unrecognized qualifiers do not match each other
not ok 16 - an unrecognized qualifier still matches its own exact entry
not ok 17 - an unrecognized qualifier is not rescued by a provider prefix or region
# tests 29  # pass 24  # fail 5

With the fix, in a network-off container:

catalog-check.test.mjs       # pass 9  # fail 0
catalog-validation.test.mjs  # pass 10 # fail 0
emitters.test.mjs            # pass 6  # fail 0
feed-fixture.test.mjs        # pass 9  # fail 0
matcher-precision.test.mjs   # pass 29 # fail 0
models-json-regen.test.mjs   # pass 3  # fail 0
non-chat-filter.test.mjs     # pass 10 # fail 0
prime-agent.test.mjs         # pass 10 # fail 0

node scripts/build-catalog.mjs --check  ->  catalog.json is up to date (20 models)
python scripts/validate_repo.py         ->  All validators passed

No real model loses data. I ran shapeFromFeeds over the committed fixtures before and after: the same 4 models (claude_4_5_haiku, claude_4_5_sonnet, claude_4_7_opus, claude_4_8_opus) were already unenriched under the old guard and still are. The stricter comparison rejected nothing that was previously enriched.

One thing I want to flag myself

IDENTITY_NOISE is still a list, so it has the same structural weakness as the one I removed — just pointed in the safe direction. If a vendor ships a genuinely new packaging token, that model goes unenriched until the list learns it. That is a visible, reported outcome rather than a wrong number, which is why I accepted it, but it is a maintenance surface and worth naming rather than leaving for the next reviewer to find.

Also: gemini-2.5-flash vs gemini-2.5-flash-tts is now rejected by the guard rather than losing on score (-60). The existing ranking test still passes because the chat model is present in that fixture, but the modality-suffix scoring branch is now partly unreachable. I left it in place as defense-in-depth rather than widening this diff further; say the word if you would rather it be removed.

Branch updated to current main (2a0816a) so the review path is clear — the previous head was 1 commit behind, which showed as BEHIND on the merge state.

The merge was clean. Worth noting why that wasn't a given: 800ba7e ("serialize feed data before config emission") touches the same integrations/providers/usai/scripts/build-catalog.mjs this PR modifies, but it was already an ancestor of this branch, so the only incoming change was CHANGELOG.md from the 1.13.0 release commit.

Re-ran the full suite after merging rather than assuming it still held:

providers/usai tests                    ->  86 passed, 0 failed
node scripts/build-catalog.mjs --check  ->  catalog.json is up to date (20 models)
CI                                      ->  7/7 SUCCESS

No code changed in this push — it is a merge commit only. The fix itself is still 3cf43f0, described in my previous comment.

@basiliskus this is ready for another look when you have time. mergeStateStatus is now BLOCKED purely on the missing approval, not on a failing check or a stale branch.

One process note: the repository ruleset has dismiss_stale_reviews_on_push: true, so an approval gets dismissed by any subsequent push. I'm not planning further pushes here, but if you approve and then ask for a change, the approval will need re-granting after I push the fix.

AI-assisted (OpenCode), advisory — a human should confirm before this merges.

AI-assisted follow-up (OpenCode). Pushed 933169d9cf73e4ea05b7f68881a312964c87b06f and addressed the current-head date/region over-refusal finding.

The matcher now:

• models valid terminal basic/extended ISO dates separately from semantic major/minor identity;
• requires exact dates for dated gateway IDs;
• prefers an undated identity regardless of catalog order, or deterministically selects the latest valid dated alias;
• treats provider/region text as noise only in leading prefixes and marketplace revisions only terminally;
• keeps malformed dates, extra/large numeric qualifiers, duplicate structural tokens, unknown qualifiers, and modality markers as ordered identity residue;
• compares numeric identity as strings, avoiding Number precision collisions.

Verification at the committed head:

matcher-precision.test.mjs: 43 passed, 0 failed
providers/usai full suite: 100 passed, 0 failed
pre-commit hooks: PASS
git diff --check: PASS
independent fixed-head QA/security: no findings

Please re-review exact head 933169d9cf73e4ea05b7f68881a312964c87b06f.