You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
build-catalog: fuzzy matcher can enrich a model with a DIFFERENT model's pricing #387
Defect (data-correctness, caught during a live gateway regeneration)
integrations/providers/usai/scripts/build-catalog.mjs's models.dev fuzzy matcher is too permissive: when it cannot find a correct match for a gateway model id, it falls through to a different model's data instead of leaving the fields absent.
Reproduced: with only claude-opus-5 / claude-sonnet-5 present under the anthropic provider in the enrichment subset, claude_4_5_haiku was enriched with Opus numbers (context 1,000,000, $5/$25 — correct is 200,000, $1/$5) and claude_4_5_sonnet picked up Sonnet-5 pricing ($2/$10 vs the correct $3/$15).
This is worse than missing data: the fields look populated and plausible, so a completeness check ("every model has cost/context") passes while the values are wrong. Shipping that would tell users Haiku costs $5/$25.
Fix
Add a precision guard: a candidate is accepted only when it agrees on both
family/tier — haiku must not match opus/sonnet; flash must not match pro; flash-lite must not match plain flash; mini/nano/pro variants must not match the base model; and
major.minor version — 4.5 must not match 5 or 4.8 (handle 4_5, 4.5, -4-5, bare 5).
If no candidate satisfies both, return no match and emit the model with absent context/output/cost. Absent-but-honest beats populated-but-wrong.
Keep the legitimate normalizations that were verified working: gateway-suffix strips (guardrails.*, latest.*, -default-?v\d+) so gpt_5_5_default_v2 still matches gpt-5.5, and the llama4→llama-4 glue-split for the Bedrock meta.llama4-maverick-… form.
Tests to add
Hermetic (inline models.dev-shaped fixtures, not the committed subset):
claude_4_5_haiku must not match an opus/sonnet entry; claude_4_5_sonnet must not match claude-sonnet-5.
gemini-2.5-flash must not match gemini-2.5-pro; gemini-2.5-flash-lite must not match plain gemini-2.5-flash.
Positive controls that must still match: gpt_5_5_default_v2→gpt-5.5, llama_4_maverick→bedrock llama4-maverick, claude-opus-5→claude-opus-5, gemini-3.5-flash→gemini-3.5-flash.
Context
Deferred from the urgent model-refresh PR (which shipped correct values by carrying forward the reviewed numbers for the unchanged ids rather than trusting the matcher). The guard is what makes future live regenerations safe to trust.
Activity
fix(usai-catalog): refuse a near-miss instead of enriching from the w…