Skip to content

Commit c3a6dac

Browse files
wz-gsaOpenCode Agentmogul
authored
fix(usai-catalog): refuse a near-miss instead of enriching from the wrong model (#484)
Fixes #387. ## The defect `integrations/providers/usai/scripts/build-catalog.mjs`'s models.dev matcher scored candidates but never **rejected** one. A gateway model absent from the enrichment source was enriched with the nearest same-family model's data. Reproduced offline against the committed fixtures (`--models-url tests/fixtures/models-list.json --models-dev tests/fixtures/models-dev.json`): ``` findModelsDevMatch("claude_4_5_haiku") -> anthropic.claude-opus-5 ctx out cost in/out reviewed/shipped 200000 64000 1 / 5 enriched 1000000 128000 5 / 25 <- Opus's numbers ``` A **5x price overstatement**, presented as fact. The fixture contains no haiku entry at all. Scoring alone cannot fix this — it ranks whatever field it is handed, and the best of a bad field still wins. There was no floor, so ids that exist nowhere in the source still matched: ``` claude_9_9_nonexistent -> anthropic.claude-opus-5 claude_totally_made_up -> anthropic.claude-opus-5 llama_99_fake -> meta.llama4-maverick-17b-instruct-v1:0 ``` Mechanism: a variant **match** scored `+50`, but a variant **mismatch** scored nothing. Only `lite` carried a mismatch penalty. Base score `10` survived every disagreement. This is worse than missing data: the fields come out populated and plausible, so a completeness check ("every model has a price") **passes** while the values are wrong. ## The fix An **eligibility guard** ahead of the score (`isPreciseMatch`). A candidate must agree on all of: | Axis | Because | |---|---| | family | claude is not gpt | | tier / variant | haiku is not opus is not sonnet; flash is not pro | | major.minor | 4.5 is not 5, and is not 4.8 | | `lite` | flash-lite is not flash | No match now means the model is emitted **visibly unenriched** rather than silently mis-enriched. The scoring terms the guard subsumes are removed — a hard tier check plus a `-20` penalty for `pro` would be two mechanisms for one rule. What remains discriminates only between duplicate listings of the **same** model: dated SKUs, `us.`/`eu.` regional prefixes, modality siblings (`-tts`, `-vision`). `lite` detection moves from a substring test to a token test. The verified normalizations are unchanged: gateway deployment suffixes still strip (`gpt_5_5_default_v2` matches `gpt-5.5`) and the `llama4` -> `llama-4` glue-split still matches the Bedrock form. ### Unenrichment is reported, not silent The issue asked for absent fields. Limits cannot be absent here — `shapeFromFeeds` applies `FALLBACK_LIMITS` (128000/8192), and making them absent would make the opencode emitter write literal `undefined` into the JSON (a separate defect, not fixed in this PR). Cost has no fallback, so it genuinely goes absent. So the fallback is made **loud** instead: ``` UNENRICHED (4): no models.dev entry agreed on family, tier and major.minor; limits fall back to 128000/8192 and cost is ABSENT: claude_4_5_haiku claude_4_5_sonnet claude_4_7_opus claude_4_8_opus ``` Suppressed under `--no-enrichment`, so "did not look" cannot masquerade as "looked and failed". Note `claude_4_7_opus` / `claude_4_8_opus`: these previously matched `claude-opus-5` — a **different model** — and their numbers merely *coincided* with the reviewed values. The fixture has no 4.7/4.8 entry. They were right by luck, and now report honestly. ## Tests 23 new hermetic tests in `tests/matcher-precision.test.mjs`, covering the four negative classes the issue names plus the positive controls it requires. Fixtures are **inline**, not the committed `models-dev.json` subset — a future refresh of that subset must not be able to quietly turn a negative case positive. **Proved failable:** neutering the guard with an early `return true` fails **11 of 23**, while all 12 positive controls stay green. The tests exercise the guard rather than merely passing alongside it. ## Verification ``` providers/usai 63/63 (was 40) usai-provider kit 59/59 paseo 13/13 make validate clean make validate-kits clean (10 kits) ``` `catalog.json` is **unchanged** — it is bootstrapped from the shipped `opencode.jsonc`, and `--bootstrap` output is byte-identical to the committed file. The byte-exact round-trip guards for both shipped `opencode.jsonc` copies still pass. This is therefore a **latent** fix: it changes nothing shipped today, and makes the next live regeneration trustworthy. That is the condition the issue describes as the reason the guard exists. --- AI-assisted (OpenCode). Advisory — requires human review and sign-off. --------- Co-authored-by: OpenCode Agent <[email protected]> Co-authored-by: Bret Mogilefsky <[email protected]>
1 parent 89e87f9 commit c3a6dac

3 files changed

Lines changed: 570 additions & 33 deletions

File tree

‎integrations/providers/usai/scripts/build-catalog.mjs‎

Lines changed: 197 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -179,19 +179,74 @@ function normalizeModelId(id) {
179179
return out.replace(/-+$/, "")
180180
}
181181

182+
// Token classes shared by the tokenizer and the precision guard. They are
183+
// defined once so the guard's identity residue cannot drift out of agreement
184+
// with what the tokenizer consumed — a token recognized here as a family or
185+
// variant must not also count as identity-bearing remainder.
186+
const FAMILY_TOKEN = /^(claude|gpt|gemini|llama\d*|command)$/i
187+
const VARIANT_TOKEN = /^(opus|sonnet|haiku|pro|flash|mini|nano|maverick)$/i
188+
const LEADING_PROVIDER_TOKEN = /^(anthropic|openai|google|meta|amazon|aws|bedrock|azure|vertex|mistral|ai21|deepseek|microsoft|xai|cohere)$/
189+
const LEADING_REGION_TOKEN = /^(us|eu|au|uk|ca|jp|ap|sa|global|gov|govcloud)$/
190+
const MARKETPLACE_REVISION_TOKEN = /^v\d+(?::\d+)?$/
191+
192+
// A terminal ISO date is deployment metadata, not part of the model version.
193+
// Accept the extended and basic ISO forms used by models.dev, but only after a
194+
// calendar check. A date-shaped but invalid suffix stays on the identity below
195+
// so it cannot collapse onto an undated model or a different invalid suffix.
196+
function terminalDateMetadata(normalized) {
197+
// Marketplace revisions can follow the date (`...-20251101-v1:0`). Peel the
198+
// recognized packaging token only for date parsing, then restore it to the
199+
// identity so the ordinary noise filter consumes it.
200+
const revision = normalized.match(/(-v\d+(?::\d+)?)$/)?.[1] ?? ""
201+
const dateInput = revision ? normalized.slice(0, -revision.length) : normalized
202+
const match = dateInput.match(/-(\d{4})(?:-(\d{2})-(\d{2})|(\d{2})(\d{2}))$/)
203+
if (!match) return { identity: normalized, date: null, invalidDate: null }
204+
205+
const year = Number.parseInt(match[1], 10)
206+
const month = Number.parseInt(match[2] ?? match[4], 10)
207+
const day = Number.parseInt(match[3] ?? match[5], 10)
208+
const leap = year % 4 === 0 && (year % 100 !== 0 || year % 400 === 0)
209+
const daysInMonth = [31, leap ? 29 : 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
210+
const valid = month >= 1 && month <= 12 && day >= 1 && day <= daysInMonth[month - 1]
211+
if (!valid) return { identity: normalized, date: null, invalidDate: match[0].slice(1) }
212+
213+
return {
214+
identity: `${dateInput.slice(0, match.index)}${revision}`,
215+
date: `${match[1]}-${String(month).padStart(2, "0")}-${String(day).padStart(2, "0")}`,
216+
invalidDate: null,
217+
}
218+
}
219+
182220
function extractModelTokens(id) {
183221
const normalized = normalizeModelId(id)
184-
const parts = normalized.split("-")
222+
const dateMetadata = terminalDateMetadata(normalized)
223+
// Dots delimit provider/region prefixes (`eu.anthropic.claude`), while
224+
// colons stay inside marketplace revision tokens (`v1:0`). Numeric version
225+
// dots were already normalized to hyphens by normalizeModelId().
226+
const parts = dateMetadata.identity.split("-").flatMap((part) => part.split(/[./]/))
185227
let family = ""
186228
let variant = ""
187229
const versionParts = []
188-
for (const part of parts) {
189-
if (/^(claude|gpt|gemini|llama\d*|cohere|command)$/i.test(part)) {
190-
family = /^llama/i.test(part) ? "llama" : /^command$/i.test(part) ? "cohere" : part
191-
} else if (/^(opus|sonnet|haiku|pro|flash|mini|nano|maverick)$/i.test(part)) {
192-
variant = part
193-
} else if (/^\d+$/.test(part)) {
194-
versionParts.push(Number.parseInt(part, 10))
230+
const identityParts = []
231+
for (const [index, part] of parts.entries()) {
232+
if (!family && (LEADING_PROVIDER_TOKEN.test(part) || LEADING_REGION_TOKEN.test(part))) {
233+
continue
234+
}
235+
if (FAMILY_TOKEN.test(part)) {
236+
const candidate = /^llama/i.test(part) ? "llama" : /^command$/i.test(part) ? "cohere" : part
237+
if (!family) family = candidate
238+
else identityParts.push(part)
239+
} else if (VARIANT_TOKEN.test(part)) {
240+
if (!variant) variant = part
241+
else identityParts.push(part)
242+
} else if (/^\d+$/.test(part) && versionParts.length < 2) {
243+
// Compare version tokens as canonical strings: converting an arbitrary
244+
// decimal id to Number loses precision above MAX_SAFE_INTEGER.
245+
versionParts.push(part.replace(/^0+(?=\d)/, ""))
246+
} else if (index === parts.length - 1 && MARKETPLACE_REVISION_TOKEN.test(part)) {
247+
continue
248+
} else {
249+
identityParts.push(part)
195250
}
196251
}
197252
if (!family) {
@@ -201,47 +256,128 @@ function extractModelTokens(id) {
201256
else if (/llama/i.test(normalized)) family = "llama"
202257
else if (/cohere|command/i.test(normalized)) family = "cohere"
203258
}
204-
return { family, version: versionParts, variant }
259+
return { family, version: versionParts, variant, identityParts, ...dateMetadata }
260+
}
261+
262+
// Tokens that carry no model identity, only packaging. Each is a FULL token
263+
// (the id is split on separators first), so no substring can match by accident.
264+
// Anything NOT listed here and not a family/variant/version token is treated as
265+
// identity-bearing — see identityResidue below for why that direction matters.
266+
const IDENTITY_NOISE = [
267+
// Parameter counts and tuning descriptors name packaging, not identity.
268+
/^\d+(\.\d+)?x?\d*[bmk]$/,
269+
/^(instruct|instructed|it|chat|text)$/,
270+
]
271+
272+
// Does an id carry the `lite` qualifier as its own token? Substring testing is
273+
// wrong here: it would see "lite" inside an unrelated word. `flash-lite` is a
274+
// DIFFERENT model from `flash`, with different limits and pricing.
275+
function hasLiteToken(normalizedId) {
276+
return /(^|[.\-/])lite([.\-]|$)/.test(normalizedId)
277+
}
278+
279+
// Every token an id carries that is NOT its family, variant, version, or known
280+
// packaging noise. This is the identity-bearing remainder.
281+
//
282+
// WHY A RESIDUE AND NOT A LIST OF QUALIFIERS. The earlier guard compared a
283+
// hard-coded set (`codex`, `mini`, `nano`, `max`). That fails OPEN for any
284+
// qualifier not on the list, which is the whole class of tokens a future
285+
// gateway id might add: `gpt-5.2-luna` collapsed onto `gpt-5.2`,
286+
// `claude-opus-5-thinking` onto `claude-opus-5`, and
287+
// `gemini-2.5-flash-preview` onto `gemini-2.5-flash` — each inheriting a
288+
// sibling's limits and pricing, the exact defect this guard exists to prevent.
289+
// Enumerating qualifiers cannot work, because the set is open and owned by the
290+
// vendors. Enumerating NOISE can, because that set is small, observable in the
291+
// catalogs we actually read, and safe to be wrong about in the closed
292+
// direction: an unrecognized noise token only costs a missed enrichment, which
293+
// is reported as unenriched rather than silently wrong.
294+
function identityResidue(tokens) {
295+
const residue = []
296+
for (const part of tokens.identityParts) {
297+
if (!part) continue
298+
if (IDENTITY_NOISE.some((re) => re.test(part))) continue
299+
residue.push(/^\d+$/.test(part) ? part.replace(/^0+(?=\d)/, "") : part)
300+
}
301+
if (tokens.invalidDate) residue.push(tokens.invalidDate)
302+
return residue
303+
}
304+
305+
function sameIdentityResidue(usaiTokens, catalogTokens) {
306+
const a = identityResidue(usaiTokens)
307+
const b = identityResidue(catalogTokens)
308+
return a.length === b.length && a.every((token, index) => token === b[index])
309+
}
310+
311+
// PRECISION GUARD. Scoring alone cannot be trusted to reject a wrong candidate:
312+
// it ranks whatever it is given and the best of a bad field still wins. These
313+
// are the agreements a candidate MUST satisfy to be considered at all, because
314+
// disagreeing on any of them means it is a DIFFERENT model:
315+
//
316+
// family claude is not gpt
317+
// tier/variant haiku is not opus is not sonnet; flash is not pro
318+
// major.minor 4.5 is not 5 and is not 4.8
319+
// lite flash-lite is not flash
320+
// residue every other identity-bearing token must agree, so an
321+
// unrecognized qualifier fails CLOSED instead of being ignored
322+
//
323+
// Rationale for failing closed: enriching a model with a different model's
324+
// numbers is worse than having none, because the fields come out populated and
325+
// plausible, so a completeness check ("every model has a price") passes while
326+
// the values are wrong. A gateway model absent from the enrichment source must
327+
// end up visibly unenriched, never silently mis-enriched.
328+
function isPreciseMatch(usaiTokens, catalogTokens) {
329+
if (usaiTokens.family !== catalogTokens.family) return false
330+
if ((usaiTokens.variant || "") !== (catalogTokens.variant || "")) return false
331+
if (hasLiteToken(usaiTokens.identity) !== hasLiteToken(catalogTokens.identity)) return false
332+
if (!sameIdentityResidue(usaiTokens, catalogTokens)) return false
333+
// A dated gateway deployment names one exact snapshot. An undated deployment
334+
// is an alias and may use either an undated entry or a dated snapshot.
335+
if (usaiTokens.date && usaiTokens.date !== catalogTokens.date) return false
336+
// Compare major and minor positionally. Both absent is agreement (a bare
337+
// family id on both sides); one absent is not.
338+
if (usaiTokens.version[0] !== catalogTokens.version[0]) return false
339+
if (usaiTokens.version[1] !== catalogTokens.version[1]) return false
340+
return true
205341
}
206342

207343
function findModelsDevMatch(usaiId, catalog) {
208344
const usaiTokens = extractModelTokens(usaiId)
209345
const usaiNorm = normalizeModelId(usaiId)
210-
const usaiHasLite = /lite/.test(usaiNorm)
211346
let bestMatch = null
212-
let bestScore = -Infinity
347+
let bestRank = null
213348
for (const [catalogId, data] of Object.entries(catalog)) {
214349
const catalogTokens = extractModelTokens(catalogId)
215350
const catalogNorm = normalizeModelId(catalogId)
216-
if (usaiTokens.family !== catalogTokens.family) continue
351+
// The guard decides ELIGIBILITY; ranking below only orders candidates that
352+
// already agree on identity (dated aliases and regional duplicates).
353+
if (!isPreciseMatch(usaiTokens, catalogTokens)) continue
354+
// Date rank comes before packaging score. Thus an undated identity always
355+
// beats its dated aliases; when only dated aliases exist, the latest valid
356+
// date wins. The catalog has no explicit alias mapping, so latest-date is
357+
// the only deterministic freshness signal available.
358+
const undatedRank = catalogTokens.date ? 0 : 1
359+
const dateRank = catalogTokens.date ?? ""
217360
let score = 10
218-
if (usaiTokens.variant && usaiTokens.variant === catalogTokens.variant) score += 50
219-
const catalogHasLite = /lite/.test(catalogNorm)
220-
if (usaiHasLite !== catalogHasLite) score -= 40
221-
const versionMatch = usaiTokens.version.every((v, i) => catalogTokens.version[i] === v)
222-
if (versionMatch && usaiTokens.version.length > 0) {
223-
score += 30
224-
if (catalogTokens.version.length === usaiTokens.version.length) score += 3
225-
}
226-
for (const suffix of ["tts", "image", "audio", "vision", "embed", "embedding", "search", "realtime"]) {
227-
const re = new RegExp(`(^|[.\\-/])${suffix}([.\\-]|$)`)
228-
if (re.test(catalogNorm) && !re.test(usaiNorm)) score -= 60
229-
}
230-
for (const v of ["pro", "codex", "mini", "nano", "max"]) {
231-
const re = new RegExp(`(^|[.\\-/])${v}([.\\-]|$)`)
232-
if (re.test(catalogNorm) && !re.test(usaiNorm)) score -= 20
233-
}
361+
// Regional duplicates of one model: prefer the unprefixed or us/global
362+
// listing, since USAi serves from a US deployment.
234363
if (/^(eu|au)\./.test(catalogId)) score -= 15
235364
else if (/^(us|global)\./.test(catalogId)) score += 2
236365
else if (/^[a-z]{2}\./.test(catalogId)) score += 1
237366
else score += 3
238-
if (normalizeModelId(catalogId).startsWith(normalizeModelId(usaiId).slice(0, 10))) score += 5
239-
if (score > bestScore) {
240-
bestScore = score
367+
if (catalogNorm.startsWith(usaiNorm.slice(0, 10))) score += 5
368+
const rank = [undatedRank, dateRank, score]
369+
const isBetter =
370+
!bestRank ||
371+
rank[0] > bestRank[0] ||
372+
(rank[0] === bestRank[0] && rank[1] > bestRank[1]) ||
373+
(rank[0] === bestRank[0] && rank[1] === bestRank[1] && rank[2] > bestRank[2]) ||
374+
(rank[0] === bestRank[0] && rank[1] === bestRank[1] && rank[2] === bestRank[2] && catalogId < bestMatch.id)
375+
if (isBetter) {
376+
bestRank = rank
241377
bestMatch = { id: catalogId, data }
242378
}
243379
}
244-
return bestMatch && bestScore > 0 ? bestMatch : null
380+
return bestMatch
245381
}
246382

247383
function providerModelMaps(vendor, catalog) {
@@ -613,6 +749,23 @@ function reportExclusions(excluded) {
613749
}
614750
}
615751

752+
// A model the precision guard could not match is NOT a silent condition. Its
753+
// limits fall back to generic defaults and its cost is absent, so the catalog
754+
// entry looks complete while carrying no derived pricing at all. Say so, by id,
755+
// and make the fallback visible — "could not enrich" has to be distinguishable
756+
// from "enriched", or a reviewer cannot tell a generic 128000/8192 from a real
757+
// measurement.
758+
export function reportUnenriched(unenriched, enrichmentAttempted) {
759+
if (!enrichmentAttempted || unenriched.length === 0) return
760+
process.stderr.write(` UNENRICHED (${unenriched.length}): no precise models.dev match; cost is ABSENT:\n`)
761+
for (const item of unenriched) {
762+
const suffix = item.fallbacks.length
763+
? `; fallback ${item.fallbacks.join("+")} limit(s) used`
764+
: "; USAi feed limits retained"
765+
process.stderr.write(` ${item.id}${suffix}\n`)
766+
}
767+
}
768+
616769
function shapeFromFeeds(usaiList, modelsDevCatalog) {
617770
usaiList = extractUsaiList(usaiList)
618771
// Chat models only, preserve incoming list order but group by vendor display
@@ -643,17 +796,20 @@ function shapeFromFeeds(usaiList, modelsDevCatalog) {
643796

644797
const vendorOrder = []
645798
const models = []
799+
const unenriched = []
646800
for (const p of parsed) {
647801
if (!vendorOrder.includes(p.vendor)) vendorOrder.push(p.vendor)
648802
const model = { id: p.id, vendor: p.vendor, name: p.name }
649803
let context = Number.isInteger(p.raw.context_window) && p.raw.context_window > 0 ? p.raw.context_window : null
650804
let output = Number.isInteger(p.raw.max_output_tokens) && p.raw.max_output_tokens > 0 ? p.raw.max_output_tokens : null
651805
let baseCost = null
652806
let above = null
807+
let matched = false
653808
if (modelsDevCatalog) {
654809
for (const { models: providerModels } of providerModelMaps(p.vendor, modelsDevCatalog)) {
655810
const match = findModelsDevMatch(p.id, providerModels)
656811
if (match?.data) {
812+
matched = true
657813
if (match.data.limit) {
658814
const matchedContext = match.data.limit.context
659815
const matchedOutput = match.data.limit.output
@@ -666,13 +822,22 @@ function shapeFromFeeds(usaiList, modelsDevCatalog) {
666822
break
667823
}
668824
}
825+
// The feed itself may carry limits even when enrichment finds nothing;
826+
// only a model with no derived data at all is reported as unenriched.
827+
if (!matched) {
828+
const fallbacks = []
829+
if (!(Number.isInteger(context) && context > 0)) fallbacks.push("context")
830+
if (!(Number.isInteger(output) && output > 0)) fallbacks.push("output")
831+
unenriched.push({ id: p.id, fallbacks })
832+
}
669833
}
670834
model.contextWindow = normalizeLimit(context, FALLBACK_LIMITS.context)
671835
model.maxOutputTokens = normalizeLimit(output, FALLBACK_LIMITS.output)
672836
if (baseCost) model.cost = baseCost
673837
if (above) model.costAbove200kContext = above
674838
models.push(model)
675839
}
840+
reportUnenriched(unenriched, Boolean(modelsDevCatalog))
676841
return { models, vendorOrder }
677842
}
678843

‎integrations/providers/usai/tests/feed-fixture.test.mjs‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ import path from "node:path"
1212
import { fileURLToPath } from "node:url"
1313
import { test } from "node:test"
1414

15-
import { nonChatReason, shapeFromFeeds } from "../scripts/build-catalog.mjs"
15+
import { nonChatReason, reportUnenriched, shapeFromFeeds } from "../scripts/build-catalog.mjs"
1616

1717
const HERE = path.dirname(fileURLToPath(import.meta.url))
1818
const FIXTURES = path.join(HERE, "fixtures")
@@ -83,6 +83,25 @@ test("a NEW underscore-delimited embedding id is excluded by the pattern", () =>
8383
assert.equal(ids.includes("cohere_embed_v4"), false)
8484
})
8585

86+
test("unenriched reporting distinguishes feed limits from hardcoded fallbacks", () => {
87+
const writes = []
88+
const original = process.stderr.write
89+
process.stderr.write = (chunk) => { writes.push(String(chunk)); return true }
90+
try {
91+
reportUnenriched([
92+
{ id: "feed-limits", fallbacks: [] },
93+
{ id: "fallback-limits", fallbacks: ["context", "output"] },
94+
], true)
95+
reportUnenriched([{ id: "suppressed", fallbacks: ["context"] }], false)
96+
} finally {
97+
process.stderr.write = original
98+
}
99+
const output = writes.join("")
100+
assert.match(output, /feed-limits; USAi feed limits retained/)
101+
assert.match(output, /fallback-limits; fallback context\+output limit\(s\) used/)
102+
assert.doesNotMatch(output, /suppressed/)
103+
})
104+
86105
test("shapeFromFeeds rejects oversized and malformed feed entries", () => {
87106
assert.throws(() => shapeFromFeeds(Array.from({ length: 1001 }, (_, i) => ({ id: `m${i}` })), null), /exceeds limit/)
88107
assert.throws(() => shapeFromFeeds([null], null), /must be an object/)

0 commit comments

Comments
 (0)