Comparing changes

…rong model (#484) Fixes #387. ## The defect `integrations/providers/usai/scripts/build-catalog.mjs`'s models.dev matcher scored candidates but never **rejected** one. A gateway model absent from the enrichment source was enriched with the nearest same-family model's data. Reproduced offline against the committed fixtures (`--models-url tests/fixtures/models-list.json --models-dev tests/fixtures/models-dev.json`): ``` findModelsDevMatch("claude_4_5_haiku") -> anthropic.claude-opus-5 ctx out cost in/out reviewed/shipped 200000 64000 1 / 5 enriched 1000000 128000 5 / 25 <- Opus's numbers ``` A **5x price overstatement**, presented as fact. The fixture contains no haiku entry at all. Scoring alone cannot fix this — it ranks whatever field it is handed, and the best of a bad field still wins. There was no floor, so ids that exist nowhere in the source still matched: ``` claude_9_9_nonexistent -> anthropic.claude-opus-5 claude_totally_made_up -> anthropic.claude-opus-5 llama_99_fake -> meta.llama4-maverick-17b-instruct-v1:0 ``` Mechanism: a variant **match** scored `+50`, but a variant **mismatch** scored nothing. Only `lite` carried a mismatch penalty. Base score `10` survived every disagreement. This is worse than missing data: the fields come out populated and plausible, so a completeness check ("every model has a price") **passes** while the values are wrong. ## The fix An **eligibility guard** ahead of the score (`isPreciseMatch`). A candidate must agree on all of: | Axis | Because | |---|---| | family | claude is not gpt | | tier / variant | haiku is not opus is not sonnet; flash is not pro | | major.minor | 4.5 is not 5, and is not 4.8 | | `lite` | flash-lite is not flash | No match now means the model is emitted **visibly unenriched** rather than silently mis-enriched. The scoring terms the guard subsumes are removed — a hard tier check plus a `-20` penalty for `pro` would be two mechanisms for one rule. What remains discriminates only between duplicate listings of the **same** model: dated SKUs, `us.`/`eu.` regional prefixes, modality siblings (`-tts`, `-vision`). `lite` detection moves from a substring test to a token test. The verified normalizations are unchanged: gateway deployment suffixes still strip (`gpt_5_5_default_v2` matches `gpt-5.5`) and the `llama4` -> `llama-4` glue-split still matches the Bedrock form. ### Unenrichment is reported, not silent The issue asked for absent fields. Limits cannot be absent here — `shapeFromFeeds` applies `FALLBACK_LIMITS` (128000/8192), and making them absent would make the opencode emitter write literal `undefined` into the JSON (a separate defect, not fixed in this PR). Cost has no fallback, so it genuinely goes absent. So the fallback is made **loud** instead: ``` UNENRICHED (4): no models.dev entry agreed on family, tier and major.minor; limits fall back to 128000/8192 and cost is ABSENT: claude_4_5_haiku claude_4_5_sonnet claude_4_7_opus claude_4_8_opus ``` Suppressed under `--no-enrichment`, so "did not look" cannot masquerade as "looked and failed". Note `claude_4_7_opus` / `claude_4_8_opus`: these previously matched `claude-opus-5` — a **different model** — and their numbers merely *coincided* with the reviewed values. The fixture has no 4.7/4.8 entry. They were right by luck, and now report honestly. ## Tests 23 new hermetic tests in `tests/matcher-precision.test.mjs`, covering the four negative classes the issue names plus the positive controls it requires. Fixtures are **inline**, not the committed `models-dev.json` subset — a future refresh of that subset must not be able to quietly turn a negative case positive. **Proved failable:** neutering the guard with an early `return true` fails **11 of 23**, while all 12 positive controls stay green. The tests exercise the guard rather than merely passing alongside it. ## Verification ``` providers/usai 63/63 (was 40) usai-provider kit 59/59 paseo 13/13 make validate clean make validate-kits clean (10 kits) ``` `catalog.json` is **unchanged** — it is bootstrapped from the shipped `opencode.jsonc`, and `--bootstrap` output is byte-identical to the committed file. The byte-exact round-trip guards for both shipped `opencode.jsonc` copies still pass. This is therefore a **latent** fix: it changes nothing shipped today, and makes the next live regeneration trustworthy. That is the condition the issue describes as the reason the guard exists. --- AI-assisted (OpenCode). Advisory — requires human review and sign-off. --------- Co-authored-by: OpenCode Agent <[email protected]> Co-authored-by: Bret Mogilefsky <[email protected]>

## Summary Adds an opt-in `opendesign` ACQ kit that runs OpenDesign headlessly inside the sandbox as a browser-accessible design workspace. The kit reuses the existing sandbox OpenCode CLI and keeps model credentials on the ACQ/OpenCode path, especially when composed with `usai-provider`. ## Content Type - [ ] Skill - [ ] Prompt - [ ] Workflow - [ ] Agent instructions - [ ] Lesson learned - [x] Integration / ACQ kit ## What Changed - Added `integrations/isolation/acq-kits/opendesign/` with `hybrid/v1` spec, installer, supervisor, verifier, README, troubleshooting, and design records. - Pins OpenDesign to `open-design-v0.24.0` and verifies commit `0d3a14c1df6dc5017f3cc3ef05b24558250c220b`. - Installs kit-managed Node `24.21.0` with SHA-256 verification and pnpm `10.33.2`. - Runs OpenDesign on guest port `7456` and publishes it via neutral `publishedPorts`. - Adds a `10G` persistent volume for OpenDesign source/state/tooling under `/home/agent/.local/share/opendesign`. - Seeds OpenDesign app config for `agentId: "opencode"`, `onboardingCompleted: true`, and local telemetry disabled. - Does not seed a default model. - Does not copy `USAI_API_KEY` or any provider key into OpenDesign state. - Registers the kit in `kits.yaml` and the ACQ kits README. ## Safety Checklist - [x] No secrets, tokens, credentials, or private keys - [x] No PII, CUI, customer data, or sensitive info - [x] No internal URLs or system details - [x] Examples use placeholders - [x] Credentials remain with ACQ/OpenCode, not persisted by the kit ## Security Impact This adds a local browser daemon to the sandbox attack surface. - OpenDesign binds `0.0.0.0:7456` inside the guest so ACQ/MSB create-time port publishing can reach it. - The host side is expected to be loopback-only through ACQ port publishing. - `OD_DISABLE_API_AUTH=1` is intentional for this trusted, single-user ACQ browser kit posture, matching the approved direction during implementation. - Documentation explicitly warns not to forward the mapped port or use it on an untrusted multi-user host. - Live verification now checks both in-guest daemon/UI health and host-loopback reachability. ## Adversarial Review Follow-Up A sub-agent adversarial review flagged these issues and they were addressed: - Pinned pnpm store to the OpenDesign persistent volume. - Replaced recursive startup `chown -R` with shallow ownership initialization. - Made verifier output explicit about skipped schema validation. - Strengthened live credential check to look for the injected secret value, not just the variable name. - Kept auth-disabled posture as an intentional accepted risk documented in README and ADR. ## Verification Local verification run in this agent sandbox: ```text integrations/isolation/acq-kits/opendesign/scripts/verify Result: 15 passed, 0 failed, 1 skipped Note: schema validator skipped here because this sandbox lacks jsonschema/pyyaml. ``` ```text bash -n integrations/isolation/acq-kits/opendesign/scripts/verify sh -n integrations/isolation/acq-kits/opendesign/files/home/opendesign-install.sh sh -n integrations/isolation/acq-kits/opendesign/files/home/opendesign-start.sh git diff --check Result: passed ``` Host verification performed by maintainer during development: ```text python integrations/isolation/acq-kits/validate-kits.py --strict Result: all 8 acq-kits valid, including opendesign ``` ```text integrations/isolation/acq-kits/opendesign/scripts/verify Result: 13 passed, 0 failed before later verifier hardening ``` ```text KEEP=1 RUN_ACQ=1 READY_TIMEOUT=1200 integrations/isolation/acq-kits/opendesign/scripts/verify Observed: - volume ownership corrected to agent:agent - OpenDesign source install completed - Next/OpenDesign web build completed after increasing Node heap - daemon/UI reachable inside sandbox - OpenCode present on PATH - app-config seeded correctly: {"agentId":"opencode","onboardingCompleted":true,"telemetry":{"metrics":false,"content":false,"artifactManifest":false}} - guest port 7456 reported as mapped by `acq ports` ``` Residual host-side follow-up: ```text A browser attempt against http://127.0.0.1:7456 initially returned ERR_CONNECTION_REFUSED despite `acq ports` reporting a mapping. The verifier was updated to curl the reported host-loopback URL so this cannot be missed in the next live run. ``` ## Rollback Remove `integrations/isolation/acq-kits/opendesign/` and delete the `opendesign` entries from `integrations/isolation/acq-kits/kits.yaml` and `integrations/isolation/acq-kits/README.md`. ## AI Assistance This PR was developed with AI assistance from OpenCode. --------- Co-authored-by: OpenCode Agent <[email protected]> Co-authored-by: OpenCode Agent <[email protected]>

…tries (#519) ## Summary USAi released new models. This regenerates the harness-neutral catalog and both downstream kit artifacts from the live gateway and models.dev feeds, and withholds three models the gateway advertises but cannot serve. Nothing here was hand-edited. Every artifact is generated output. ## Model delta The gateway now lists 26 models against the committed catalog's 20. **Added and shipped** - `claude-sonnet-5.5` - `gpt-6-astra` **Retired upstream** - `gemini-2.5-pro` — no longer in `/v1/models` **Already filtered by the existing non-chat pattern** - `cohere_english_v3`, `text-embedding-005` — embedding models Final shipped catalog: **21 chat models.** ## Three advertised GPT-6 models are withheld The gateway lists these in `/v1/models`, but a `/chat/completions` call to each returns HTTP 404: ```text {"detail":"404: The API deployment for this resource does not exist. …"} ``` | Model | HTTP | |---|---| | `gpt-6-luna` | 404 | | `gpt-6-sol-latest-guardrails-defaultv2` | 404 | | `gpt-6.1-sol-latest-guardrails-defaultv2` | 404 | Each was confirmed with a real completion request and **re-probed** to rule out a transient. Against a control group on the same route, the failure is specific rather than environmental: | Control | HTTP | |---|---| | `gpt-5.6-luna` | 200 | | `gpt-5.6-sol-latest-guardrails-defaultv2` | 200 | | `gpt-5.6-terra` | 200 | | `claude-sonnet-5` | 200 | | `gemini-2.5-flash` | 200 | | `claude-sonnet-5.5` (new) | 200 | | `gpt-6-astra` (new) | 200 | Shipping them would put a model in the picker that fails the moment a user selects it. They are suppressed through the **existing** non-chat exclusion map in both the catalog builder and the kit generator, each carrying the measured reason, so the mechanism is not new and the rationale travels with the code. Remove the entries once the deployments land. This is a gateway-side inconsistency, not a defect in this repo. Reporting it to the USAi team separately so the model list and the deployments can be reconciled at the source. ## Snapshot subset correction `model-catalog-snapshot.json` is a hand-trimmed subset with one model per vendor, guarded by a byte-equality test against the normalizer. It kept `gemini-2.5-pro`, which no longer exists upstream, so that test failed. It moves to `gemini-2.5-flash`, the surviving model on that line, preserving the subset's vendor coverage. The subset is deliberately small; this is the minimum change that keeps the gate meaningful rather than dropping a vendor. ## prime-agent lockstep `prime-agent/files/home/prime-agent-config/models.json` is generated from the same `catalog.json` by the prime-agent emitter. Its byte-exact lockstep test correctly caught the committed file as stale once the catalog moved, so it is regenerated from the emitter rather than edited. That test earning its keep here is worth noting: without it, the prime-agent kit would have silently shipped a stale model list. ## One behavioral note `gpt-6-astra` rejects `max_tokens` below 16: ```text {"detail":"integer below minimum value. Expected a value >= 16, but got 8 instead."} ``` Other models accept 8. A reasoning-model floor. Harmless with the shipped config, which sets no such cap, but it will surface for anyone who sets a very small limit. Not worked around here, since the right fix belongs to whatever sets the cap. ## Verification - `build-catalog.mjs --check` — up to date, 21 models - `usai-provider` tests — **76/76** - `providers/usai` tests — **100/100**, including both byte-exact emitter guards - `validate-kits.py --strict` — 10 kits plus 2 templates valid - `generate_index.py --check` — INDEX current - `git diff --check` — clean Beyond the suites, the shipped JSONC was parsed through the **real** startup merge path (`parseJsonc` from `merge-global-config.mjs`), confirming: - 21 models parse - both working new models present - all three withheld models, `gemini-2.5-pro`, and both embedding models absent - `model` and `small_model` defaults still resolve to models that exist Default selections are unchanged: `usai/claude-sonnet-5` and `usai/claude_4_5_haiku`. `make validate` fails 4 validators (frontmatter schema, cross-reference, network-tier baseline, balanced-allowlist guard). I verified by stashing that **all four fail identically on unmodified `origin/main`** — pre-existing and unrelated to this change. Not introduced here, and not fixed here. ## Downstream Kits are consumed at a pinned ref, so this does not reach users until a patterns release is cut and `GSA-TTS/agentic-coding-quickstart` bumps `PATTERNS_KIT_REF`. That pin currently sits two releases behind, which is a separate problem already being addressed. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Safety Checklist - [x] No secrets, API keys, or tokens. The catalog's `apiKeyEnv` holds the env-var **name**, never a value; a test asserts no key material appears in any emitted artifact. - [x] No PII, CUI, customer data, or internal URLs - [x] Validation run before submitting - [x] Generated artifacts regenerated, not hand-edited ## AI Assistance Disclosure - [x] AI tools were used in this contribution (OpenCode). - [x] I verified all AI-generated content, including every model probe above. --------- Co-authored-by: Bret Mogilefsky <[email protected]>

## Summary Define the validated, provider-neutral files that provider kits give to agent kits. USAi and a fictitious second provider use the same protocol contract, proving consumers do not need USAi-specific branches. ## Issue resolved **#517** - Separates provider connection facts from neutral model catalogs. - Defines stable packaged and projected paths. - Declares the bounded API protocol used by harness-owned static mappings. - Allows optional advisory model recommendations without harness-specific keys. - Validates identifiers, URLs, credentials, files, catalogs, and cross-file references. ## Before and after **Before this PR:** Provider metadata existed in overlapping USAi-specific shapes, and agent kits had no single validated contract or stable location from which to consume it. **After merge:** Every provider kit can publish the same validated facts-and-catalog bundle, including a precise API protocol, while each agent kit retains ownership of its native configuration. ## Verification - Provider contract tests: 143 passed - `make validate-provider-input-contract`: passed - `make validate-kits`: passed - `make test-kits`: passed - Scoped Ruff and `git diff --check`: passed - Independent adversarial review: speculative platform adapters removed; no remaining blockers ## Security impact The validator fails closed on unsafe paths, symlink escapes, credential ownership mismatches, malformed URLs, unsupported protocols, duplicate IDs, dangling recommendations, and invalid normalizer output. Protocol values are mapped by consumers through static allowlists, not dynamic imports or execution. No credential values are stored. ## Relationships Fixes: #517 Parent epic: #357 Blocks: #516 and GSA-TTS/agentic-coding-quickstart#600 ## Rollback Revert this PR. It is additive and does not cut over existing OpenCode configuration generation.

🤖 I have created a release *beep* *boop* --- ## [1.15.0](v1.14.0...v1.15.0) (2026-10-21) ### Features * **acq-kits:** add OpenDesign browser kit ([#438](#438)) ([83e202f](83e202f)) * **isolation:** define provider-neutral input contract ([#518](#518)) ([28a8ebd](28a8ebd)) * **usai-catalog:** refresh models and withhold undeployed gateway entries ([#519](#519)) ([03421e0](03421e0)) ### Bug Fixes * **usai-catalog:** refuse a near-miss instead of enriching from the wrong model ([#484](#484)) ([c3a6dac](c3a6dac)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Navigation Menu

Repository navigation

Comparing changes

Open a pull request

Commits on Oct 21, 2026

This comparison is taking too long to generate.

Uh oh!