Skip to content

Commit 53ee2e5

Browse files
committed
docs(devlog): audit every id in lidge-jun#964's list — 26 ship, 6 reversed, 32 dropped
003 made per-id verification a gating step. This is that audit, run against build.nvidia.com model pages and the NIM LLM/Visual API indexes on 2026-08-04. lidge-jun#964 submitted ~64 ids. Fewer than half survive: 26 confirmed text-only (explicit 'Input Modalities: Text') — these ship 6 confirmed image-capable — moved to the vision list 32 unverified or absent from NVIDIA's catalog — dropped The 26 include z-ai/glm-5.2, deepseek-v4-flash/pro and the nemotron-3 family, so the models issue lidge-jun#956 actually names are all fixed. No seventh false positive was found, which is the first evidence the correction has converged rather than merely advanced. The 32 dropped are mostly delisted models — harmless in isolation, since nobody can route to a model NVIDIA no longer serves. But the set includes nvidia/nemotron-nano-3-30b-a3b, a reversed-name typo of the real nvidia/nemotron-3-nano-30b-a3b which the same list also spells correctly, and mistralai/mixtral-8x22b-v0.1 where NVIDIA documents mixtral-8x22b-instruct-v0.1. Half the list was assembled rather than verified; the six reversed entries were the visible damage, this is the extent of it. Kimi is now split correctly across two independent axes: k2.5 and k2.6 join the vision list, k2-thinking and k2-instruct stay text-only, and all four remain in NVIDIA_NIM_KIMI_MODELS for reasoning suppression. google/codegemma-7b verifies while google/codegemma-1.1-7b does not — adjacent names, opposite outcomes, which is why name-based classification was rejected.
1 parent d422273 commit 53ee2e5

3 files changed

Lines changed: 113 additions & 3 deletions

File tree

‎devlog/_plan/260804_stack7_service_vision/000_scope.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,9 @@ answered in `020` before any code is written.
8080
| Doc | Contents |
8181
|---|---|
8282
| `001_audit_response.md` | A-gate FAIL — five blockers, synthesis, and what changed |
83+
| `002_audit_response_r2.md` | A-gate FAIL round 2 — root cause and the design that follows |
84+
| `003_audit_response_r3.md` | A-gate FAIL round 3 — a sixth false positive changes the method |
8385
| `010_nim_vision_classification.md` | #964 reconstruction — the classification design and its diff |
86+
| `011_nim_id_audit.md` | per-id verification of #964's list: 26 ship, 6 reversed, 32 dropped |
8487
| `020_service_repair_path.md` | #970 reconstruction — call-site inventory and the after-stop safety proof |
8588
| `030_merge_and_close_sequence.md` | bottom-up merge order, retargeting, and issue closure evidence |

‎devlog/_plan/260804_stack7_service_vision/010_nim_vision_classification.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,7 @@ google/gemma-4-31b-it
194194
google/diffusiongemma-26b-a4b-it
195195
minimaxai/minimax-m3
196196
moonshotai/kimi-k2.6
197+
moonshotai/kimi-k2.5
197198
stepfun-ai/step-3.7-flash
198199
thinkingmachines/inkling
199200
mistralai/mistral-medium-3.5-128b
@@ -222,9 +223,12 @@ edits — the reason this draft is implementable where draft 2 was not.
222223
1. `NVIDIA_NIM_VISION_MODELS` — the 15 verified ids, with a comment recording the
223224
verification date, per-model source, and the standing instruction that a new
224225
NIM model must be classified deliberately, never assumed.
225-
2. `NVIDIA_NIM_NO_VISION_MODELS` — the text-only enumeration, seeded from #964's
226-
list with the five false positives removed and the existing
227-
`NVIDIA_NIM_KIMI_MODELS` reconciled (kimi-k2.6 moves to the vision list).
226+
2. `NVIDIA_NIM_NO_VISION_MODELS` — **the 26 ids verified in `011_nim_id_audit.md`
227+
and no others.** #964 submitted ~64; six were image-capable and 32 could not
228+
be verified against any current NVIDIA page, so they are dropped rather than
229+
carried. `NVIDIA_NIM_KIMI_MODELS` is reconciled: k2.5 and k2.6 move to the
230+
vision list while k2-thinking and k2-instruct stay text-only, and all four
231+
remain in the reasoning-suppression list, which is an independent axis.
228232
3. `modelInputModalities` pinning `["text","image"]` for each vision id (B2),
229233
following `ZHIPU_BIGMODEL_INPUT_MODALITIES` (`src/providers/registry.ts:327-331`).
230234
4. Set `noVisionModels` on the `nvidia` entry and extend the entry comment with
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# 011 — Per-id audit of #964's list against NVIDIA documentation
2+
3+
`003` made per-id verification a gating step: an id ships only if NVIDIA's own
4+
documentation says text-only, otherwise it is dropped. This is that audit,
5+
performed 2026-08-04 against `build.nvidia.com` model pages,
6+
`docs.api.nvidia.com/nim/reference/*`, and the
7+
[LLM APIs index](https://docs.api.nvidia.com/nim/reference/llm-apis)
8+
cross-checked against the
9+
[Visual Models index](https://docs.api.nvidia.com/nim/reference/visual-models-apis).
10+
11+
## Result
12+
13+
| Bucket | Count |
14+
|---|---|
15+
| Confirmed text-only — ship | 26 |
16+
| Confirmed image-capable — moved to the vision list | 6 |
17+
| Unverified / absent from NVIDIA's catalog — dropped | 32 |
18+
19+
#964 submitted ~64 ids. **Fewer than half survive verification.**
20+
21+
## Confirmed text-only (26) — these ship
22+
23+
Each carries an explicit `Input Modalities: Text`, `Input Type(s): Text`, or
24+
equivalent on its NVIDIA page:
25+
26+
```
27+
deepseek-ai/deepseek-v4-flash nvidia/llama-3.1-nemotron-nano-8b-v1
28+
deepseek-ai/deepseek-v4-pro nvidia/llama-3.1-nemotron-ultra-253b-v1
29+
google/codegemma-7b nvidia/llama-3.3-nemotron-super-49b-v1
30+
meta/llama-3.1-70b-instruct nvidia/llama-3.3-nemotron-super-49b-v1.5
31+
meta/llama-3.1-8b-instruct nvidia/nemotron-3-nano-30b-a3b
32+
meta/llama-3.2-1b-instruct nvidia/nemotron-3-super-120b-a12b
33+
meta/llama-3.2-3b-instruct nvidia/nemotron-3-ultra-550b-a55b
34+
meta/llama-3.3-70b-instruct nvidia/nemotron-mini-4b-instruct
35+
meta/llama2-70b nvidia/nvidia-nemotron-nano-9b-v2
36+
mistralai/mistral-7b-instruct-v0.3 openai/gpt-oss-120b
37+
mistralai/mistral-nemotron openai/gpt-oss-20b
38+
moonshotai/kimi-k2-thinking poolside/laguna-xs-2.1
39+
moonshotai/kimi-k2-instruct z-ai/glm-5.2
40+
```
41+
42+
`z-ai/glm-5.2`, `deepseek-v4-flash`/`-pro` and the nemotron-3 family are the ids
43+
issue #956 actually names, so the reported bug is fixed for every model in the
44+
report.
45+
46+
Two notes for the implementation:
47+
48+
- `moonshotai/kimi-k2-thinking` and `kimi-k2-instruct` are text-only and stay in
49+
`NVIDIA_NIM_KIMI_MODELS` for reasoning suppression. Only k2.5 and k2.6 move to
50+
the vision list. The two axes are independent fields
51+
(`registry.ts:1238-1240`), verified by probe.
52+
- `google/codegemma-7b` is confirmed while `google/codegemma-1.1-7b` is not — the
53+
point-release id has no current page. Near-identical names, different outcomes;
54+
exactly why name-based reasoning was rejected.
55+
56+
## Confirmed image-capable (6) — moved to the vision list
57+
58+
`thinkingmachines/inkling`, `minimaxai/minimax-m3`, `moonshotai/kimi-k2.6`,
59+
`moonshotai/kimi-k2.5`, `stepfun-ai/step-3.7-flash`,
60+
`mistralai/mistral-medium-3.5-128b`. Sources in `010` and `003`.
61+
62+
The audit found **no seventh** false positive among the remaining ids. That is
63+
the first evidence that the correction has converged rather than merely advanced.
64+
65+
## Unverified — dropped (32)
66+
67+
Dropped rather than carried, per `003`. Dropping an id costs today's behavior;
68+
carrying an unverified one risks the silent substitution this whole unit exists
69+
to prevent.
70+
71+
**Absent from NVIDIA's current catalog (27).** `01-ai/yi-large`,
72+
`ai21labs/jamba-1.5-large-instruct`, `aisingapore/sea-lion-7b-instruct`,
73+
`bigcode/starcoder2-15b`, `databricks/dbrx-instruct`,
74+
`deepseek-ai/deepseek-coder-6.7b-instruct`, `google/codegemma-1.1-7b`,
75+
`google/gemma-2b`, `google/recurrentgemma-2b`, the four `ibm/granite-*`,
76+
`meta/codellama-70b`, `microsoft/phi-3.5-moe-instruct`,
77+
`mistralai/codestral-22b-instruct-v0.1`, `mistralai/mistral-large`,
78+
`nv-mistralai/mistral-nemo-12b-instruct`,
79+
`nvidia/llama-3.1-nemotron-51b-instruct`, `nvidia/llama3-chatqa-1.5-70b`,
80+
`nvidia/mistral-nemo-minitron-8b-8k-instruct`, `nvidia/nemotron-4-340b-instruct`,
81+
the four `writer/palmyra-*`, `zyphra/zamba2-7b-instruct`.
82+
83+
**Page exists, modality field absent (2).** `moonshotai/kimi-k2-instruct-0905`
84+
and `nvidia/llama-3.1-nemotron-70b-instruct` — both deprecated endpoints whose
85+
specifications omit modalities.
86+
87+
**Id does not match NVIDIA's catalog (3).** `mistralai/mistral-large-2-instruct`
88+
(no such id), `mistralai/mixtral-8x22b-v0.1` (NVIDIA documents
89+
`mixtral-8x22b-instruct-v0.1`), and `nvidia/nemotron-nano-3-30b-a3b` — a
90+
**reversed-name typo** of the real `nvidia/nemotron-3-nano-30b-a3b`, which #964
91+
also lists correctly. Both spellings were in the submitted list; only one is a
92+
real model.
93+
94+
## What this audit demonstrates
95+
96+
The dropped ids are mostly harmless-looking noise — a delisted model classified
97+
wrongly costs nothing because nobody can route to it. But 32 of 64 entries being
98+
unverifiable, including a typo'd duplicate, is the measure of how much of #964's
99+
list was assembled rather than verified. The six reversed entries were the
100+
visible damage; this is the extent of it.
101+
102+
It also bounds the fix honestly. 26 verified ids is a real fix for real models,
103+
not a claim about NIM as a whole.

0 commit comments

Comments
 (0)