Skip to content

Commit 7612e4c

Browse files
authored
fix(routing): source capability evidence from explicit catalog provenance (#1799)
* docs(devlog): plan the local-model capability evidence and plugin routing unit * docs(devlog): fold audit round 1 blockers into the capability provenance design * docs(devlog): fold audit round 2 precision fixes into the phase docs * docs(devlog): make the remote exact-head verifier self-contained * docs(devlog): fold audit round 4 blockers and link issues #1796/#1797 * docs(devlog): fold audit round 5 blockers (fail-closed remote steps, antigravity tri-state) * docs(devlog): make each remote verifier step self-contained; widen phase 1 scope * docs(devlog): record the six-round audit ledger and verdict-capture caveat * docs(devlog): capture jawcode generated metadata in provenance (round 7) * docs(devlog): own the lookup extraction, add the context-cap regression * docs(devlog): record rounds 7-8 and the writer-vs-producer lesson * fix(routing): source capability evidence from explicit catalog provenance (#1796) candidateCapabilityEvidence() read the cached Codex catalog through a filter expecting `id`/`provider`/`contextWindow`/`inputModalities`, while the catalog writer emits `slug`/`context_window`/`input_modalities`. Every row was discarded -- 17 of 17 on a live install -- so the catalog branch of the evidence chain was dead code. It stayed invisible because the catalog is the fourth fallback: any provider declaring inline capability maps is answered earlier. A model registered through `ocx models add` has no such maps, so it routed as image-blind and context-unknown. Reading the written fields directly would have been wrong. ensureStrictCatalogFields synthesizes `input_modalities: ["text"]` and `context_window: 128000` so Codex's strict parser accepts the file, so their presence cannot distinguish an assertion from a placeholder, and reading them would turn unknown into a confident `image: false` -- the opposite of this module's "unknown is not zero" contract. Instead applyCatalogModelMetadata now stamps `opencodex_capability_provenance` carrying only values a real source asserted, plus exact provider/model_id, and capability.ts reads that block and nothing else. Two real sources exist and both are consulted: the CatalogModel and the generated jawcode metadata table, whose lookup is now exported from parsing.ts so the two callers cannot drift. The generated context is capped the same way applyCatalogMetadata caps it. Three secondary defects fixed along the way: - The adapter tool fallback was gated on `catalogRow === undefined`, which was only safe while the lookup never matched. Repairing the lookup would have silently revoked `tools: true` for every openai-chat and anthropic candidate. - Synthesized combo rows (a generic 128k/text fallback) are excluded from the stamp; their values are placeholders, not assertions. - parseAntigravityAvailableModels collapsed an absent `supportsImages` into `["text"]`, making "nobody said" indistinguishable from "the provider said no". The tri-state is restored; the strict catalog still gets its default downstream. Evidence: the new test fails 5/10 without the fix and passes 10/10 with it. End to end, with the lidge provider's inline maps removed so the catalog is the only source, evidence goes from `{tools:true}` to `{contextWindow:262144, image:true, tools:true}`. Plan and eight-round audit: devlog/_plan/260816_local_model_capability_and_plugin_routing/ * fix(catalog): read llama.cpp served context from meta.n_ctx (#1797) A llama.cpp server reports its context under `meta.n_ctx` (what the server was started with) and `meta.n_ctx_train` (the model's trained maximum). Neither was in the recognized context list, so a correct local server produced no context evidence at all. Both are appended LAST in the precedence chain, so any provider already supplying a recognized field keeps its current behavior. `n_ctx` is preferred over `n_ctx_train` because routing must not promise a window the running server will refuse. The image half of #1797 is deliberately NOT fixed here. The `multimodal` token lives in the Ollama-style `models[]` array while extractProviderModelItems reads only `data[]` envelopes, and its comment is explicit that a stray `models` key must not be trusted. Even a hand-merged item would stay image-unknown, because `multimodal` is not among the recognized capability strings. Both halves need an identity-safe join and a capability mapping, which is its own audited change. The fourth test characterizes that gap so the follow-up has a live witness. Verified: tests 1, 2 and 4 fail without this change and pass with it; test 3 passes either way and guards the precedence ordering. * docs(devlog): record the implementation, verification evidence, and CI baseline * docs: address CodeRabbit review findings on #1799 Five threads, all accepted: - Move the browser/Computer Use plugin routing guidance out of the untracked host file and into the tracked AGENTS_INSTALL.md. Listing ~/.codex/AGENTS.md as shipped was wrong: it is not in the PR and cannot be reproduced from a clone. The host copy is now recorded as host-only verification. - Mark the first provenance snippet in 010 as superseded and add one authoritative "Final stamp algorithm" section carrying the combo guard, the generated-metadata fallback, and the context cap, verbatim from effort.ts. Three separate audit rounds amended that snippet; leaving the original inline invited implementing from the wrong one. - Correct the 004 ledger scope from six rounds to eight, matching the rounds it actually records. - Renumber the 020 accept criteria, which restarted at 2 and repeated 3. - Add console language tags to the two transcript fences (MD040). * docs: address the second CodeRabbit round on #1799 - Criterion 5 in 020 claimed the full suite was green. It was not: the recorded run has seven GUI dependency failures. Restated to the honest bar - the targeted suites pass, the full run carries only documented environment failures, and no failure is attributable to this change. - 030 still selected ~/.codex/AGENTS.md as the placement and verified it there, contradicting the correction already recorded in 040. Marked the placement superseded and pointed every row at the tracked AGENTS_INSTALL.md. - Backticked the bare PR references in the triage doc (MD018).
1 parent e1769b5 commit 7612e4c

18 files changed

Lines changed: 2132 additions & 22 deletions

‎AGENTS_INSTALL.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,35 @@ Regression coverage: `tests/startup-prompt.test.ts`,
7575
If a future action spends the user's identity, credits, or reputation, gate it
7676
the same way rather than relying on a prompt an agent can answer, and document
7777
it here.
78+
79+
## Browser and Computer Use plugins: entry point
80+
81+
This is not a consent boundary — it is a routing fact that costs agents real
82+
time when they do not know it, and it belongs here because it applies to an
83+
agent *operating* opencodex rather than changing it.
84+
85+
The Chrome, Browser, and Computer Use plugins run **only** through the
86+
privileged Node REPL tool (`mcp__node_repl__js`). Call it directly. If it is not
87+
in the current tool set, search available tools for `node_repl js` before
88+
concluding anything is unavailable.
89+
90+
These do not work and are not worth attempting:
91+
92+
- `node` / `node -e` importing the plugin's `scripts/browser-client.mjs`. It
93+
refuses with `Browser use requires privileged node_repl capabilities` — the
94+
bundle reads `globalThis.nodeRepl` and ships its own `process` shim, both
95+
injected by the privileged REPL host and by nothing else.
96+
- Filesystem searches for `@oai/sky`. Computer Use injects it at runtime; there
97+
is no package on disk, so `find` and `mdfind` can only ever come back empty.
98+
- `osascript` / AppleScript / JXA as a substitute for the plugin API.
99+
100+
A failed shell attempt is evidence about the shell, not about plugin
101+
availability.
102+
103+
Smaller local models misroute here for a specific reason worth naming: the
104+
bundled Chrome skill asks an agent not to *mention* the REPL tool in
105+
user-facing prose while simultaneously requiring it to *use* that tool. Both
106+
hold at once. The naming restriction governs what you say to the user; it never
107+
means the tool is off-limits. A model that resolves the tension by avoiding the
108+
tool will exhaust every shell path and then report the plugins as unavailable,
109+
which is what prompted writing this down.
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# 260816 — Local-model capability evidence and browser-plugin routing
2+
3+
## Objective
4+
5+
A local Qwen model (`lidge/qwen3.8-27b-nvfp4`, llama.cpp behind an
6+
`openai-chat` adapter) could not drive the Chrome or Computer Use browser
7+
plugins. Investigating that failure surfaced a defect in
8+
`src/routing/capability.ts` that is **not local-model specific**: the cached
9+
Codex catalog is read on every policy-routed request and then discarded in
10+
full, because the reader expects a field shape the catalog writer never
11+
produces.
12+
13+
This unit fixes the capability-evidence defect, records the live-catalog
14+
ingestion gap, and writes durable routing guidance for weaker local models
15+
that must reach the browser plugins through the privileged Node REPL tool.
16+
17+
## Constraints
18+
19+
- `src/routing/capability.ts` is on the request path for every policy-routed
20+
request. The memoized catalog read exists precisely so the parse cost is not
21+
paid per candidate; a fix must not turn it into a per-candidate parse.
22+
- "Unknown is not zero" is the module's stated contract (file header): a
23+
dimension without canonical evidence must stay `undefined`, never `false`.
24+
The fix must not convert a missing catalog field into a negative assertion.
25+
- `src/routing/capability.ts` is reachable from `src/router.ts`, so the
26+
core/lab boundary in `tests/core-lab-boundary.test.ts` applies: no import
27+
may reach `src/lab/`.
28+
- Out of scope: promotion to `main`, npm publish, GUI redesign, unrelated
29+
provider adapters.
30+
31+
## Dependency-ordered work-phase map
32+
33+
The order is build-order, not effort order: the catalog reader is the
34+
foundation both later phases depend on.
35+
36+
| Phase | Doc | Depends on | Independently verifiable by |
37+
|-------|-----|------------|------------------------------|
38+
| 1 | `010_catalog_row_shape.md` | — | New focused test: catalog-sourced evidence survives for a routed row |
39+
| 2 | `020_live_capability_ingestion.md` | Phase 1 | Focused test driving the observed llama.cpp `/v1/models` payload |
40+
| 3 | `030_local_model_plugin_routing.md` | — (docs surface) | The written guidance resolves on the documented path |
41+
42+
Phase 3 has no code dependency on 1 or 2 and could land in any order; it is
43+
listed last because it is documentation, not because it is smaller.
44+
45+
## Source-of-truth sync target (SOT-SYNC-01)
46+
47+
`structure/` holds maintainer invariants. Phase 1 changes how routing evidence
48+
is sourced, so C patches the structure note that describes routing evidence if
49+
one exists; if none does, the D summary recommends creating it.
50+
51+
## Research documents
52+
53+
- `001_capability_evidence_defect.md` — the reproduction, the field-shape
54+
mismatch, and why every existing test passes over it.
55+
- `002_local_model_plugin_failure.md` — why the local model could not reach
56+
the Chrome and Computer Use plugins.
57+
58+
## Filed issues
59+
60+
| Issue | Covers | Fixed by |
61+
|-------|--------|----------|
62+
| [#1796](https://github.com/lidge-jun/opencodex/issues/1796) | Routing discards every catalog row (field-shape mismatch) | Phase 1 |
63+
| [#1797](https://github.com/lidge-jun/opencodex/issues/1797) | llama.cpp `multimodal` token + dual-envelope join | Deferred; Phase 2 ships the context half only |
64+
65+
Phase 3 files no issue: it is host-side guidance with no opencodex defect
66+
behind it (see `002_local_model_plugin_failure.md`).
Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# 001 — Catalog capability evidence never reaches routing
2+
3+
Research document. No diffs here; the fix design is `010_catalog_row_shape.md`.
4+
5+
## Symptom that started this
6+
7+
`lidge/qwen3.8-27b-nvfp4` was registered through the supported CLI:
8+
9+
```
10+
$ ocx models add lidge qwen3.8-27b-nvfp4 --context-window 262144 --modalities text,image
11+
Error: custom model "lidge/qwen3.8-27b-nvfp4" already exists
12+
```
13+
14+
The row was already present and complete in `~/.opencodex/config.json`:
15+
16+
```json
17+
{ "id": "83ca0b4c-06bb-475d-b585-6c47b9d6be71", "provider": "lidge",
18+
"modelId": "qwen3.8-27b-nvfp4", "displayName": "Qwen3.8 27B NVFP4 (lidge 5090)",
19+
"contextWindow": 262144, "inputModalities": ["text", "image"] }
20+
```
21+
22+
It also reached the on-disk Codex catalog correctly, as
23+
`/Users/jun/.codex/opencodex-catalog.json`:
24+
25+
```json
26+
{ "slug": "lidge/qwen3.8-27b-nvfp4", "context_window": 262144,
27+
"input_modalities": ["text", "image"], "supports_parallel_tool_calls": true }
28+
```
29+
30+
Yet routing evidence carried neither value. NOTE (audit round 1, B9): this
31+
one-liner reproduces only with the provider's `modelContextWindows` and
32+
`modelInputModalities` ABSENT. They were added by hand later while
33+
diagnosing, so on today's live config the earlier branches win and the
34+
symptom is masked. The 17-to-0 catalog proof below is independent of that
35+
and still reproduces exactly. Use a catalog-only fixture to reproduce:
36+
37+
```
38+
candidateCapabilityEvidence(config, "lidge", "qwen3.8-27b-nvfp4")
39+
=> { tools: true, serviceTier: "unsupported", encryptedCodexTasks: false }
40+
```
41+
42+
No `image`. No `contextWindow`. A model registered through the documented
43+
path is image-blind to the router.
44+
45+
## Root cause: reader and writer disagree on field shape
46+
47+
`src/routing/capability.ts` `cachedCatalogModels()` filters rows with:
48+
49+
```ts
50+
typeof model.id === "string" && typeof model.provider === "string"
51+
```
52+
53+
and then reads `model.contextWindow` / `model.inputModalities`.
54+
55+
The catalog file has none of those four fields. Its actual keys are
56+
`slug` (a combined `provider/id`), `context_window`, and `input_modalities`.
57+
Verified against the live file:
58+
59+
```
60+
KEYS: slug,display_name,description,default_reasoning_level,
61+
supported_reasoning_levels,...,input_modalities,...,context_window,
62+
max_context_window,auto_compact_token_limit,...
63+
provider? undefined id? undefined context_window? 262144
64+
```
65+
66+
So the filter rejects every row:
67+
68+
```
69+
TOTAL: 17 | SURVIVING capability.ts filter: 0
70+
```
71+
72+
**This is not a local-model bug.** All 17 rows are discarded — native OpenAI
73+
rows, `anthropic/claude-opus-5`, `xai/grok-4.6`, everything. The whole
74+
`catalogRow` branch of the evidence chain is dead code in practice.
75+
76+
## Why nobody noticed
77+
78+
The catalog is the *fourth* fallback. For a provider with populated config
79+
maps the earlier branches answer first:
80+
81+
```
82+
provider.modelContextWindows[id] ?? provider.contextWindow
83+
?? registryEntry.modelContextWindows[id] ?? catalogRow.contextWindow ?? ...
84+
```
85+
86+
`kimi`, `anthropic`, `xai`, and `alibaba-token-plan-intl` all declare
87+
`modelContextWindows` and `modelInputModalities` inline, so their evidence
88+
looks correct and the dead branch stays invisible. Only a provider that
89+
relies on the catalog — exactly what `ocx models add` produces — is exposed.
90+
91+
Confirmed by contrast on the same tree:
92+
93+
```
94+
xai/grok-4.6 => contextWindow 500000, image true (provider maps)
95+
anthropic/claude-opus-5 => contextWindow 1000000, no image (provider maps)
96+
lidge/qwen3.8-27b-nvfp4 => nothing (catalog only)
97+
```
98+
99+
`claude-opus-5` is itself a smaller instance of the same hole: the provider
100+
block declares `modelContextWindows` but no `modelInputModalities`, and the
101+
catalog that could have supplied `image` is discarded.
102+
103+
## Why the test suite is green
104+
105+
`tests/routing-profile.test.ts` and `tests/routing-compatibility.test.ts`
106+
construct `capability` objects inline:
107+
108+
```ts
109+
{ provider: "a", model: "m1", capability: { contextWindow: 200000, tools: true } }
110+
```
111+
112+
They exercise the *policy evaluator* with pre-made evidence and never call
113+
`candidateCapabilityEvidence`, so no test ever reads a real catalog file.
114+
The assembly step between the catalog on disk and the evaluator is untested.
115+
That is the coverage gap this unit closes, and it is why "all tests green"
116+
said nothing about this defect.
117+
118+
## Field-chain note (PLAN-FIELD-CHAIN-01)
119+
120+
The chain for a custom model is:
121+
122+
| Stage | Path | State |
123+
|-------|------|-------|
124+
| creation | `src/cli/models.ts` `ocx models add` | works |
125+
| serialization | `config.customModels[]` -> `src/codex/catalog/provider-fetch.ts:1758` | works |
126+
| on-disk form | `~/.codex/opencodex-catalog.json` (`slug`, `context_window`, `input_modalities`) | works |
127+
| consumer | `src/routing/capability.ts` `cachedCatalogModels()` | **broken — expects `provider`/`id`/`contextWindow`/`inputModalities`** |
128+
129+
Only the last stage is wrong, which is why the value is visible everywhere a
130+
human looks (config, CLI, catalog file) and absent exactly where routing
131+
decides.
132+
133+
## Post-audit correction (round 1)
134+
135+
The fix originally proposed here — read `context_window` and
136+
`input_modalities` straight off the row — is WRONG and was rejected in
137+
audit. `ensureStrictCatalogFields()` synthesizes both fields for Codex's
138+
strict parser, so reading them would convert unknown into `image:false`
139+
and a fabricated `128000`. See `003_audit_synthesis_round1.md` B2 and the
140+
provenance design in `010_catalog_row_shape.md`.
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# 002 — Why a local model could not reach the browser plugins
2+
3+
Research document. Environment observation, not an opencodex code defect;
4+
the durable-guidance design is `030_local_model_plugin_routing.md`.
5+
6+
## What was observed
7+
8+
A local Qwen model asked to browse with the Chrome and Computer Use plugins
9+
tried, in order: `node -e "import('@oai/sky')"`, `find` and `mdfind` sweeps
10+
for a `sky` package, `osascript` against Google Chrome, and finally a
11+
hand-written `/tmp/chrome_probe.mjs` importing the plugin's
12+
`browser-client.mjs` directly. It then reported the tooling unavailable.
13+
14+
The tooling was available the whole time.
15+
16+
## Why every shell attempt fails by construction
17+
18+
`scripts/browser-client.mjs` in the Chrome plugin is a ~1.15 MB bundle that
19+
expects a privileged host. Importing it from an ordinary Node process
20+
resolves its exports and then refuses at runtime:
21+
22+
```console
23+
$ node -e "import('.../scripts/browser-client.mjs').then(m => m.setupBrowserRuntime())"
24+
RUNTIME FAIL: Browser use requires privileged node_repl capabilities
25+
```
26+
27+
The bundle carries its own `process` shim and reads `globalThis.nodeRepl`;
28+
those are injected by the privileged REPL host, not by Node. Computer Use is
29+
stricter still — `@oai/sky` has no on-disk package at all, so filesystem
30+
searches for it can only ever come back empty.
31+
32+
The single working entry point is the `mcp__node_repl__js` tool. Driving the
33+
same plugin through it succeeded immediately in this session: browser bound,
34+
`chrome.user.openTabs()` returned the live tab list, navigation and a
35+
screenshot both worked.
36+
37+
## Why a weaker model misroutes here
38+
39+
The Chrome skill deliberately obscures its own mechanism for user-facing
40+
reasons:
41+
42+
> "Never mention `Node REPL`, `node_repl`, `REPL`, JavaScript sessions ...
43+
> unless a user is asking for that exact information."
44+
45+
while simultaneously requiring it:
46+
47+
> "Run browser setup code through the Node REPL `js` tool ... If it is not
48+
> already available, use tool discovery for `node_repl js`."
49+
50+
A model with strong instruction-following holds both. A weaker one resolves
51+
the conflict by treating the named tool as off-limits and substitutes a
52+
shell, which is exactly the observed failure. The skill text plus its
53+
`documentation()` payload is also ~60 KB before any work begins, which
54+
compounds the problem for a small local context.
55+
56+
## Scope boundary
57+
58+
This is host/skill routing behavior, not opencodex runtime behavior. Nothing
59+
in `src/` can fix it. The actionable output is durable guidance on a surface
60+
a local model actually reads, and that is all Phase 3 does.

0 commit comments

Comments
 (0)