Skip to content

Commit 7794485

Browse files
authored
Merge pull request #2790 from lidge-jun/codex/ocx-gap-closure
fix(cli): close the gap-audit findings where the CLI misreported its own outcome
2 parents 6d0d7f0 + ba79798 commit 7794485

14 files changed

Lines changed: 1079 additions & 26 deletions
Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# 090 — wp10: close the gap-audit findings
2+
3+
The wp9 close-out gap audit answered YES to "does a material gap remain in agentic CLI
4+
control of GUI capabilities", with three High findings and one Medium. Each was
5+
independently reproduced against source before this plan was written, so none of it is
6+
taken on the reviewer's word.
7+
8+
Reproductions (run from the branch, not the released `ocx` on PATH — the first attempt at
9+
this measurement used the installed binary and got "Unknown command: capabilities", which
10+
is a stale build, not evidence):
11+
12+
```
13+
bun run src/cli/index.ts capabilities --route /api/keys --json -> {"capabilities": []}, exit 4
14+
bun run src/cli/index.ts logout --json -> "Logged out of --json.", exit 0
15+
bun run src/cli/index.ts definitely-not-a-command -> exit 1
16+
capabilities declared: 26 routes declared: 207
17+
```
18+
19+
## 090.1 — `logout --json` silently no-ops (highest severity)
20+
21+
`src/cli/dispatch.ts` `logout` takes `args[1]` as the provider name with no flag parsing, so
22+
`ocx logout --json` lowercases `--json`, calls `removeCredential("--json")`, prints
23+
`Logged out of --json.` and exits 0. An agent gets a success exit for an operation that did
24+
nothing, which is the worst failure mode in this whole unit: silent, and indistinguishable
25+
from success.
26+
27+
**Plan audit correction.** My first reading called `removeCredential("--json")` harmless
28+
because `store.ts:598` returns early on an unknown key. That is wrong for a store this code
29+
does not control: `normalizeAuthStore` (`store.ts:346-355`) copies **every** top-level key
30+
it finds, so a hand-edited, legacy, or corrupted `auth.json` containing a `--json` key would
31+
have its active account deleted — and the key removed entirely if that was its last account.
32+
The severity is therefore "can destroy credentials in an unusual but reachable store", not
33+
"wastes a call". Recorded because it changes what the regression has to prove.
34+
35+
Fix: parse flags out of argv **before any store I/O**, and split the outcomes the vocabulary
36+
already distinguishes rather than collapsing them:
37+
38+
| Case | Exit |
39+
|---|---|
40+
| omitted provider, unknown flag, invalid argument | 2 (usage), before touching the store |
41+
| valid provider name with no stored credential | 4 (not found) |
42+
| removed | 0 |
43+
44+
The regression must assert that a malformed invocation causes **zero** store mutation, not
45+
merely a non-zero exit.
46+
47+
## 090.2 — three GUI capabilities have no CLI equivalent
48+
49+
| GUI call | CLI today | Missing verb |
50+
|---|---|---|
51+
| `POST /api/oauth/logout` | `ocx logout` only calls `removeCredential` locally | API-backed logout |
52+
| `GET /api/system/codex-app-server`, `POST /api/system/codex-restart` | `sync --restart-codex` only as a side effect after a write | `system codex-app-server status` / `restart` |
53+
| `GET /api/claude-desktop/status` | `claude desktop show` emits locally-built state only | `claude desktop status` |
54+
55+
The OAuth one is not cosmetic. The route does five things past `removeCredential`:
56+
`reconcileLiveStateStores`, `clearLoginState`, `clearModelCache`,
57+
`clearGatherRoutedModelsInflight`, and quota-cache eviction
58+
(`oauth-account-routes.ts:228-243`). A CLI logout that skips them leaves a running proxy
59+
serving a removed credential's cached models and quota, so the local-only path is a
60+
correctness gap and not just a parity gap.
61+
62+
**Plan audit correction — the stopped-proxy contract was missing, and it is the whole
63+
design question.** An API-backed verb needs a live proxy; discovery failure throws
64+
"Proxy is not running" (`runtime-api.ts:43`). So this phase must state the behaviour:
65+
66+
- Proxy running: go through `POST /api/oauth/logout` so the five invalidations happen.
67+
- Proxy not running: exit non-zero, say how to start it, and **mutate nothing**.
68+
69+
No automatic local fallback. A best-effort fallback is actively dangerous here: if a proxy
70+
is alive but momentarily undiscoverable, falling back would delete the credential on disk
71+
while that proxy keeps serving its cached models, login state and quota — the precise
72+
divergence the API path exists to prevent. An offline mode is acceptable only as an explicit
73+
opt-in that proves no proxy is live, not as error recovery.
74+
75+
Two further under-specifications to settle before implementing:
76+
77+
- The route reports success even when nothing was removed, because `removeCredential`
78+
returns no disposition (`oauth-account-routes.ts:231`). Exit 4 is impossible until the
79+
store or route returns a removed/not-found result. Do it there, not via a CLI preflight
80+
read — a preflight is racy by construction.
81+
- Define whether logout removes only the active account or all of them, and whether legacy
82+
`ocx logout` becomes an alias or keeps its offline behaviour. Silence here is how two
83+
commands end up meaning different things.
84+
85+
## 090.3 — the parity gate only checks one direction
86+
87+
`tests/cli-capabilities.test.ts` asserts every capability's route exists. It never asserts
88+
the converse, so 139 non-exempt routes carry no capability entry and nothing fails. The
89+
observable consequence is that `capabilities --route` — the agent's discovery entry point —
90+
returns empty and exits 4 for `/api/keys` and `/api/routing-profiles` while `ocx access key`
91+
and `ocx route policy` work.
92+
93+
**Plan audit correction — the allowlist was the wrong mechanism, and the counts were off.**
94+
`ManagementRoute` already carries an `exempt` field with a typed `ExemptionReason` union
95+
(`route-registry.ts:24-46`, `:68`) covering exactly the categories a parallel allowlist would
96+
have re-invented: `session-only`, `disabled`, `capability-principal`, `test-seam`,
97+
`local-transport`, `dead`. A second list would duplicate the source of truth and relocate the
98+
omission problem rather than close it.
99+
100+
The reverse gate is therefore: **every route is either covered by `capabilityRouteKeys()` or
101+
carries a justified `route.exempt`** — never neither, and ideally never both.
102+
103+
The audit's fresh counts also correct mine: 206 routes (I said 207), 35 capability-covered
104+
route keys, 32 exempt, 139 unexplained. More importantly it disproved my framing of what
105+
those 139 are: 122 of them are already referenced from CLI source, and of the remaining 17
106+
about 15 are operator-facing. So the split is roughly ~137 user-facing to ~2 genuine plumbing
107+
(`/api/update/badge`, `/api/system/windows-replace-retries`) — not 139 internal exceptions.
108+
109+
That changes the work: most of these need a capability declaration, not an exemption. Since
110+
139 declarations will not land in one phase, the gate goes in with the currently-known
111+
user-facing families declared (access keys, routing profiles, provider CRUD/test, custom
112+
models/visibility/presets/discovery, injection and subagent settings) and the remainder
113+
marked with a bounded deferred exemption that names an owner — so the debt is visible and
114+
dated rather than silent.
115+
116+
## 090.4 — `--json` and exit codes still not uniform
117+
118+
- `doctor --json` is documented in the skill's recipes but `runDoctor` only looks for its own
119+
flags; `--json` is ignored and human output is printed.
120+
121+
**Plan audit correction — this is a refactor, not a flag.** I filed it beside two one-line
122+
fixes, which understated it badly. `runDoctor` has no report collection at all: it keeps a
123+
module-level failure bit and emits directly through ~90 `console.log` calls across a
124+
1,309-line surface (`doctor.ts:73`), and `dispatch.ts:175` appends the Codex Log Guard's
125+
human output *after* `runDoctor` returns. Adding a JSON print would therefore emit human
126+
lines and a JSON document on the same stdout — invalid output, worse than the ignored flag.
127+
My flag inventory was also incomplete: `--yes`, `--reclaim-response-temps` and reclaim-flag
128+
typo detection exist too (`doctor.ts:1028`).
129+
130+
Doing it properly needs a structured report DTO, separate human and JSON renderers, Log
131+
Guard output folded into the report rather than printed alongside it, a stable schema,
132+
exit derivation from collected findings, unknown-flag rejection, and a decision for
133+
`--json` combined with each mutating recovery mode. **Split to its own work-phase (wp11).**
134+
What lands in wp10 is the honest interim: `doctor` rejects `--json` with the usage exit and
135+
a message saying it is not yet supported, and the skill recipe stops recommending it. A
136+
documented flag that silently does nothing is the defect; refusing it is not ideal but it is
137+
not a lie.
138+
- JSON-mode API failures print `Error: …` on stderr and discard the structured body
139+
(`runtime-api.ts:333`), contradicting the error envelope the skill promises.
140+
- Unknown command exits 1, but the declared vocabulary says 2 for usage.
141+
142+
## 090.5 — the skill test validates command heads only
143+
144+
`tests/skill-ocx.test.ts` reduces each documented invocation to its leading `ocx <word>`
145+
before checking it, so a nonexistent subcommand or an ignored flag passes. `doctor --json`
146+
is exactly that case: head exists, documented flag does nothing.
147+
148+
**Plan audit correction — there is no full-invocation oracle to validate against.** I wrote
149+
"validate full invocations" without checking what could answer the question. `CAPABILITIES`
150+
does carry structured command paths and flags, but only for the 26 declared capabilities
151+
(`capabilities.ts:48`); `src/cli/registry.ts` holds only top-level commands and free-form
152+
usage strings, and the real subcommand parsers are scattered across command modules.
153+
Appending `--help` is not an oracle either: `root.ts:40` intercepts help before subcommand
154+
dispatch, so a nonexistent subcommand still prints help and exits 0.
155+
156+
So the achievable scope for wp10 is narrower and stated as such: validate every documented
157+
invocation **that maps to a declared capability** against that capability's command path and
158+
flag list — a real oracle, covering 26 capabilities — and assert that any invocation outside
159+
that set is explicitly listed as unverified rather than silently reduced to its head. A
160+
hand-maintained subcommand table is refused: it would be a third drifting source of truth.
161+
162+
The general answer is a declarative command grammar shared by parsing, help, capabilities and
163+
skill generation. That is real work and belongs with the doctor refactor in wp11, not smuggled
164+
into this phase.
165+
166+
Also: `skills/ocx/SKILL.md` calls the generated map "every verb" while the map contains 26
167+
capabilities. Either the claim narrows or the map grows; it cannot stay as-is.
168+
169+
## Accept criteria
170+
171+
1. `ocx logout` never reads a flag as a provider name, and a malformed invocation mutates the
172+
credential store **zero** times — asserted directly, not inferred from the exit code.
173+
Usage errors exit 2 before any I/O; a valid provider with no stored credential exits 4.
174+
2. The three missing verbs exist, are API-backed, declare capabilities, and emit `--json`.
175+
With the proxy stopped they exit non-zero, explain how to start it, and mutate nothing;
176+
there is no automatic local fallback.
177+
3. Logout returns a removed/not-found disposition from the store or route, so exit 4 is real
178+
rather than assumed, and the active-account-vs-all-accounts semantics are written down.
179+
4. The parity gate fails when a route is neither capability-covered nor `route.exempt`-
180+
justified, using the existing `ExemptionReason` union rather than a parallel allowlist.
181+
5. `capabilities --route` resolves for every family named in 090.3 that has a working CLI
182+
command; the rest carry a bounded deferred exemption naming an owner, so the debt is dated
183+
rather than silent.
184+
6. JSON-mode API failures emit the documented error envelope instead of a bare stderr line.
185+
7. `doctor --json` is refused with the usage exit and the skill recipe no longer recommends
186+
it; the structured-report refactor is wp11, not this phase.
187+
8. The skill test validates full invocations against the capability oracle where one exists,
188+
and every invocation outside it is explicitly listed as unverified.
189+
9. Every new assertion is driven red before being trusted.

0 commit comments

Comments
 (0)