|
| 1 | +# 090 — wp10: close the gap-audit findings |
| 2 | + |
| 3 | +The wp9 close-out gap audit answered YES to "does a material gap remain in agentic CLI |
| 4 | +control of GUI capabilities", with three High findings and one Medium. Each was |
| 5 | +independently reproduced against source before this plan was written, so none of it is |
| 6 | +taken on the reviewer's word. |
| 7 | + |
| 8 | +Reproductions (run from the branch, not the released `ocx` on PATH — the first attempt at |
| 9 | +this measurement used the installed binary and got "Unknown command: capabilities", which |
| 10 | +is a stale build, not evidence): |
| 11 | + |
| 12 | +``` |
| 13 | +bun run src/cli/index.ts capabilities --route /api/keys --json -> {"capabilities": []}, exit 4 |
| 14 | +bun run src/cli/index.ts logout --json -> "Logged out of --json.", exit 0 |
| 15 | +bun run src/cli/index.ts definitely-not-a-command -> exit 1 |
| 16 | +capabilities declared: 26 routes declared: 207 |
| 17 | +``` |
| 18 | + |
| 19 | +## 090.1 — `logout --json` silently no-ops (highest severity) |
| 20 | + |
| 21 | +`src/cli/dispatch.ts` `logout` takes `args[1]` as the provider name with no flag parsing, so |
| 22 | +`ocx logout --json` lowercases `--json`, calls `removeCredential("--json")`, prints |
| 23 | +`Logged out of --json.` and exits 0. An agent gets a success exit for an operation that did |
| 24 | +nothing, which is the worst failure mode in this whole unit: silent, and indistinguishable |
| 25 | +from success. |
| 26 | + |
| 27 | +**Plan audit correction.** My first reading called `removeCredential("--json")` harmless |
| 28 | +because `store.ts:598` returns early on an unknown key. That is wrong for a store this code |
| 29 | +does not control: `normalizeAuthStore` (`store.ts:346-355`) copies **every** top-level key |
| 30 | +it finds, so a hand-edited, legacy, or corrupted `auth.json` containing a `--json` key would |
| 31 | +have its active account deleted — and the key removed entirely if that was its last account. |
| 32 | +The severity is therefore "can destroy credentials in an unusual but reachable store", not |
| 33 | +"wastes a call". Recorded because it changes what the regression has to prove. |
| 34 | + |
| 35 | +Fix: parse flags out of argv **before any store I/O**, and split the outcomes the vocabulary |
| 36 | +already distinguishes rather than collapsing them: |
| 37 | + |
| 38 | +| Case | Exit | |
| 39 | +|---|---| |
| 40 | +| omitted provider, unknown flag, invalid argument | 2 (usage), before touching the store | |
| 41 | +| valid provider name with no stored credential | 4 (not found) | |
| 42 | +| removed | 0 | |
| 43 | + |
| 44 | +The regression must assert that a malformed invocation causes **zero** store mutation, not |
| 45 | +merely a non-zero exit. |
| 46 | + |
| 47 | +## 090.2 — three GUI capabilities have no CLI equivalent |
| 48 | + |
| 49 | +| GUI call | CLI today | Missing verb | |
| 50 | +|---|---|---| |
| 51 | +| `POST /api/oauth/logout` | `ocx logout` only calls `removeCredential` locally | API-backed logout | |
| 52 | +| `GET /api/system/codex-app-server`, `POST /api/system/codex-restart` | `sync --restart-codex` only as a side effect after a write | `system codex-app-server status` / `restart` | |
| 53 | +| `GET /api/claude-desktop/status` | `claude desktop show` emits locally-built state only | `claude desktop status` | |
| 54 | + |
| 55 | +The OAuth one is not cosmetic. The route does five things past `removeCredential`: |
| 56 | +`reconcileLiveStateStores`, `clearLoginState`, `clearModelCache`, |
| 57 | +`clearGatherRoutedModelsInflight`, and quota-cache eviction |
| 58 | +(`oauth-account-routes.ts:228-243`). A CLI logout that skips them leaves a running proxy |
| 59 | +serving a removed credential's cached models and quota, so the local-only path is a |
| 60 | +correctness gap and not just a parity gap. |
| 61 | + |
| 62 | +**Plan audit correction — the stopped-proxy contract was missing, and it is the whole |
| 63 | +design question.** An API-backed verb needs a live proxy; discovery failure throws |
| 64 | +"Proxy is not running" (`runtime-api.ts:43`). So this phase must state the behaviour: |
| 65 | + |
| 66 | +- Proxy running: go through `POST /api/oauth/logout` so the five invalidations happen. |
| 67 | +- Proxy not running: exit non-zero, say how to start it, and **mutate nothing**. |
| 68 | + |
| 69 | +No automatic local fallback. A best-effort fallback is actively dangerous here: if a proxy |
| 70 | +is alive but momentarily undiscoverable, falling back would delete the credential on disk |
| 71 | +while that proxy keeps serving its cached models, login state and quota — the precise |
| 72 | +divergence the API path exists to prevent. An offline mode is acceptable only as an explicit |
| 73 | +opt-in that proves no proxy is live, not as error recovery. |
| 74 | + |
| 75 | +Two further under-specifications to settle before implementing: |
| 76 | + |
| 77 | +- The route reports success even when nothing was removed, because `removeCredential` |
| 78 | + returns no disposition (`oauth-account-routes.ts:231`). Exit 4 is impossible until the |
| 79 | + store or route returns a removed/not-found result. Do it there, not via a CLI preflight |
| 80 | + read — a preflight is racy by construction. |
| 81 | +- Define whether logout removes only the active account or all of them, and whether legacy |
| 82 | + `ocx logout` becomes an alias or keeps its offline behaviour. Silence here is how two |
| 83 | + commands end up meaning different things. |
| 84 | + |
| 85 | +## 090.3 — the parity gate only checks one direction |
| 86 | + |
| 87 | +`tests/cli-capabilities.test.ts` asserts every capability's route exists. It never asserts |
| 88 | +the converse, so 139 non-exempt routes carry no capability entry and nothing fails. The |
| 89 | +observable consequence is that `capabilities --route` — the agent's discovery entry point — |
| 90 | +returns empty and exits 4 for `/api/keys` and `/api/routing-profiles` while `ocx access key` |
| 91 | +and `ocx route policy` work. |
| 92 | + |
| 93 | +**Plan audit correction — the allowlist was the wrong mechanism, and the counts were off.** |
| 94 | +`ManagementRoute` already carries an `exempt` field with a typed `ExemptionReason` union |
| 95 | +(`route-registry.ts:24-46`, `:68`) covering exactly the categories a parallel allowlist would |
| 96 | +have re-invented: `session-only`, `disabled`, `capability-principal`, `test-seam`, |
| 97 | +`local-transport`, `dead`. A second list would duplicate the source of truth and relocate the |
| 98 | +omission problem rather than close it. |
| 99 | + |
| 100 | +The reverse gate is therefore: **every route is either covered by `capabilityRouteKeys()` or |
| 101 | +carries a justified `route.exempt`** — never neither, and ideally never both. |
| 102 | + |
| 103 | +The audit's fresh counts also correct mine: 206 routes (I said 207), 35 capability-covered |
| 104 | +route keys, 32 exempt, 139 unexplained. More importantly it disproved my framing of what |
| 105 | +those 139 are: 122 of them are already referenced from CLI source, and of the remaining 17 |
| 106 | +about 15 are operator-facing. So the split is roughly ~137 user-facing to ~2 genuine plumbing |
| 107 | +(`/api/update/badge`, `/api/system/windows-replace-retries`) — not 139 internal exceptions. |
| 108 | + |
| 109 | +That changes the work: most of these need a capability declaration, not an exemption. Since |
| 110 | +139 declarations will not land in one phase, the gate goes in with the currently-known |
| 111 | +user-facing families declared (access keys, routing profiles, provider CRUD/test, custom |
| 112 | +models/visibility/presets/discovery, injection and subagent settings) and the remainder |
| 113 | +marked with a bounded deferred exemption that names an owner — so the debt is visible and |
| 114 | +dated rather than silent. |
| 115 | + |
| 116 | +## 090.4 — `--json` and exit codes still not uniform |
| 117 | + |
| 118 | +- `doctor --json` is documented in the skill's recipes but `runDoctor` only looks for its own |
| 119 | + flags; `--json` is ignored and human output is printed. |
| 120 | + |
| 121 | + **Plan audit correction — this is a refactor, not a flag.** I filed it beside two one-line |
| 122 | + fixes, which understated it badly. `runDoctor` has no report collection at all: it keeps a |
| 123 | + module-level failure bit and emits directly through ~90 `console.log` calls across a |
| 124 | + 1,309-line surface (`doctor.ts:73`), and `dispatch.ts:175` appends the Codex Log Guard's |
| 125 | + human output *after* `runDoctor` returns. Adding a JSON print would therefore emit human |
| 126 | + lines and a JSON document on the same stdout — invalid output, worse than the ignored flag. |
| 127 | + My flag inventory was also incomplete: `--yes`, `--reclaim-response-temps` and reclaim-flag |
| 128 | + typo detection exist too (`doctor.ts:1028`). |
| 129 | + |
| 130 | + Doing it properly needs a structured report DTO, separate human and JSON renderers, Log |
| 131 | + Guard output folded into the report rather than printed alongside it, a stable schema, |
| 132 | + exit derivation from collected findings, unknown-flag rejection, and a decision for |
| 133 | + `--json` combined with each mutating recovery mode. **Split to its own work-phase (wp11).** |
| 134 | + What lands in wp10 is the honest interim: `doctor` rejects `--json` with the usage exit and |
| 135 | + a message saying it is not yet supported, and the skill recipe stops recommending it. A |
| 136 | + documented flag that silently does nothing is the defect; refusing it is not ideal but it is |
| 137 | + not a lie. |
| 138 | +- JSON-mode API failures print `Error: …` on stderr and discard the structured body |
| 139 | + (`runtime-api.ts:333`), contradicting the error envelope the skill promises. |
| 140 | +- Unknown command exits 1, but the declared vocabulary says 2 for usage. |
| 141 | + |
| 142 | +## 090.5 — the skill test validates command heads only |
| 143 | + |
| 144 | +`tests/skill-ocx.test.ts` reduces each documented invocation to its leading `ocx <word>` |
| 145 | +before checking it, so a nonexistent subcommand or an ignored flag passes. `doctor --json` |
| 146 | +is exactly that case: head exists, documented flag does nothing. |
| 147 | + |
| 148 | +**Plan audit correction — there is no full-invocation oracle to validate against.** I wrote |
| 149 | +"validate full invocations" without checking what could answer the question. `CAPABILITIES` |
| 150 | +does carry structured command paths and flags, but only for the 26 declared capabilities |
| 151 | +(`capabilities.ts:48`); `src/cli/registry.ts` holds only top-level commands and free-form |
| 152 | +usage strings, and the real subcommand parsers are scattered across command modules. |
| 153 | +Appending `--help` is not an oracle either: `root.ts:40` intercepts help before subcommand |
| 154 | +dispatch, so a nonexistent subcommand still prints help and exits 0. |
| 155 | + |
| 156 | +So the achievable scope for wp10 is narrower and stated as such: validate every documented |
| 157 | +invocation **that maps to a declared capability** against that capability's command path and |
| 158 | +flag list — a real oracle, covering 26 capabilities — and assert that any invocation outside |
| 159 | +that set is explicitly listed as unverified rather than silently reduced to its head. A |
| 160 | +hand-maintained subcommand table is refused: it would be a third drifting source of truth. |
| 161 | + |
| 162 | +The general answer is a declarative command grammar shared by parsing, help, capabilities and |
| 163 | +skill generation. That is real work and belongs with the doctor refactor in wp11, not smuggled |
| 164 | +into this phase. |
| 165 | + |
| 166 | +Also: `skills/ocx/SKILL.md` calls the generated map "every verb" while the map contains 26 |
| 167 | +capabilities. Either the claim narrows or the map grows; it cannot stay as-is. |
| 168 | + |
| 169 | +## Accept criteria |
| 170 | + |
| 171 | +1. `ocx logout` never reads a flag as a provider name, and a malformed invocation mutates the |
| 172 | + credential store **zero** times — asserted directly, not inferred from the exit code. |
| 173 | + Usage errors exit 2 before any I/O; a valid provider with no stored credential exits 4. |
| 174 | +2. The three missing verbs exist, are API-backed, declare capabilities, and emit `--json`. |
| 175 | + With the proxy stopped they exit non-zero, explain how to start it, and mutate nothing; |
| 176 | + there is no automatic local fallback. |
| 177 | +3. Logout returns a removed/not-found disposition from the store or route, so exit 4 is real |
| 178 | + rather than assumed, and the active-account-vs-all-accounts semantics are written down. |
| 179 | +4. The parity gate fails when a route is neither capability-covered nor `route.exempt`- |
| 180 | + justified, using the existing `ExemptionReason` union rather than a parallel allowlist. |
| 181 | +5. `capabilities --route` resolves for every family named in 090.3 that has a working CLI |
| 182 | + command; the rest carry a bounded deferred exemption naming an owner, so the debt is dated |
| 183 | + rather than silent. |
| 184 | +6. JSON-mode API failures emit the documented error envelope instead of a bare stderr line. |
| 185 | +7. `doctor --json` is refused with the usage exit and the skill recipe no longer recommends |
| 186 | + it; the structured-report refactor is wp11, not this phase. |
| 187 | +8. The skill test validates full invocations against the capability oracle where one exists, |
| 188 | + and every invocation outside it is explicitly listed as unverified. |
| 189 | +9. Every new assertion is driven red before being trusted. |
0 commit comments