|
| 1 | +# Lane C2 — telemetry projections on the landed vocabulary |
| 2 | + |
| 3 | +Status: OPEN. Branch `codex/260920-lane-c2-telemetry-projections`, cut from `dev` at |
| 4 | +`043aa435ff8f86095f55cbe08f74d45b9858da59`, which is where lane C's stage, cause and resend |
| 5 | +vocabulary landed ([020_lane_c.md](020_lane_c.md)). One branch, ordered commits, one pull request |
| 6 | +to `dev`. |
| 7 | + |
| 8 | +## What this lane fixes |
| 9 | + |
| 10 | +The completion condition for bundle 14 is that the UI, the durable log and Prometheus agree on |
| 11 | +logical and physical counts and on terminal classification. They did not, and the disagreement was |
| 12 | +not subtle: |
| 13 | + |
| 14 | +- Three surfaces classified a terminal three different ways. The durable row carried |
| 15 | + `terminalStatus` and `closeReason`; the exporter kept its own private `classifyResult`; the |
| 16 | + dashboard read the numeric HTTP status and nothing else. A turn cut short by |
| 17 | + `max_output_tokens` is durably `status: 200, terminalStatus: "incomplete"`, which the exporter |
| 18 | + reported as `incomplete` and the dashboard rendered as a green 200. The metric said incident and |
| 19 | + the operator saw success, for the same request. |
| 20 | +- The dashboard showed no physical send count at all. `sendCount` and `spend` were never rendered, |
| 21 | + so an attempt that sent three times appeared as one row with nothing to say otherwise. |
| 22 | +- The dashboard's recovery-kind union had drifted to nine of the durable thirteen, so `key-401`, |
| 23 | + `oauth-account-429`, `opaque-blob-rejection` and `reasoning-effort-downgrade` all reached the |
| 24 | + operator as "Unknown recovery reason" — four real causes rendered as the absence of one. |
| 25 | + |
| 26 | +The fix is one classifier in `src/usage/request-outcome.ts` that the exporter imports, the |
| 27 | +management payload already carries, and the dashboard calls. Agreement is structural rather than a |
| 28 | +rule someone maintains. The data was never missing: `requestLogDto` spreads the whole durable |
| 29 | +entry, so the page only had to declare the fields and stop reinventing the precedence. |
| 30 | + |
| 31 | +The exporter's result label set **is** the shared vocabulary rather than a copy of it. Restating |
| 32 | +those four strings is what let the two drift while both looked correct. |
| 33 | + |
| 34 | +### The dashboard may only reach a contract leaf |
| 35 | + |
| 36 | +Adversarial review caught this before the first push and it is worth recording, because the |
| 37 | +mistake is invisible from the backend side. The dashboard is a separate TypeScript project with |
| 38 | +`erasableSyntaxOnly`, and a **type-only** import still pulls the imported file's entire import |
| 39 | +graph into that project. Importing the recovery roster from `src/usage/log.ts` therefore dragged |
| 40 | +`node:fs`, `node:crypto` and the config barrel into the browser build, where a parameter property |
| 41 | +in `src/config/atomic-write.ts` does not compile. "It is only a type import" is not a defence. |
| 42 | + |
| 43 | +So the names a browser legitimately needs now live in `src/usage/telemetry-contract.ts`, which has |
| 44 | +no imports at all and must keep none. `src/usage/log.ts` re-exports them so every existing importer |
| 45 | +keeps its path, and `src/lib/request-failure-model.ts` stopped depending on the ledger module as a |
| 46 | +side effect. Three cases hold the boundary: the page must not name `src/usage/log`, the contract |
| 47 | +must have no imports, and the outcome module must reach nothing but the contract. |
| 48 | + |
| 49 | +The same review caught the send total disagreeing in the other direction. An earlier draft reported |
| 50 | +`max(sends, reserved)`, on the reasoning that a budget charge with no attempt row behind it is |
| 51 | +still a send that left. That is true, and it still made the dashboard say four where the exporter, |
| 52 | +summing the same attempts the recorder summed, said three. Two defensible formulas are two answers; |
| 53 | +the surfaces now read the recorded totals and recompute nothing, and a case asserts the exporter's |
| 54 | +`opencodex_physical_sends_total` equals what the dashboard shows. |
| 55 | + |
| 56 | +## Dispositions, updated |
| 57 | + |
| 58 | +### #2366 (chilung-cgu) — partially carried |
| 59 | + |
| 60 | +**Carried:** the rehydration and UI-projection half. The durable terminal facts now reach an |
| 61 | +operator instead of stopping at the API boundary. |
| 62 | + |
| 63 | +**Not carried, and why:** `FailureSide` and the seven-member `FailureStage` are a second |
| 64 | +attribution vocabulary beside the one that just landed, and defining two is the exact class that |
| 65 | +blocked 2.60.0. The PR also widens `transportPhase` and `terminalSource` from their existing closed |
| 66 | +unions to arbitrary strings, which would let bounded upstream-controlled text into the durable row; |
| 67 | +those validators stay. Copying the request-relative timeline into an attempt at finalization is |
| 68 | +simply wrong — request-relative elapsed values do not become attempt-relative by being copied. |
| 69 | + |
| 70 | +**Next step, specified:** persist `failureStage?: RequestFailureStage` and |
| 71 | +`failureCause?: RequestFailureCause` on `PersistedUsageAttempt`, projected onto the entry, with |
| 72 | +`resendPermission` computed at read time and never persisted. That is the smallest durable record |
| 73 | +that makes a failure attributable, and it is the prerequisite for #3748 below. It is not in this |
| 74 | +branch because it is new classification logic on the finalization path, and a branch whose only |
| 75 | +verification is static review plus hosted CI should not add a new derivation and the surface that |
| 76 | +consumes it in the same change. |
| 77 | + |
| 78 | +### #3748 (yansigit) — still deferred, reason updated |
| 79 | + |
| 80 | +The earlier reason was the parallel SQLite store. That still holds, but the blocking reason today is |
| 81 | +narrower and more useful: **the recorder does not yet record why a request finally failed.** |
| 82 | +`causeForRecoveryKind` answers why a *recovery* was attempted, which is a different question — a |
| 83 | +request that failed without any recovery, or that recovered and then failed for another reason, |
| 84 | +has no cause to group by. A derived failure ledger therefore cannot compute a grouping key today |
| 85 | +without reading `errorCode` or `upstreamError`, which are open strings. |
| 86 | + |
| 87 | +The design is otherwise settled and should be built once the field above exists: group failed rows |
| 88 | +scanned through the existing `scanUsageLedgerCooperatively` by a versioned fingerprint over closed |
| 89 | +vocabularies only — cause, status class, inbound protocol, terminal status, close reason, transport |
| 90 | +phase, terminal source — with fixed tuple positions so a missing field cannot collide structurally. |
| 91 | +No provider, no model, no account label, no free-text signature. First-seen, last-seen and count |
| 92 | +fall out of the scan; no second timestamp list is retained. |
| 93 | + |
| 94 | +Two parts of the original are not derivable from request history at all: the mutable |
| 95 | +`monitoring/dispatched/fixed/ignored` remediation status and its free-text notes. Those are |
| 96 | +operator state, not event history, and need their own owner rather than being presented as a |
| 97 | +derived ledger. |
| 98 | + |
| 99 | +### #3983 (yansigit) — still deferred as an emission path, reason updated |
| 100 | + |
| 101 | +The earlier reason was "a second emission path". The updated reason is stronger: the path is not |
| 102 | +ephemeral. `emitDebugLine` writes the in-process ring **and** stderr, and stderr is redirected to |
| 103 | +the service log under both launchd and systemd, so an installed service gets a durable per-event |
| 104 | +record with its own retention, sequencing, request identity and masking — beside the ledger and |
| 105 | +sourced from something other than it. |
| 106 | + |
| 107 | +Two further facts: four of its eighteen files no longer apply, including |
| 108 | +`run-turn-execution.ts` where carrying it literally would regress the current send-budget |
| 109 | +accounting; and its per-content HMAC is a process-global random key, so equality of every prompt, |
| 110 | +tool name and error message is correlatable for the process lifetime. |
| 111 | + |
| 112 | +**The useful half, specified:** a bounded normalized summary on the attempt — adapter events, |
| 113 | +actually relayed events, semantic bytes, side-effect events, terminal events — counted where the |
| 114 | +event is delivered rather than where it is read. That keeps the signals worth having (missing |
| 115 | +terminal, adapter-to-relay loss, empty output, partial output size) and inherits the ledger's |
| 116 | +normalization, masking and retention instead of inventing its own. |
| 117 | + |
| 118 | +### #5063 (Vocllum) — still deferred, reason updated |
| 119 | + |
| 120 | +The earlier reason was "a separate product slice with GUI surface". The updated reason is a |
| 121 | +correctness one found while reviewing it against current `dev`: |
| 122 | + |
| 123 | +- Retention captures the file size, copies a retained suffix to a temp file and renames. A row |
| 124 | + appended by **another process** between the size snapshot and the rename is silently dropped. |
| 125 | + The PR's own "concurrent re-entrancy" test performs two sequential calls and says it cannot test |
| 126 | + true concurrency. |
| 127 | +- Both the temp-file fsync and the parent-directory fsync failures are swallowed, and replacement |
| 128 | + proceeds anyway. There is no revision, inode or digest comparison immediately before the rename. |
| 129 | +- It hand-rolls the copy rather than using `src/config/atomic-write.ts`. |
| 130 | +- It does not clear the 2,000-entry request-log ring, so rows deleted from disk stay visible |
| 131 | + through `/api/logs` until eviction or restart. |
| 132 | +- `src/server/index.ts` is 892 lines against a 893-line cap, so the startup wiring cannot land as |
| 133 | + written without offsetting a line elsewhere. |
| 134 | + |
| 135 | +Retention on the canonical ledger is the right architecture and its derived-index invalidation is |
| 136 | +correct in shape. It needs a ledger-owner serialization and revision contract first. |
| 137 | + |
| 138 | +### The dashboard roster defect — fixed here |
| 139 | + |
| 140 | +Recorded in [020_lane_c.md](020_lane_c.md) as deferred because it needed strings across ten |
| 141 | +catalogs and a screenshot. It is fixed in this branch because the agreement condition required |
| 142 | +touching this page anyway. The union is now derived from the durable roster and a `satisfies` |
| 143 | +clause makes the next added kind a typecheck failure rather than a silent fallback. |
| 144 | + |
| 145 | +**This trips the `missing_ui_screenshot` gate.** That gate fires on changed paths under `gui/`, |
| 146 | +not on words in the description, and this lane may not build or run the GUI, so it cannot produce |
| 147 | +the screenshot itself. A maintainer comment or the `gui-screenshot-waived` label is the documented |
| 148 | +resolution; the visible change is four attempt rows that said "Unknown recovery reason" now naming |
| 149 | +their cause, plus an outcome row and a send count in the detail dialog. |
| 150 | + |
| 151 | +## Retention and masking, stated in one place |
| 152 | + |
| 153 | +The instruction was to unify rather than add a fifth policy. The honest form of that today is to |
| 154 | +record what already exists and bind projections to it, because every bound below is already |
| 155 | +enforced and none of them needs a new owner: |
| 156 | + |
| 157 | +| Surface | Bound | |
| 158 | +| --- | --- | |
| 159 | +| Usage ledger rows (scanner) | 1 MiB per row, 1 MiB read chunks, 64 KiB boundary digest | |
| 160 | +| Startup request-log hydration | 64 MiB read ceiling | |
| 161 | +| Management usage snapshot | 64 MiB, 500,000 rows | |
| 162 | +| Usage metadata strings | sanitized, capped at 64 characters | |
| 163 | +| Upstream error text | `redactSecretString`, capped at 500 characters | |
| 164 | +| Conversation correlation | rejects over 4,096 characters, persists a 32-hex digest | |
| 165 | +| Debug ring | 2,000 lines, 16 KiB per line, 64 subscribers | |
| 166 | +| Request-log ring | 2,000 entries plus app-owned-memory eviction | |
| 167 | +| Routing history | rebuildable projection; pages capped, records over 1 MiB omitted | |
| 168 | +| Email masking | masked unless `privacy.maskEmails: false` | |
| 169 | + |
| 170 | +The policy this lane adopts: **masking happens at capture and projection boundaries, retention |
| 171 | +deletes canonical rows, and every derived surface inherits both.** A projection does not get its |
| 172 | +own TTL, its own row cap or its own redaction pass. The failure projection specified above obeys |
| 173 | +this by construction — it holds only aggregates and a scanner checkpoint, and discards them when |
| 174 | +the source is replaced. |
| 175 | + |
| 176 | +## Issues |
| 177 | + |
| 178 | +#4191 and #5180 stay open and are not closed here. What narrowed: the dashboard now reports the |
| 179 | +terminal classification and the send count the durable row always carried, so an operator can tell |
| 180 | +an incomplete turn from a successful one without reading the ledger. What remains unchanged: the |
| 181 | +WebSocket-to-SSE fallback for #4191, and the shared cooldown and `Retry-After` handling for #5180. |
| 182 | + |
| 183 | +## Verification |
| 184 | + |
| 185 | +Static source review plus exact-head hosted CI. |
| 186 | + |
| 187 | +NOT RUN on this branch, by instruction: `bun run test`, any individual `bun test` file, |
| 188 | +`bun run typecheck`, `bun run build:gui`, `bun run lint:gui`, `bun install`, |
| 189 | +`bun run structure:check`, `bun run privacy:scan`, and any live `ocx` execution. None of these may |
| 190 | +be recorded as passing. |
| 191 | + |
| 192 | +Checked statically on this branch: |
| 193 | + |
| 194 | +- all eleven new label keys are present in all ten catalogs, and the catalog edits are purely |
| 195 | + additive (+11 lines, 0 removed, per file); |
| 196 | +- the ten catalogs are explicitly exempt from the file-size ratchet, for the reason the exemption |
| 197 | + list gives: they grow by one line per UI string across every locale at once; |
| 198 | +- no ratchet-capped file is touched by this branch; |
| 199 | +- `scripts/test-layout/layout.json` and `tests/fixtures/test-layout-expected.json` agree key for |
| 200 | + key, and the new test's regex seed resolves to the same domain it is registered to, which is the |
| 201 | + oracle that failed lane C on its first push; |
| 202 | +- no test restates a source constant: the outcome vocabulary, the recovery roster and the label |
| 203 | + keys are all read from the modules that declare them. |
0 commit comments