Repository navigation
Commit 173dfff
authored
Usage panel: Codex accuracy, transcript attribution, and a self-verifying docs system (#60)
* fix(usage): stop Codex model rows undercounting responses and subagent replays double-billing tokens
Two bugs in the ADR-0009 usage scorecard's Codex ingestion:
1. parseCodex never passed `responses` into addUsage (Claude's parser
does, per assistant turn), so every Codex model in byModel showed
"0 resp" in the dashboard regardless of real token/cost volume.
2. Codex's thread_spawn subagent delegation writes a rollout file that
replays its parent thread's entire cumulative token history as
duplicate events before its own new turns (a documented Codex CLI
behavior — see ccusage/ccusage#950, which measured up to 91x cost
inflation from exactly this). parseCodex took each file's last
cumulative total_token_usage at face value with no de-dup, so a
subagent-sourced rollout would double-count the parent's spend.
Now guarded by session_meta.thread_source: a 'subagent' rollout's
token/cost contribution is skipped (the session itself stays
visible, with threadSource surfaced, same as the existing
`sidechain` flag for Claude).
Verified against 5 real local Codex rollout files (no resumed-session
carryover reproduced); the subagent-replay guard is covered by a
synthetic fixture but not yet verified against a real thread_source:
subagent file on this machine.
* docs(usage): add cited maintainer reference for every Scorecard metric
Every figure on the Usage tab's Scorecard — sessions, API-equivalent
cost, tokens, engaged time, cache read %, cost per day, by host, when
you work, models in play, projects, what you worked on — now has a
dedicated maintainer-facing writeup: exact formula, file:line source
citation (pinned to this commit), a worked numeric example, and what
it deliberately does not model.
Pricing claims are grounded against primary sources where fetchable
(Anthropic's own pricing and prompt-caching docs, fetched in full) and
against corroborated third-party sources where not (OpenAI has no
single canonical machine-readable pricing page, same limitation
pricing.mjs's own comments already document) — the sourcing tier is
stated explicitly rather than implied, so a reader can tell primary
from corroborated at a glance.
Includes a dedicated section documenting the two Codex-parsing bugs
fixed in the prior commit, with citations to the public GitHub issues
(openai/codex#14489, ccusage/ccusage#884, ccusage/ccusage#950,
openai/codex#23001) that establish the subagent thread-replay pattern
is a real, previously-reported Codex CLI behavior and not a hypothesis
invented for this audit — plus an honest statement of what was and was
not verified against real local data.
ADR-0009 gets a one-line pointer to the new doc from its References
section.
* feat(scripts): add standalone Codex usage diagnostic for third-party verification
A read-only, zero-dependency script anyone can run against their own
~/.codex/sessions to independently check the subagent thread-replay
bug (docs/USAGE-SCORECARD-METRICS.md §15 Bug B) without sharing any
transcript content.
It computes token/cost totals two ways from its own standalone
reimplementation of the relevant parsing (not by importing the fix
itself, so the comparison is a genuine check, not a tautology):
"before" (every rollout counted, matching the current unpatched
dashboard) and "after" (thread_source:"subagent" rollouts excluded,
matching this branch's fix). The printed report is aggregate counts
only — no prompts, titles, session ids, file paths, or timestamps —
by construction: the parser never retains those fields.
Verified against this machine's real Codex history (5 sessions, all
thread_source:"user" — 0% excluded, before == after, consistent with
the earlier finding that no subagent-replay case exists locally) and
against a synthetic user+subagent pair, where it correctly excluded
98.86% of tokens attributable to the synthetic replay.
* docs(usage): add a friend/third-party-facing Codex verification writeup
docs/CODEX-USAGE-DIAGNOSTIC.md is a standalone companion to the
maintainer-facing metrics reference — self-contained (no repo context
assumed), explaining the two bugs in plain terms, why the fix can be
trusted (prior-art citations, regression tests, an independently
computed before/after check), and exactly how to run
scripts/codex-usage-diagnostic.mjs and what to report back: the
printed output block, in full, and nothing else.
Cross-linked from USAGE-SCORECARD-METRICS.md §15 so a maintainer
pointed at either document finds the other.
* feat(usage): stop dropped-connection/rate-limit/auth-failure turns from appearing as a $0 model
Claude Code synthesizes a local placeholder assistant turn — model:
"<synthetic>", isApiErrorMessage: true, every usage field zero — when
a request's connection drops, hits a rate limit, or fails auth before
a real completion returns. Before this change, that placeholder's
model id was treated like any other and surfaced in "Models in Play"
as its own ranked row: real-looking, $0.00, 0 tokens, N responses —
misleading, since no model ever ran.
parseClaude now branches on isApiErrorMessage: the turn still counts
toward rec.responses and the punchcard (it's real engaged time — a
request was genuinely in flight), but is never pushed into rec.models
and never reaches addUsage(), so it can no longer create a byModel
row of any kind. It increments a new rec.exceptions counter instead,
rolled up into totals.exceptions and surfaced per-session alongside
the existing sidechain/threadSource flags — inspectable in the
Sessions tab, never hidden. The dashboard shows a small "N
dropped/errored turns excluded" note under the Models in Play header
when the count is nonzero.
SCHEMA_VERSION bumped 3 -> 4: exceptions is a new required field on
every cached session record, and summing `undefined` from a stale
v3-cached record into totals.exceptions silently produces NaN
(serialized as null over JSON) instead of a real count. This was
caught by querying the real on-disk cache during verification, not by
the unit tests, which only ever exercise a fresh parse — a live query
against this machine's actual 90-day index (1,656 sessions) returned
totals.exceptions: null and still showed a <synthetic> byModel row on
the first run after this change; re-querying after the schema bump
returned totals.exceptions: 20 with <synthetic> gone from byModel
entirely.
docs/USAGE-SCORECARD-METRICS.md §10 documents the behavior, the real
verified numbers, and the cache-schema gotcha; every citation in the
document (65 across usage-index.mjs and dashboard-server.mjs) was
re-verified against the final file state after this change, since
several edits landed at different points in the file and shifted
line numbers more than once.
* fix(usage): attribute transcript turns by kind — a tool result is never "you"
The Messages API records tool results and harness context injections
under role "user", so the transcript reader's role-only labeling
attributed the harness's work to the person: a measured real session
had 20 human prompts and 276 tool results, every one labeled YOU.
parseClaude now stamps each user turn with kind — 'prompt' (the
person, including image-only pastes that carry no text block),
'tool-result' (a tool_result block fed back after a tool call), or
'context' (isMeta harness injections) — via userTurnKind, which is
deliberately broader than isHumanPrompt on the image-only edge:
"not countable as a text prompt" and "not the human" are different
claims. The prompt boolean (which drives prompt counts) is unchanged.
Codex user turns are kind 'prompt' by construction — rollouts only
record real prompts as user_message events.
The renderer labels from kind, never role: "you" is reserved for
prompts; tool results and context render purple (rhyming with the
tool chips) with a hover title stating the harness — not the person —
sent them. maskTurns already passes non-string fields through, so no
wire change; turns are never cached, so no SCHEMA_VERSION bump —
both properties now pinned by tests.
Verified against real transcripts from both providers: a Claude
session with 884 turns ({prompt:20, context:6, tool-result:276,
assistant:588}, zero missing kinds) and a Codex rollout (8 user
turns, all 'prompt').
Docs: new docs/TRANSCRIPTS.md — the maintainer reference for the
whole per-session pipeline (stores → parsers → turn model →
readSession gates → masking/truncation → HTTP → UI), every citation
content-verified; ADR-0009 §8 amendment records the misattribution
as the honesty failure it was; USAGE-SCORECARD-METRICS.md citations
remapped (67 checked, 0 failures) after the line shifts. OpenRouter
ingestion tracked as #59.
* feat(usage): render harness sentinel markup as structure, not XML soup
Claude Code writes XML wrappers into transcript text — slash commands
as a <command-name>/<command-message>/<command-args> triple, injected
context as <system-reminder> / <local-command-caveat> /
<local-command-stdout> blocks. The transcript reader rendered them as
literal angle-bracket soup. Measured against the real corpus, exactly
six shapes occur (command-contents/stderr: zero), so fmtHarness
handles exactly those: the command triple becomes a "/clear"-style
chip (args kept; the redundant message becomes a hover title only
when it differs from the name), and the three block wrappers become
quiet labelled panels (system reminder / caveat / command output).
Presentation only, per the same no-silent-alteration rule that
governs masking: wrapped content stays verbatim, only the wrapper
tags become styling. Runs on escaped text after markRedactions —
capture groups can never smuggle unescaped markup — and an unmatched
tag (e.g. cut mid-block by turn truncation) is left raw rather than
half-formatted. Verified end-to-end by extracting the served
function from the live page and executing it against the real
sentinel shapes (chip, panel, and unmatched-tag cases all exact).
The client JS lives inside a server-side template literal, so the
regex backslashes are doubled — \1 would otherwise parse as an octal
escape and take the whole page down with a SyntaxError (caught by
node --check before it shipped).
Docs: TRANSCRIPTS.md §6.2 documents the formatter and the measured
tag census; citation integrity re-verified across both maintainer
docs (127 citations: content-checked, one out-of-range range and two
remap-corrupted entries fixed).
* fix(usage): harness-output envelopes are neither the person nor the model
An open-ended envelope census (the first census only counted tags it
guessed at — that was the flaw) found three user-role families still
labelled "you": task-notification (550 in the corpus), bash-stdout
(85), and local-command-stdout (60). None carries isMeta or a
tool_result, so text shape is the only signal — and the answer to
"is it the person or the model?" is neither: the HARNESS wrote them.
- userTurnKind now classifies a user turn whose text opens with a
harness-output envelope (HARNESS_OUTPUT_RE: task-notification,
bash-stdout/-stderr, local-command-stdout/-stderr,
local-command-caveat) as 'context'. bash-input (a "! command" the
person typed) and slash-command records stay 'prompt' — the person
initiated those.
- isHumanPrompt shares the same rule, so prompt COUNTS stop being
inflated by stdout dumps and notifications. On the exact session
reported (keel, "Promote provisioning-fly-hetzer pipeline"): 32
claimed prompts -> 20 real ones; bash-input=prompt,
bash-stdout=context, task-notification=context, verified live.
- SCHEMA_VERSION 4 -> 5: cached session records carry the old
inflated prompts figure and must be re-derived.
- fmtHarness learns the new shapes: bash-input renders as a
"!"-prefixed chip; task-notification / bash-stdout / bash-stderr /
local-command-stderr join the labelled quiet panels.
Tests cover the envelope rules at both layers; docs updated
(TRANSCRIPTS.md §3.2 taxonomy + census, USAGE-SCORECARD-METRICS.md §2
prompts definition), and all 128 citations across both maintainer
docs re-verified against the shifted sources.
* docs: add dual-theme figure set; restructure around active fact vs history
Seven hand-authored SVG figures (docs/assets/, light+dark via a
prefers-color-scheme block, palette machine-validated for CVD safety in
both modes) now illustrate the transcript and scorecard pipelines: the
per-session gate corridor, the scan/reader fork, turn-kind attribution,
mask-then-truncate ordering, the classifier tournament, the confidence
formula, and per-turn vs cumulative token accounting.
All seven docs directly under docs/ are restructured on one rule: the
main body states only what is presently true; fix narratives, incident
stories, and ADR citations move to per-doc appendices (Fix history /
Verification / Design-rationale ADR maps). METRICS keeps its §0-§14
numbering stable so code comments citing those sections stay valid;
§15-§17 became Appendices A/B/D. The "pinned to commit" citation
disclaimer is replaced by a machine-check contract (the gate lands in
the next commit). Also corrects a miscitation in the Bug B evidence
trail (C4/C5/C6 -> C5/C6/C7) and scrubs the same history/ADR language
from the figures themselves.
* test(docs): gate every file:line citation against the current source
tests/kit/doc-citations.test.mjs extracts every backticked file:line
citation from the two usage docs and asserts that an identifier or
quoted string named beside the citation appears within the cited range
(+/-10 lines) of the source as it exists now — so citation rot fails
pnpm test with a re-anchoring hint instead of accumulating silently
behind a "line numbers will drift" disclaimer.
Maiden run: 93/96 METRICS citations were already fresh against HEAD,
one real drift was caught and re-anchored (the midnight-boundary test,
556 -> 608), and two citations were strengthened to name their subject.
Matching is whitespace-normalized on both sides and fence-aware, so
neither doc wrapping nor source wrapping can defeat an honest anchor.
Also updates the diagnostic script's stale §15 pointer to Appendix A.1 parent 4999ddc commit 173dfff
21 files changed
Lines changed: 3336 additions & 44 deletions
File tree
- docs
- adr
- assets
- scripts
- src/lib
- tests
- kit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
| 8 | + | |
| 9 | + | |
13 | 10 | | |
14 | 11 | | |
15 | 12 | | |
| |||
28 | 25 | | |
29 | 26 | | |
30 | 27 | | |
31 | | - | |
32 | | - | |
33 | | - | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
34 | 31 | | |
35 | 32 | | |
36 | 33 | | |
| |||
98 | 95 | | |
99 | 96 | | |
100 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
153 | | - | |
154 | | - | |
| 153 | + | |
155 | 154 | | |
156 | 155 | | |
157 | 156 | | |
| |||
215 | 214 | | |
216 | 215 | | |
217 | 216 | | |
218 | | - | |
| 217 | + | |
219 | 218 | | |
220 | 219 | | |
221 | 220 | | |
| |||
240 | 239 | | |
241 | 240 | | |
242 | 241 | | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
0 commit comments