Skip to content

Commit 173dfff

Browse files
authored
Usage panel: Codex accuracy, transcript attribution, and a self-verifying docs system (#60)
* fix(usage): stop Codex model rows undercounting responses and subagent replays double-billing tokens Two bugs in the ADR-0009 usage scorecard's Codex ingestion: 1. parseCodex never passed `responses` into addUsage (Claude's parser does, per assistant turn), so every Codex model in byModel showed "0 resp" in the dashboard regardless of real token/cost volume. 2. Codex's thread_spawn subagent delegation writes a rollout file that replays its parent thread's entire cumulative token history as duplicate events before its own new turns (a documented Codex CLI behavior — see ccusage/ccusage#950, which measured up to 91x cost inflation from exactly this). parseCodex took each file's last cumulative total_token_usage at face value with no de-dup, so a subagent-sourced rollout would double-count the parent's spend. Now guarded by session_meta.thread_source: a 'subagent' rollout's token/cost contribution is skipped (the session itself stays visible, with threadSource surfaced, same as the existing `sidechain` flag for Claude). Verified against 5 real local Codex rollout files (no resumed-session carryover reproduced); the subagent-replay guard is covered by a synthetic fixture but not yet verified against a real thread_source: subagent file on this machine. * docs(usage): add cited maintainer reference for every Scorecard metric Every figure on the Usage tab's Scorecard — sessions, API-equivalent cost, tokens, engaged time, cache read %, cost per day, by host, when you work, models in play, projects, what you worked on — now has a dedicated maintainer-facing writeup: exact formula, file:line source citation (pinned to this commit), a worked numeric example, and what it deliberately does not model. Pricing claims are grounded against primary sources where fetchable (Anthropic's own pricing and prompt-caching docs, fetched in full) and against corroborated third-party sources where not (OpenAI has no single canonical machine-readable pricing page, same limitation pricing.mjs's own comments already document) — the sourcing tier is stated explicitly rather than implied, so a reader can tell primary from corroborated at a glance. Includes a dedicated section documenting the two Codex-parsing bugs fixed in the prior commit, with citations to the public GitHub issues (openai/codex#14489, ccusage/ccusage#884, ccusage/ccusage#950, openai/codex#23001) that establish the subagent thread-replay pattern is a real, previously-reported Codex CLI behavior and not a hypothesis invented for this audit — plus an honest statement of what was and was not verified against real local data. ADR-0009 gets a one-line pointer to the new doc from its References section. * feat(scripts): add standalone Codex usage diagnostic for third-party verification A read-only, zero-dependency script anyone can run against their own ~/.codex/sessions to independently check the subagent thread-replay bug (docs/USAGE-SCORECARD-METRICS.md §15 Bug B) without sharing any transcript content. It computes token/cost totals two ways from its own standalone reimplementation of the relevant parsing (not by importing the fix itself, so the comparison is a genuine check, not a tautology): "before" (every rollout counted, matching the current unpatched dashboard) and "after" (thread_source:"subagent" rollouts excluded, matching this branch's fix). The printed report is aggregate counts only — no prompts, titles, session ids, file paths, or timestamps — by construction: the parser never retains those fields. Verified against this machine's real Codex history (5 sessions, all thread_source:"user" — 0% excluded, before == after, consistent with the earlier finding that no subagent-replay case exists locally) and against a synthetic user+subagent pair, where it correctly excluded 98.86% of tokens attributable to the synthetic replay. * docs(usage): add a friend/third-party-facing Codex verification writeup docs/CODEX-USAGE-DIAGNOSTIC.md is a standalone companion to the maintainer-facing metrics reference — self-contained (no repo context assumed), explaining the two bugs in plain terms, why the fix can be trusted (prior-art citations, regression tests, an independently computed before/after check), and exactly how to run scripts/codex-usage-diagnostic.mjs and what to report back: the printed output block, in full, and nothing else. Cross-linked from USAGE-SCORECARD-METRICS.md §15 so a maintainer pointed at either document finds the other. * feat(usage): stop dropped-connection/rate-limit/auth-failure turns from appearing as a $0 model Claude Code synthesizes a local placeholder assistant turn — model: "<synthetic>", isApiErrorMessage: true, every usage field zero — when a request's connection drops, hits a rate limit, or fails auth before a real completion returns. Before this change, that placeholder's model id was treated like any other and surfaced in "Models in Play" as its own ranked row: real-looking, $0.00, 0 tokens, N responses — misleading, since no model ever ran. parseClaude now branches on isApiErrorMessage: the turn still counts toward rec.responses and the punchcard (it's real engaged time — a request was genuinely in flight), but is never pushed into rec.models and never reaches addUsage(), so it can no longer create a byModel row of any kind. It increments a new rec.exceptions counter instead, rolled up into totals.exceptions and surfaced per-session alongside the existing sidechain/threadSource flags — inspectable in the Sessions tab, never hidden. The dashboard shows a small "N dropped/errored turns excluded" note under the Models in Play header when the count is nonzero. SCHEMA_VERSION bumped 3 -> 4: exceptions is a new required field on every cached session record, and summing `undefined` from a stale v3-cached record into totals.exceptions silently produces NaN (serialized as null over JSON) instead of a real count. This was caught by querying the real on-disk cache during verification, not by the unit tests, which only ever exercise a fresh parse — a live query against this machine's actual 90-day index (1,656 sessions) returned totals.exceptions: null and still showed a <synthetic> byModel row on the first run after this change; re-querying after the schema bump returned totals.exceptions: 20 with <synthetic> gone from byModel entirely. docs/USAGE-SCORECARD-METRICS.md §10 documents the behavior, the real verified numbers, and the cache-schema gotcha; every citation in the document (65 across usage-index.mjs and dashboard-server.mjs) was re-verified against the final file state after this change, since several edits landed at different points in the file and shifted line numbers more than once. * fix(usage): attribute transcript turns by kind — a tool result is never "you" The Messages API records tool results and harness context injections under role "user", so the transcript reader's role-only labeling attributed the harness's work to the person: a measured real session had 20 human prompts and 276 tool results, every one labeled YOU. parseClaude now stamps each user turn with kind — 'prompt' (the person, including image-only pastes that carry no text block), 'tool-result' (a tool_result block fed back after a tool call), or 'context' (isMeta harness injections) — via userTurnKind, which is deliberately broader than isHumanPrompt on the image-only edge: "not countable as a text prompt" and "not the human" are different claims. The prompt boolean (which drives prompt counts) is unchanged. Codex user turns are kind 'prompt' by construction — rollouts only record real prompts as user_message events. The renderer labels from kind, never role: "you" is reserved for prompts; tool results and context render purple (rhyming with the tool chips) with a hover title stating the harness — not the person — sent them. maskTurns already passes non-string fields through, so no wire change; turns are never cached, so no SCHEMA_VERSION bump — both properties now pinned by tests. Verified against real transcripts from both providers: a Claude session with 884 turns ({prompt:20, context:6, tool-result:276, assistant:588}, zero missing kinds) and a Codex rollout (8 user turns, all 'prompt'). Docs: new docs/TRANSCRIPTS.md — the maintainer reference for the whole per-session pipeline (stores → parsers → turn model → readSession gates → masking/truncation → HTTP → UI), every citation content-verified; ADR-0009 §8 amendment records the misattribution as the honesty failure it was; USAGE-SCORECARD-METRICS.md citations remapped (67 checked, 0 failures) after the line shifts. OpenRouter ingestion tracked as #59. * feat(usage): render harness sentinel markup as structure, not XML soup Claude Code writes XML wrappers into transcript text — slash commands as a <command-name>/<command-message>/<command-args> triple, injected context as <system-reminder> / <local-command-caveat> / <local-command-stdout> blocks. The transcript reader rendered them as literal angle-bracket soup. Measured against the real corpus, exactly six shapes occur (command-contents/stderr: zero), so fmtHarness handles exactly those: the command triple becomes a "/clear"-style chip (args kept; the redundant message becomes a hover title only when it differs from the name), and the three block wrappers become quiet labelled panels (system reminder / caveat / command output). Presentation only, per the same no-silent-alteration rule that governs masking: wrapped content stays verbatim, only the wrapper tags become styling. Runs on escaped text after markRedactions — capture groups can never smuggle unescaped markup — and an unmatched tag (e.g. cut mid-block by turn truncation) is left raw rather than half-formatted. Verified end-to-end by extracting the served function from the live page and executing it against the real sentinel shapes (chip, panel, and unmatched-tag cases all exact). The client JS lives inside a server-side template literal, so the regex backslashes are doubled — \1 would otherwise parse as an octal escape and take the whole page down with a SyntaxError (caught by node --check before it shipped). Docs: TRANSCRIPTS.md §6.2 documents the formatter and the measured tag census; citation integrity re-verified across both maintainer docs (127 citations: content-checked, one out-of-range range and two remap-corrupted entries fixed). * fix(usage): harness-output envelopes are neither the person nor the model An open-ended envelope census (the first census only counted tags it guessed at — that was the flaw) found three user-role families still labelled "you": task-notification (550 in the corpus), bash-stdout (85), and local-command-stdout (60). None carries isMeta or a tool_result, so text shape is the only signal — and the answer to "is it the person or the model?" is neither: the HARNESS wrote them. - userTurnKind now classifies a user turn whose text opens with a harness-output envelope (HARNESS_OUTPUT_RE: task-notification, bash-stdout/-stderr, local-command-stdout/-stderr, local-command-caveat) as 'context'. bash-input (a "! command" the person typed) and slash-command records stay 'prompt' — the person initiated those. - isHumanPrompt shares the same rule, so prompt COUNTS stop being inflated by stdout dumps and notifications. On the exact session reported (keel, "Promote provisioning-fly-hetzer pipeline"): 32 claimed prompts -> 20 real ones; bash-input=prompt, bash-stdout=context, task-notification=context, verified live. - SCHEMA_VERSION 4 -> 5: cached session records carry the old inflated prompts figure and must be re-derived. - fmtHarness learns the new shapes: bash-input renders as a "!"-prefixed chip; task-notification / bash-stdout / bash-stderr / local-command-stderr join the labelled quiet panels. Tests cover the envelope rules at both layers; docs updated (TRANSCRIPTS.md §3.2 taxonomy + census, USAGE-SCORECARD-METRICS.md §2 prompts definition), and all 128 citations across both maintainer docs re-verified against the shifted sources. * docs: add dual-theme figure set; restructure around active fact vs history Seven hand-authored SVG figures (docs/assets/, light+dark via a prefers-color-scheme block, palette machine-validated for CVD safety in both modes) now illustrate the transcript and scorecard pipelines: the per-session gate corridor, the scan/reader fork, turn-kind attribution, mask-then-truncate ordering, the classifier tournament, the confidence formula, and per-turn vs cumulative token accounting. All seven docs directly under docs/ are restructured on one rule: the main body states only what is presently true; fix narratives, incident stories, and ADR citations move to per-doc appendices (Fix history / Verification / Design-rationale ADR maps). METRICS keeps its §0-§14 numbering stable so code comments citing those sections stay valid; §15-§17 became Appendices A/B/D. The "pinned to commit" citation disclaimer is replaced by a machine-check contract (the gate lands in the next commit). Also corrects a miscitation in the Bug B evidence trail (C4/C5/C6 -> C5/C6/C7) and scrubs the same history/ADR language from the figures themselves. * test(docs): gate every file:line citation against the current source tests/kit/doc-citations.test.mjs extracts every backticked file:line citation from the two usage docs and asserts that an identifier or quoted string named beside the citation appears within the cited range (+/-10 lines) of the source as it exists now — so citation rot fails pnpm test with a re-anchoring hint instead of accumulating silently behind a "line numbers will drift" disclaimer. Maiden run: 93/96 METRICS citations were already fresh against HEAD, one real drift was caught and re-anchored (the midnight-boundary test, 556 -> 608), and two citations were strengthened to name their subject. Matching is whitespace-normalized on both sides and fence-aware, so neither doc wrapping nor source wrapping can defeat an honest anchor. Also updates the diagnostic script's stale §15 pointer to Appendix A.
1 parent 4999ddc commit 173dfff

21 files changed

Lines changed: 3336 additions & 44 deletions

‎docs/CODEX-USAGE-DIAGNOSTIC.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Verifying the Codex usage-scorecard fix on your own machine
2+
3+
**For:** anyone who uses Codex CLI and wants to check whether the two bugs
4+
fixed on this branch affected their own numbers — without sharing any
5+
transcript content, session ids, titles, or file paths with anyone.
6+
7+
**You don't need to read the rest of this repo to use this document.**
8+
Everything you need is below: what was wrong, why the fix can be trusted
9+
without re-auditing the code yourself, how to run one script, and exactly
10+
what to send back.
11+
12+
---
13+
14+
## The short version
15+
16+
A usage dashboard in this project ([`ak x dashboard`](../README.md), Usage
17+
tab) reads local Claude Code and Codex CLI session logs and estimates what
18+
they'd have cost at list-price API rates. On one real machine, the Codex
19+
side of that estimate looked implausible — a single model row showing
20+
roughly $935,000 and 1.46 trillion tokens, attributed to under 2,000
21+
sessions, with **zero** recorded responses despite that volume.
22+
23+
Investigation found two real bugs in how this project's code parses Codex's
24+
session logs (`src/lib/usage-index.mjs`, function `parseCodex`) — nothing
25+
wrong with Codex CLI itself, and nothing you did. Both are fixed on this
26+
branch. This document lets you check, on your own machine, whether either
27+
bug was actually inflating *your* numbers, and by how much.
28+
29+
## What was wrong
30+
31+
**Bug A — response counts were dropped for Codex.** Cosmetic only: every
32+
Codex model row showed "0 responses" in the dashboard regardless of real
33+
activity. Never affected any dollar or token figure.
34+
35+
**Bug B — subagent delegation could double-bill tokens.** This is the one
36+
that matters for your numbers. Codex CLI's `thread_spawn` subagent
37+
delegation writes a session-log file for the spawned subagent that
38+
**replays its parent thread's entire prior token history** as duplicate
39+
events, before the subagent's own new turns even begin. This is documented,
40+
previously-reported Codex CLI behavior:
41+
42+
- [`openai/codex#14489`](https://github.com/openai/codex/issues/14489) —
43+
Codex re-emitting a stale cumulative token count on rate-limit-only
44+
updates, which a naive reader double-counts.
45+
- [`ccusage/ccusage#884`](https://github.com/ccusage/ccusage/issues/884) —
46+
a different Codex usage-analytics tool (same job as this project's Usage
47+
tab) documenting that summing raw cumulative snapshots instead of taking
48+
the last one matched only 131 of 732 real sessions correctly.
49+
- [`ccusage/ccusage#950`](https://github.com/ccusage/ccusage/issues/950) —
50+
the closest match to what was found here: a parent session that spawned
51+
12 subagents had its usage effectively counted 13 times over, because
52+
every subagent's log replayed the full parent history. Measured **91×**
53+
cost inflation in a real corpus (reported ~$9,041 against actual spend of
54+
~$100).
55+
56+
This project's parser took each Codex log file's final cumulative token
57+
count at face value, with no check for whether that file was a subagent
58+
replaying its parent's history. The fix reads the field Codex CLI itself
59+
writes to mark this (`session_meta.thread_source`) and excludes a
60+
`"subagent"`-sourced file's tokens/cost from every total — while still
61+
showing the session itself in the dashboard's session list, so nothing is
62+
silently hidden.
63+
64+
## Why you can trust this without re-reading the diff
65+
66+
Three independent layers, so you don't have to take any single one on
67+
faith:
68+
69+
1. **Prior art.** The three GitHub issues above describe this exact Codex
70+
CLI behavior, observed by other people, in other tools, before this
71+
project encountered it.
72+
2. **Regression tests.** The fix is pinned by test cases built from the
73+
documented bug signature (`tests/kit/usage-index.test.mjs`), currently
74+
passing 87/87.
75+
3. **A before/after comparison you can run yourself**, on your own data,
76+
computed by code that does **not** import or reuse the fix — so it's an
77+
independent check, not a restatement of the same claim. That's the
78+
script below.
79+
80+
If you want the full engineering detail — exact formulas, file:line
81+
citations, provider pricing sources — that's
82+
[`docs/USAGE-SCORECARD-METRICS.md`](USAGE-SCORECARD-METRICS.md). You do not
83+
need to read it to use this document.
84+
85+
## How to check your own numbers
86+
87+
You'll need Node.js 18 or newer (`node --version` to check) and a terminal.
88+
Nothing else — no `npm install`, no cloning this repo, no network access
89+
beyond the one download below, and nothing gets written to your disk except
90+
the script file itself.
91+
92+
**1. Get the script.** If you already have this branch checked out:
93+
94+
```bash
95+
node scripts/codex-usage-diagnostic.mjs
96+
```
97+
98+
If you don't have the repo, just download the one file:
99+
100+
```bash
101+
curl -fsSL https://raw.githubusercontent.com/pacphi/agentic-kit/fix/codex-usage-scorecard-metrics/scripts/codex-usage-diagnostic.mjs -o codex-usage-diagnostic.mjs
102+
node codex-usage-diagnostic.mjs
103+
```
104+
105+
**2. Read the output before sending anything.** The script prints a report
106+
with two totals: **"before the fix"** (every session log counted — this
107+
should be close to what you'd currently see if you ran the unpatched
108+
dashboard) and **"after the fix"** (subagent-replay logs excluded). It also
109+
tells you what percentage of tokens/cost, if any, is attributable to
110+
subagent-replay logs.
111+
112+
The script's own source is short and readable — open it in an editor if
113+
you want to confirm for yourself what it does before running it. In short:
114+
it walks `~/.codex/sessions/**/rollout-*.jsonl`, reads exactly three kinds
115+
of field per line (`type`, `session_meta.thread_source`, and the numeric
116+
fields inside `token_count.info.total_token_usage`), and never touches,
117+
stores, or prints anything else — no prompts, no titles, no session ids, no
118+
file paths, no timestamps. There is no code path in the script that could
119+
emit those even by accident, because it never reads them into a variable in
120+
the first place.
121+
122+
## What to send back
123+
124+
**Just the printed output block, in full — nothing else.** That's the
125+
complete, minimal report; there's no need to trim it further or describe
126+
your sessions in your own words. If you'd rather have a copy-pasteable
127+
single blob instead of the human-readable report, run it with `--json`
128+
instead.
129+
130+
If "before" and "after" are close (a low or zero percentage excluded), this
131+
fix doesn't materially change what you were seeing, and the earlier
132+
discrepancy has a different explanation worth digging into further. If a
133+
meaningful percentage is excluded, that's the concrete number behind it —
134+
not a guess.
135+
136+
## Appendix — references
137+
138+
- Script: [`scripts/codex-usage-diagnostic.mjs`](../scripts/codex-usage-diagnostic.mjs)
139+
- Full metrics reference: [`docs/USAGE-SCORECARD-METRICS.md`](USAGE-SCORECARD-METRICS.md) (its Appendix A covers these two bugs in full engineering detail)
140+
- Design record: [`docs/adr/0009-usage-scorecard-local-transcript-analytics.md`](adr/0009-usage-scorecard-local-transcript-analytics.md)

‎docs/MANAGED-TOOLS.md‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,8 @@ version-detected, and displayed. This doc states the contract's four
55
invariants, maps every managed tool onto them, and gives the checklist for
66
adding a new tool without breaking them.
77

8-
The contract exists because the failure modes it prevents were all observed
9-
live: a version stamp disagreeing with what was actually on disk, a statusline
10-
showing a different version than `ak status`, a third-party self-updater
11-
rewriting managed files behind ak's back, and an "update available" banner
12-
that stayed silent for tools it didn't know about.
8+
Each invariant traces to a live failure it prevents — the appendix records
9+
them.
1310

1411
## The four invariants
1512

@@ -28,9 +25,9 @@ that stayed silent for tools it didn't know about.
2825

2926
3. **Same-namespace comparisons.** Installed and latest are always compared
3027
in the same version namespace: npm semver vs npm semver, GitHub release
31-
tag vs release tag. A comparison across namespaces (the original
32-
ruvnet-brain bug: plugin semver `0.5.0-dev` vs release tag `3.0.1`) can
33-
never converge. "Latest" is not always npm-latest either — agentdb's
28+
tag vs release tag. A comparison across namespaces can never converge
29+
(the appendix records the live case). "Latest" is not always npm-latest
30+
either — agentdb's
3431
authority is ruflo's *bundled* version, because a latest-chasing agentdb
3532
is the store-corruption risk its coherence guard exists to prevent.
3633

@@ -98,3 +95,14 @@ dashboard.
9895
6. **Lock it with tests**: the install spec + suppression flags (regression
9996
lock), the disk-read parser's edge cases (missing / malformed / junk),
10097
and the display resolution order.
98+
99+
## Appendix — the observed failures behind the contract
100+
101+
Each invariant exists because its failure mode was observed live, not
102+
hypothesized: a version stamp disagreeing with what was actually on disk; a
103+
statusline showing a different version than `ak status`; a third-party
104+
self-updater rewriting managed files behind ak's back; an "update available"
105+
banner that stayed silent for tools it didn't know about; and the original
106+
ruvnet-brain drift bug — a cross-namespace comparison (plugin semver
107+
`0.5.0-dev` vs release tag `3.0.1`) that could never converge, the live case
108+
behind invariant 3.

‎docs/PROVIDERS.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -150,8 +150,7 @@ When **both** hosts are enabled and `agentic-qe ≥ 3.13.1` is installed, `ak` s
150150
review, …) is routed to the host and model that suits it — Claude for reasoning/review,
151151
Codex for execution — and materialized into `.agentic-qe/llm-config.json` (`agentOverrides`).
152152
It's seeded automatically on `ak x provider pick` / `ak setup`; nothing happens for
153-
claude-only projects. Grounded in ruflo's own dual-mode templates (see
154-
[docs/adr/](adr/) — ADR-0001..0005).
153+
claude-only projects.
155154

156155
```bash
157156
ak x provider pick --host claude,codex # enables both → seeds routing → prints the table
@@ -215,7 +214,7 @@ The two config stores each knob below lives in — and their precedence — are
215214

216215
| You want to… | `ak` way | The raw ruflo/aqe way it maps to |
217216
| ------------------------------------ | --------------------------------- | --------------------------------------------------- |
218-
| Enable claude/codex hosts | `ak x provider pick` | `ENABLE_CLAUDE_CODE` / `ENABLE_CODEX` env (ADR-034) + `ruflo init --dual` |
217+
| Enable claude/codex hosts | `ak x provider pick` | `ENABLE_CLAUDE_CODE` / `ENABLE_CODEX` env + `ruflo init --dual` |
219218
| Register a ruflo LLM provider | `--provider openai:gpt-5.6` | `ruflo providers configure -p openai -m gpt-5.6` |
220219
| Set which LLM runs QE | `--aqe-provider gemini` | `AQE_LLM_PROVIDER=gemini` (env) |
221220
| Order QE's fallback chain | `--aqe-fallback '…'` | edit `.agentic-qe/llm-config.json` / `aqe llm-router config` |
@@ -240,3 +239,12 @@ status` and `ak sync` keep everything converged and flag drift in between.
240239
**The shape of the whole thing:** Level 0 is the 90% case and costs nothing. Each level up is
241240
one flag, and the bottom is always the tools' own knobs — `ak` never traps your config, it
242241
just makes the good default automatic and the customization reversible.
242+
243+
## Appendix — design references
244+
245+
- Per-activity routing and dual-host seeding: [docs/adr/](adr/) ADR-0001..0005;
246+
grounded in ruflo's own dual-mode templates.
247+
- Primary-host selection and ambidextrous mirroring:
248+
[ADR-0006](adr/0006-primary-host-and-ambidextrous-mirroring.md).
249+
- Host env flags (`ENABLE_CLAUDE_CODE` / `ENABLE_CODEX`): upstream ruflo
250+
ADR-034, "Optional MCP Backends".

0 commit comments

Comments
 (0)