Skip to content

Commit 4f41a8e

Browse files
authored
feat(usage): answer today's cost from the CLI in one command (#2396)
* devlog: plan CLI usage cost query A user asking "how much did Grok cost me today?" could not answer it with ocx usage. The proxy prices every request at read time, then the CLI discards the breakdown before printing: summaryLines() stops at depth 1 and renders models/providers/accounts as "N item(s)". There is also no today window, day rows carry no cost field at all, and --surface selects the client rather than the upstream provider, so --surface grok answers a different question than the one being asked. Roadmap unit with diff-level decade docs (data -> api -> cli -> verify) plus the A-phase audit fold-back. Two independent reviews found seven blockers, including one that would have poisoned the shared usage cache: the cache key is range:surface, so a filtered summary written under it would serve filtered totals to the next unfiltered caller, GUI included. * feat(usage): add a today window and day-level cost attribution The proxy prices every request as it reads the log, but day rows carried only requests and tokens, so per-day cost did not exist anywhere - not even in --json. Anyone asking what today cost had to re-derive prices the proxy had already computed. buildDayGrid now prices through the same seam buildModels uses: combo requests are priced per attempt and each attempt's cost is attributed to its own model, everything else to the entry's model. Pricing it a second way would have made days[] disagree with models[] for exactly the combo traffic where the disagreement is hardest to spot. Three details worth naming: - rangeWindow has no exhaustive switch; its final return is the "all" window. A today member that failed to reach its own branch would compile clean and silently report all-time history, so today is handled first and the test asserts since is bounded rather than only checking the day count. - The day overflow row now sums cost. Past 256 distinct provider/model pairs the tail collapses into one "other" row, and an unsummed aggregation looks correct in every breakdown below the cap. The regression test puts the only priced model in the tail and was driven red before it passed. - 1d normalises to today at the parser instead of becoming a second union member, which would have needed its own cache slot and grid arm for nothing. USAGE_RANGES and USAGE_SURFACES are exported so consumers stop re-declaring the members in their own literals. * feat(usage): filter /api/usage by provider and model Narrowing usage to one provider previously meant fetching the whole window and filtering client-side. --surface looked like it should help and does not: it selects the client (codex/claude/grok), not the upstream provider, so --surface grok answers a different question than the one being asked. The filter is a projection over a finished summary, never a parameter to summarizeUsage. That is not a style preference. The cache key is range:surface and the warm loop writes every key on a miss, so a filtered summary that reached the producer would be stored under the unfiltered key and the next caller - the dashboard included - would be served one provider's totals as the whole window. Keeping the filter outside the producer makes that unrepresentable rather than merely discouraged. The regression test for it was verified by falsification: projecting into the cached value makes it fail, and it passes again when reverted. Totals are recomputed from retained rows, which is exact for cost (combo cost is attributed per attempt, so it partitions) but can overcount requests when a combo request participates in several models. comboOverlap reports when that is possible instead of leaving it silent. accounts is emptied under a filter because account rows cannot be honestly re-partitioned by provider, and showing whole-window account totals beside filtered model totals invites the wrong reading. The warm loop now iterates USAGE_RANGES/USAGE_SURFACES instead of its own literals. A subset literal type-checks happily, which is why today would have been silently unwarmed and never invalidated with its siblings. The cache-count assertions in settings-stream-mode are derived from the same constants for the same reason. * feat(cli): print the usage cost breakdown instead of an item count ocx usage rendered its payload through summaryLines(), a generic depth-1 flattener shared with storage/memory/debug/claude-inbound/injection. It stops at depth 1 and renders any array as "N item(s)", so models, providers and accounts printed as a row count and every per-entity cost the server had already computed was discarded. The only cost a user could see was the whole-window total across every provider, which is not the number anyone is asking for. usage now has its own renderer rather than a deeper shared one, because deepening summaryLines() would change five unrelated commands. It follows the existing house style: dynamic padEnd columns like formatAccountTable, plain text, no ANSI. Two wording decisions are load-bearing. A zero total is ambiguous between "no spend" and "no price row matched", so unpriced and unmetered counts are shown separately. And most traffic through this proxy is subscription or OAuth-plan based where no per-request charge exists at all, so the disclaimer is not decoration - a bare dollar figure would be read as a bill. Both borrow the dashboard's existing wording so the two surfaces agree. --provider and --model are registered in all three places the CLI needs (the observe USAGE constant, the registry entry, and the help banner); the banner line was also stale, omitting --surface and --json. Live evidence in devlog 031: the question that started this unit now answers in one command. * docs: document the usage today window and provider filter English carries the full explanation, including the distinction that keeps catching people out: --surface selects the calling client, --provider selects the upstream target serving the request. Locales get the flag list, which is code-shaped and locale-independent, so they cannot contradict the English source while awaiting translation. Also states why a zero cost is not automatically free: requests with no matching price row are reported as unpriced/unmetered rather than folded in. * fix(usage): re-summarise filtered windows instead of projecting rows Review found three defects that shared one cause: the projection operated on breakdown rows, which have already lost the identity it needed. - A provider or model living only past MAX_USAGE_MODEL_BREAKDOWN_ROWS is collapsed into a synthetic "other" row, so filtering for it reported matched:false and zero cost despite real usage. - A provider row is a whole-provider aggregate. Under a model filter, providers[] kept the provider's other models while models[] and the totals excluded them - one response contradicting itself. - A model row carries a single optional cost, so priced/unpriced could only be guessed per model rather than counted per request, and unmetered was always reported as zero. Filtering now re-summarises from the entries the summary was built from. The entries are already in hand on every path that filters, so the honest computation is also the simpler one. Filtered requests skip the cache, since a cached summary carries no entries and serving one would mean projecting over collapsed rows again; the unfiltered cache stays warm either way. Also: build the human report only when it will be printed. Arguments are evaluated before the call, so passing it inline ran the renderer during --json and let its assumptions touch a path meant to bypass it. And 1d now appears in every synopsis that already listed today - the alias was accepted but undocumented in all eight of them. * fix(usage): filter combos by attempt, and scope matched to the window Two more review findings, both reproduced before fixing. Keeping a whole combo entry because one of its attempts matched dragged the other attempts into the filtered totals. A two-attempt combo filtered to its cheap model reported ~$0.368 instead of ~$0.008 - the expensive model's spend, attributed to a model the user had explicitly filtered out. The predicate now applies at attribution level and the entry is rewritten down to its matching attempts, so filtered numbers mean what the flag says. Combo requests are still counted once per participating model; that overlap is documented and is what comboOverlap reports. filter.matched was computed from entries scanned before the range and surface predicates ran, so a match from an earlier day set matched:true for --range today while the summary showed zero requests. Since the CLI uses that flag to choose between a table and "no usage recorded", the message and the numbers could disagree. It now derives from the projected summary. Also adds --model to the top-level help synopsis, which listed only --provider. * fix(usage): spend each day-attribution cost once dayAttributionCosts keys by provider/model and holds the SUM of every attempt that shares a key, but usageAttributions yields one entry per attempt. A retry onto the same model therefore added that pair's total once per attempt, doubling days[].estimatedCostUsd against summary.estimatedCostUsd. buildModels never had this bug because it adds each attempt once. Delete the key on read so the first attribution carries the group's cost and its siblings get zero. The regression test uses two attempts on the SAME model, which is the case every existing combo test misses -- they all use two different models.
1 parent 46d4150 commit 4f41a8e

26 files changed

Lines changed: 1863 additions & 37 deletions

File tree

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# 260822 — CLI usage cost query
2+
3+
## Why this unit exists
4+
5+
A user asked, in order: "how much Grok did we use on ocx today?" and then
6+
"what did it cost?". Neither question could be answered with `ocx usage`.
7+
8+
Answering the first one required piping `ocx usage --range 7d --json` into a
9+
separate script that filtered `days[]` by date and `models[]` by provider.
10+
Answering the second one was not possible from the day breakdown at all,
11+
because the day breakdown carries no cost field. The proxy knows every number
12+
involved — it prices each request at read time — and the CLI throws all of it
13+
away before it reaches the terminal.
14+
15+
That is the defect. Not a missing feature: a reporting surface that computes
16+
the answer and then declines to print it.
17+
18+
## The four concrete gaps
19+
20+
1. **The default CLI view drops the cost breakdown.**
21+
`usage()` in `src/cli/observe.ts` hands the `/api/usage` payload to
22+
`summaryLines()` (`src/cli/runtime-api.ts`), a generic key/value flattener
23+
that stops at `depth > 1` and renders any array as `"models: 45 item(s)"`.
24+
`models[].estimatedCostUsd`, `providers[].estimatedCostUsd` and
25+
`accounts[].estimatedCostUsd` are all computed server-side and all
26+
invisible. `--json` is the only way to see them, which makes the human
27+
view strictly less useful than piping to `python3`.
28+
29+
2. **There is no "today".** `UsageRange` is `7d | 30d | all`. The most
30+
common question a cost surface gets asked — what am I spending right now —
31+
has no direct answer.
32+
33+
3. **Day rows carry no cost.** `UsageDay` and `UsageDayModel` have
34+
`requests`, `totalTokens` and nothing else. So even in `--json`, per-day
35+
cost does not exist. A caller who wants it must re-derive prices the proxy
36+
already computed.
37+
38+
4. **There is no way to ask about one provider.** Narrowing to xAI means
39+
fetching the whole window and filtering client-side. `--surface` looks
40+
like it might help and does not: it selects the *client* (codex / claude /
41+
grok), not the upstream provider. That near-miss actively misleads —
42+
`--surface grok` returns Grok-client traffic, which for this user was 10
43+
requests, while their actual xAI spend that window was 1,447.
44+
45+
## Scope
46+
47+
IN: `src/usage/summary.ts`, `src/cli/observe.ts`, `src/cli/help.ts`,
48+
`src/server/management/logs-usage-routes.ts`, `src/server/management/usage-summary-cache.ts`,
49+
tests, docs-site.
50+
51+
OUT: GUI dashboard rework, price-table rates in `src/usage/cost.ts`, provider
52+
quota APIs, new dependencies, `go/`.
53+
54+
## Roadmap
55+
56+
| Doc | Work-phase | Deliverable |
57+
|-----|-----------|-------------|
58+
| `001` | — | Current-state inventory (read-only research) |
59+
| `010` | wp2 | Data layer: `today` range + day-level cost fields |
60+
| `020` | wp3 | API layer: range/provider/model parsing, cache-key correctness |
61+
| `030` | wp4 | CLI layer: cost-bearing renderer, flags, help |
62+
| `040` | wp5 | Verification, docs sync, PR against `dev` |
63+
64+
Dependency order, not effort order: `030` cannot render a number `010` does
65+
not compute, and `020` cannot filter a window `010` does not define.
66+
67+
## Non-negotiables
68+
69+
- `--json` stays backward compatible. Fields may be added; none renamed or removed.
70+
- A zero cost must never be presented as "free". Unpriced and unmetered
71+
requests are counted and shown separately, because `estimatedCostUsd: 0`
72+
today can mean "no matching price row" rather than "no spend".
73+
- OAuth-plan providers genuinely have no per-request dollar price. The surface
74+
must say so instead of printing a confident `$0.00`.
Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# 001 — Current-state inventory
2+
3+
Read-only survey of the code the later phases touch. Every claim below was
4+
checked against `dev` at `7185ecc80`.
5+
6+
## The request path
7+
8+
```
9+
ocx usage -> src/cli/observe.ts usage()
10+
-> GET /api/usage?range&surface
11+
-> src/server/management/logs-usage-routes.ts (cache + parse)
12+
-> src/usage/summary.ts summarizeUsage()
13+
-> src/usage/cost.ts estimateRequestCost()
14+
```
15+
16+
Cost is computed at the bottom of that stack and survives all the way back to
17+
the CLI. It dies in the last four lines.
18+
19+
## Gap 1 — the renderer discards what it is given
20+
21+
`src/cli/observe.ts:129` `usage()` ends with:
22+
23+
```ts
24+
const result = await runtimeRequest(`/api/usage${query({ range, surface })}`, {}, deps);
25+
printData(result, wantsJson, summaryLines(result));
26+
```
27+
28+
`summaryLines()` (`src/cli/runtime-api.ts:294`) is a generic DTO flattener:
29+
30+
- `depth > 1` returns early, so nested rows are never walked.
31+
- an array of objects renders as `${child.length} item(s)`.
32+
33+
So `models`, `providers`, `accounts` and `days` — the four arrays that carry
34+
every per-entity cost — print as `models: 45 item(s)`. The totals block does
35+
print, because it is scalar at depth 1; `summary.estimatedCostUsd` is
36+
therefore the *only* cost number a user sees, and it is the whole-window
37+
total across every provider.
38+
39+
This is a shared helper used by `storage`, `memory`, `debug`,
40+
`claude-inbound` and `injection`. It must not be changed to serve usage:
41+
those callers are well-served by a flat view. Usage needs its own renderer.
42+
43+
## Gap 2 — no today window
44+
45+
`src/usage/summary.ts:7` `export type UsageRange = "7d" | "30d" | "all"`.
46+
47+
`rangeWindow()` (`:145`) already does local-midnight arithmetic through
48+
`startOfLocalDay()` (`:139`), so a `today` member is a two-line addition:
49+
`{ since: startOfLocalDay(now), days: 1 }`. The helper it needs exists.
50+
51+
`parseRange()` (`:129`) falls back to `"30d"` for anything unrecognised. It
52+
never throws, so an unknown `--range` silently returns a month of data. The
53+
CLI guards this itself (`observe.ts:132`) — that guard list must be updated in
54+
lockstep or `--range today` is rejected at the CLI before the server ever
55+
sees it.
56+
57+
## Gap 3 — day rows have no cost field
58+
59+
`UsageDay` (`:37`) and `UsageDayModel` (`:46`) declare
60+
`requests / measuredRequests / reportedRequests / totalTokens / models` and
61+
`model / provider / requests / attemptCount / totalTokens` respectively.
62+
Neither has `estimatedCostUsd`.
63+
64+
`buildDayGrid()` (`:338`) is the single construction site, and it has **four**
65+
places that materialise these objects:
66+
67+
1. `bumpDayModel()` `:351` — creates a `UsageDayModel`.
68+
2. the pre-fill loop `:369` — creates empty `UsageDay` rows for the grid.
69+
3. the entry loop `:375` — creates a `UsageDay` for a date outside the grid.
70+
4. the `retainedBreakdownRows` overflow collapse `:390` — synthesises an
71+
`other` row by summing the tail.
72+
73+
All four must set the new field, and (4) must *sum* it, or the overflow row
74+
silently zeroes the cost of everything past row 255.
75+
76+
Note the day loop attributes per-entry via `usageAttributions(entry)`, which
77+
is combo-aware (one request can attribute to several provider/model pairs).
78+
Cost attribution must go through the same seam that `buildModels()` uses
79+
(`estimateAttemptCost` for combo attempts, `estimateRequestCost` otherwise) —
80+
see `:470` and `:477` — rather than re-deriving a price, or day totals will
81+
disagree with model totals for combo traffic.
82+
83+
## Gap 4 — surface is not provider
84+
85+
`UsageSurface` is `all | codex | claude | grok` and selects the **client**
86+
that made the call. `--surface grok` means "requests that came from the Grok
87+
client", not "requests served by xAI".
88+
89+
For the user who prompted this unit those two readings differed by two orders
90+
of magnitude: `--surface grok` reported 10 requests for the 30d window, while
91+
their xAI provider traffic for that window was 1,447 requests in a single day.
92+
A flag that looks like the answer and returns a different number is worse than
93+
no flag; the fix is a real `--provider` (and `--model`) filter, not a
94+
redefinition of `--surface`.
95+
96+
## The cache is the sharp edge
97+
98+
`/api/usage` (`logs-usage-routes.ts:197`) is cached, and the cache is
99+
**precomputed as a cross-product**:
100+
101+
```ts
102+
const ranges: UsageRange[] = ["7d", "30d", "all"]; // :247
103+
const surfaces: UsageSurface[] = ["all", "codex", "claude", "grok"];
104+
for (const nextRange of ranges) for (const nextSurface of surfaces) { ... }
105+
setUsageSummaryCacheEntry(`${nextRange}:${nextSurface}`, ...)
106+
```
107+
108+
Three consequences for this unit:
109+
110+
1. That literal array is a second, independent definition of `UsageRange`'s
111+
members. Adding `today` to the type does not add it here — TypeScript is
112+
perfectly happy with a subset. `today` would simply never be warmed, and
113+
worse, would never be *invalidated* alongside its siblings.
114+
2. `refreshedUsageSummary()` (`:105`) re-derives `since` from
115+
`rangeWindow(range, now)` when serving a cache hit, so a `today` entry
116+
served from cache does get a correct `since` — but the **rows** inside it
117+
are whatever the window was when it was computed. `usageSummaryExpiresAt()`
118+
(`:96`) already expires every entry at `nextLocalMidnight(now)`, which is
119+
exactly the boundary `today` needs. That is a real piece of luck: the
120+
existing expiry policy is correct for a today-window without modification.
121+
3. Adding `provider`/`model` to the cache key would multiply the cross-product
122+
by the cardinality of providers × models — unbounded. The filters must
123+
therefore be applied **after** the cache lookup, as a projection over an
124+
already-summarised payload, not as another cache dimension.
125+
126+
That last point decides the architecture of phase `020`: filtering is a
127+
post-summary projection, not a summarisation parameter.
128+
129+
## Test surface
130+
131+
`tests/usage-summary.test.ts`, `tests/usage-cost.test.ts`,
132+
`tests/api-usage.test.ts`, and the `tests/cli-*.test.ts` family. The CLI
133+
tests are the convention to match for stdout capture; the exact helper is
134+
confirmed in phase `030` before writing new tests.
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# 010 — Data layer: `today` range and day-level cost
2+
3+
Work-phase **wp2**. Depends on nothing. Everything after this consumes it.
4+
5+
## Change map
6+
7+
### `src/usage/summary.ts`
8+
9+
**1. Range type and parser**
10+
11+
```ts
12+
export type UsageRange = "today" | "7d" | "30d" | "all";
13+
14+
export function parseRange(input: string | null | undefined): UsageRange {
15+
if (input === "today" || input === "1d") return "today";
16+
if (input === "7d" || input === "30d" || input === "all") return input;
17+
return "30d";
18+
}
19+
```
20+
21+
`1d` normalises to `today` at the parse boundary, so exactly one member
22+
reaches the rest of the system. The alias never becomes a second enum value —
23+
that is deliberate: a second member would need its own cache slot, its own
24+
grid arm, and its own test matrix for zero user-visible gain.
25+
26+
**2. `rangeWindow()`**
27+
28+
```ts
29+
if (range === "today") return { since: startOfLocalDay(now), days: 1 };
30+
```
31+
32+
Placed first. `startOfLocalDay()` already exists at `:139` and is the same
33+
helper `7d`/`30d` use, so the day boundary is consistent by construction
34+
rather than by a second implementation that agrees today and drifts later.
35+
36+
**3. Day-level cost fields**
37+
38+
```ts
39+
export interface UsageDay {
40+
// ... existing fields unchanged ...
41+
estimatedCostUsd: number;
42+
}
43+
44+
export interface UsageDayModel {
45+
// ... existing fields unchanged ...
46+
estimatedCostUsd: number;
47+
}
48+
```
49+
50+
Required (not optional). `UsageModel.estimatedCostUsd` is optional today
51+
because a model row can exist with no priced request at all; a day row is
52+
always constructed by us and always summable, so `0` is the honest value and
53+
`undefined` would only push a null check into every consumer.
54+
55+
## The four construction sites (all must be updated)
56+
57+
`buildDayGrid()` materialises these objects in four places. Missing any one
58+
leaves a row whose cost is `undefined` at runtime despite a required type —
59+
the object literals are the only thing TypeScript checks here.
60+
61+
| # | Line | Site | Action |
62+
|---|------|------|--------|
63+
| 1 | `:351` | `bumpDayModel` creates `UsageDayModel` | init `estimatedCostUsd: 0` |
64+
| 2 | `:369` | grid pre-fill creates empty `UsageDay` | init `estimatedCostUsd: 0` |
65+
| 3 | `:375` | entry loop creates off-grid `UsageDay` | init `estimatedCostUsd: 0` |
66+
| 4 | `:390` | `retainedBreakdownRows` overflow `other` row | **sum** the tail's costs |
67+
68+
Site 4 is the one that fails silently. It synthesises an aggregate row from
69+
everything past breakdown row 255; if it does not sum `estimatedCostUsd`, the
70+
`other` row reports `$0` and the day's model rows no longer add up to the
71+
day total. It is invisible in any test with fewer than 256 distinct
72+
provider/model pairs, which is every test we would naturally write.
73+
74+
## Cost attribution must reuse the existing seam
75+
76+
Do **not** re-derive prices in the day loop. `buildModels()` already
77+
establishes the correct pattern at `:470` / `:477`:
78+
79+
- combo attempts -> `estimateAttemptCost(...)`
80+
- ordinary requests -> `estimateRequestCost(...)`
81+
82+
The day loop iterates `usageAttributions(entry)`, which is combo-aware: one
83+
request can attribute to several provider/model pairs. Pricing it a second way
84+
would make `days[].estimatedCostUsd` disagree with `models[].estimatedCostUsd`
85+
for exactly the combo traffic where the disagreement is hardest to notice.
86+
87+
Implementation: compute the estimate once per entry (as the totals path
88+
already does at `:335`), then distribute per attribution using the same
89+
branch `buildModels` uses, and add the day-model contribution and the day
90+
total from that single source.
91+
92+
## Accept criteria
93+
94+
1. `rangeWindow("today", t)` returns `since === startOfLocalDay(t)` and
95+
`days === 1`, for `t` at 00:00:00, at 12:00, and at 23:59:59 local.
96+
2. `parseRange("today")` and `parseRange("1d")` both return `"today"`;
97+
`parseRange("2d")` still returns `"30d"` (unchanged fallback).
98+
3. A summary over fixtures spanning two local days, with `range: "today"`,
99+
contains exactly one day row and excludes yesterday's entries.
100+
4. `days[].estimatedCostUsd` equals the sum of that day's
101+
`days[].models[].estimatedCostUsd`.
102+
5. The sum of `days[].estimatedCostUsd` over a window equals
103+
`summary.estimatedCostUsd` for that window.
104+
6. **Overflow activation (C-ACTIVATION-GROUNDING-01):** a fixture with more
105+
than `MAX_USAGE_MODEL_BREAKDOWN_ROWS` distinct provider/model pairs in one
106+
day produces an `other` row whose `estimatedCostUsd` is non-zero and
107+
equals the summed tail. This test must be written to *fail* against an
108+
unsummed site-4 before it is accepted as passing.
109+
110+
## Verifier
111+
112+
`bun test tests/usage-summary.test.ts tests/usage-cost.test.ts`
113+
114+
Confirmed to exist and to read `src/usage/summary.ts` (both suites import from
115+
it directly).

0 commit comments

Comments
 (0)