Skip to content

Commit 16e9411

Browse files
fix: work around ccusage#1710 dropping null-model entries
- ingest-daily.sh now mirrors the ingest window's JSONL into $TMPDIR/cfg/projects with `jq 'del(.message.usage.iterations)'` and points ccusage at it via CLAUDE_CONFIG_DIR, since ccusage 20.0.20 silently drops any entry whose usage.iterations[].model is null (ccusage/ccusage#1710) - Claude Code 2.1.269 wrote null-model entries on 2026-09-12, causing a day of Fable 5.1 cost to vanish from daily_usage; totals verified identical on unaffected days since ccusage sums top-level usage - updated docs/claude-stats.html pipeline diagram and README.md's ingest-daily.sh description to document the new sanitize step Files: claude-stats/bin/ingest-daily.sh, docs/claude-stats.html, README.md
1 parent 182ffc9 commit 16e9411

3 files changed

Lines changed: 32 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ aren't tracked in git.
142142

143143
| File | Role |
144144
|---|---|
145-
| `claude-stats/bin/ingest-daily.sh` | runs `npx ccusage claude daily --json` twice (plain + `--instances --breakdown`), upserts into `daily_usage` (PK: `date,model`) and `project_daily_usage` (PK: `project_path,date,model`). Both share an 8-day rolling-rewrite window so older rows are immutable; `project_daily_usage` rolled up sums to `daily_usage` to the cent. The `claude` subcommand is required under ccusage v19+; top-level `ccusage daily` now aggregates all agents and drops `modelBreakdowns`. |
145+
| `claude-stats/bin/ingest-daily.sh` | mirrors the window's JSONL with `usage.iterations` stripped (works around [ccusage#1710](https://github.com/ccusage/ccusage/issues/1710), which drops Fable 5.1 entries), then runs `npx ccusage claude daily --json` twice (plain + `--instances --breakdown`), upserts into `daily_usage` (PK: `date,model`) and `project_daily_usage` (PK: `project_path,date,model`). Both share an 8-day rolling-rewrite window so older rows are immutable; `project_daily_usage` rolled up sums to `daily_usage` to the cent. The `claude` subcommand is required under ccusage v19+; top-level `ccusage daily` now aggregates all agents and drops `modelBreakdowns`. |
146146
| `claude-stats/bin/ingest-sessions.py` | walks `~/.claude/projects/**/*.jsonl`, parses tool-use events + usage (incl. per-session `total_output_tokens`, v6), writes `conversations`, `conversation_tool_usage`, `conversation_skill_usage`. Per-project cost lives in `project_daily_usage` (written by `ingest-daily.sh`); this script no longer touches cost. |
147147
| `claude-stats/bin/cleanup-old.sh` | 365-day retention across all five data tables |
148148
| `claude-stats/bin/build-dashboard.sh` | emits self-contained `/tmp/claude-stats-dashboard.html` (Plotly via CDN, AMOLED theme) |

‎claude-stats/bin/ingest-daily.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,24 @@ SINCE="${1:-$(date -d '8 days ago' +%Y%m%d)}"
2424
CSV="${TMPDIR}/usage.csv"
2525
PROJ_CSV="${TMPDIR}/projects.csv"
2626

27+
# --- Sanitized mirror (added 2026-09-16) ---------------------------------
28+
# ccusage 20.0.20 silently drops any entry whose usage.iterations[].model is
29+
# null (ccusage/ccusage#1710; Claude Code 2.1.269 wrote that on 2026-09-12).
30+
# Point ccusage at a mirror of the window's JSONL with `iterations` stripped.
31+
# ccusage sums top-level usage, so totals are byte-identical on unaffected
32+
# data; harmless once a fixed ccusage (>20.0.20) ships.
33+
SRC="${CLAUDE_CONFIG_DIR:-$HOME/.claude}/projects"
34+
MIRROR="${TMPDIR}/cfg"
35+
mkdir -p "$MIRROR/projects"
36+
SINCE_ISO="$(date -d "$SINCE" +%F)" # SINCE is YYYYMMDD
37+
( cd "$SRC" && find . -name '*.jsonl' -newermt "$SINCE_ISO" -print0 \
38+
| while IFS= read -r -d '' f; do
39+
mkdir -p "$MIRROR/projects/$(dirname "$f")"
40+
jq -c 'del(.message.usage.iterations)' "$f" > "$MIRROR/projects/$f" 2>/dev/null \
41+
|| cp "$f" "$MIRROR/projects/$f"
42+
done )
43+
export CLAUDE_CONFIG_DIR="$MIRROR"
44+
2745
# (1) per-(date, model) totals — feeds daily_usage
2846
# ccusage v19 split agents under subcommands; top-level `daily` now aggregates
2947
# all agents and drops modelBreakdowns. `claude daily` preserves the legacy shape.

‎docs/claude-stats.html‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,8 @@ <h2>How <code>ingest-daily.sh</code> works</h2>
106106
</p>
107107

108108
<div class="flow">
109+
<div class="box"><div class="label">sanitize</div>JSONL mirror<br><span style="color:var(--fg-muted)">jq del(.usage.iterations)</span></div>
110+
<div class="arrow">→</div>
109111
<div class="box cyan"><div class="label">source</div>2× npx ccusage claude daily<br><span style="color:var(--fg-muted)">plain + --instances --breakdown</span></div>
110112
<div class="arrow">→</div>
111113
<div class="box"><div class="label">reshape</div>2× jq → CSV<br><span style="color:var(--fg-muted)">flatten modelBreakdowns</span></div>
@@ -115,6 +117,17 @@ <h2>How <code>ingest-daily.sh</code> works</h2>
115117
<div class="box good"><div class="label">targets</div>daily_usage<br>project_daily_usage</div>
116118
</div>
117119

120+
<p>
121+
<strong>Sanitize step (2026-09-16).</strong> ccusage 20.0.20 silently drops any
122+
assistant entry whose <code>usage.iterations[].model</code> is <code>null</code>
123+
(<a href="https://github.com/ccusage/ccusage/issues/1710">ccusage#1710</a>; Claude Code
124+
2.1.269 wrote that on 2026-09-12 and a day of Fable 5.1 cost vanished). The script now
125+
mirrors the window's JSONL into <code>$TMPDIR/cfg/projects/</code> with
126+
<code>jq 'del(.message.usage.iterations)'</code> and points ccusage at it via
127+
<code>CLAUDE_CONFIG_DIR</code>. ccusage sums top-level usage, so totals are identical on
128+
unaffected data and the shim stays harmless after the upstream fix ships.
129+
</p>
130+
118131
<details>
119132
<summary>The <code>jq</code> reshape (one row per day × model)</summary>
120133
<pre><code><span class="tok-c"># ccusage emits nested JSON per day with .modelBreakdowns[]</span>

0 commit comments

Comments
 (0)