Skip to content

Commit 431a308

Browse files
committed
fix(codex): read compressed Codex rollouts
Codex now stores session and archive rollouts as .jsonl.zst, but the reader only loaded .jsonl files. On one frozen real-data snapshot, this omitted 11,977,116,872 tokens. Stream Zstandard rollouts through the existing parser and replay path. Preserve logical session IDs and prefer active sessions when encodings overlap.
1 parent b6c6e46 commit 431a308

12 files changed

Lines changed: 451 additions & 25 deletions

File tree

‎docs/guide/codex/index.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,11 @@ Most users can start with unified reports such as `ccusage daily`. Add the `code
2323

2424
## Data Source
2525

26-
The CLI reads Codex session JSONL files located under `CODEX_HOME` (defaults to `~/.codex`). `CODEX_HOME` can be one directory or a comma-separated list of directories. For each entry, ccusage discovers `sessions/` and `archived_sessions/` independently, so an entry with only `archived_sessions/` still contributes archived Codex logs. When neither directory exists, the entry is read directly as a JSONL directory, which lets saved `codex exec --json` output live beside normal Codex homes. If the same relative JSONL path exists in both `sessions/` and `archived_sessions/` for one Codex home, the active `sessions/` copy wins so archived copies are not double counted.
26+
The CLI reads plain `.jsonl` and Zstandard-compressed `.jsonl.zst` session logs under `CODEX_HOME` (defaults to `~/.codex`). Compressed logs are read directly; no manual unpacking is needed.
27+
28+
`CODEX_HOME` can be one directory or a comma-separated list of directories. For each entry, ccusage discovers `sessions/` and `archived_sessions/` independently, so an entry with only `archived_sessions/` still contributes archived Codex logs. When neither directory exists, the entry is read directly as a JSONL directory, which lets saved `codex exec --json` output live beside normal Codex homes.
29+
30+
Compression does not change the session ID. If both encodings exist in one directory, the plain copy wins. If the same relative path exists in both `sessions/` and `archived_sessions/` for one Codex home, the active `sessions/` copy wins regardless of encoding.
2731

2832
```bash
2933
CODEX_HOME="$HOME/.codex,$HOME/.codex-work,$HOME/codex-exec-logs" ccusage codex daily

‎rust/Cargo.lock‎

Lines changed: 29 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎rust/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ terminal_size = "0.4.4"
6767
toml = { version = "1.1.8", default-features = false, features = ["parse", "serde", "std"] }
6868
unicode-width = "0.2.2"
6969
ureq = { version = "3.4.2", default-features = false, features = ["rustls", "gzip"] }
70+
zstd = { version = "0.14.2", default-features = false }
7071

7172
[profile.release]
7273
codegen-units = 1

‎rust/adapters/codex/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ rustc-hash.workspace = true
1414
serde.workspace = true
1515
serde_json.workspace = true
1616
toml.workspace = true
17+
zstd.workspace = true
1718

1819
[dev-dependencies]
1920
ccusage-test-support.workspace = true

‎rust/adapters/codex/README.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
# ccusage-adapter-codex
22

3-
The Codex adapter: it turns Codex session JSONL, including forked and replayed sessions
4-
into the usage entries the reports render.
3+
The Codex adapter turns plain and Zstandard-compressed session JSONL, including
4+
forked and replayed sessions, into usage entries for reports.
55

66
## Owns
77

88
- `aggregate.rs` — grouping events into report periods, including cross-session dedupe.
99
- `loader.rs` — reading the source, dedupe, and date filtering.
1010
- `parser.rs` — raw record parsing, token mapping, and model naming.
1111
- `paths.rs` — environment variables, default directories, and file discovery.
12+
- `reader.rs` — streaming plain and compressed rollouts, with logical path normalization.
1213
- `replay.rs` — detecting history that a forked or subagent session replays.
1314
- `report.rs` — the JSON and table shapes where they differ from the shared ones.
1415
- `speed.rs` — resolving the service tier that priced each request.
@@ -20,11 +21,14 @@ Anything that is not specific to this source belongs in `ccusage-core` or
2021
## Data source
2122

2223
- `~/.codex/sessions/**/*.jsonl`
24+
- `~/.codex/sessions/**/*.jsonl.zst`
25+
- `~/.codex/archived_sessions/**/*.jsonl`
26+
- `~/.codex/archived_sessions/**/*.jsonl.zst`
2327

2428
Record shapes, token mapping, and cost rules are documented in [`src/README.md`](src/README.md).
2529

26-
Reads plain files through `ccusage-adapter-common`, which handles walking, size-balanced
27-
chunking, and ordered parallel reads.
30+
Uses `ccusage-adapter-common` for walking and size-balanced chunking. The Codex
31+
reader streams compressed files directly without unpacking them to disk.
2832

2933
## Public surface
3034

@@ -60,6 +64,7 @@ chunking, and ordered parallel reads.
6064
- `serde`
6165
- `serde_json`
6266
- `toml`
67+
- `zstd`
6368

6469
## Build layer
6570

‎rust/adapters/codex/src/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,10 @@ ${CODEX_HOME:-~/.codex}/sessions/
77
${CODEX_HOME:-~/.codex}/archived_sessions/
88
```
99

10-
When both directories contain the same relative JSONL path for one Codex home,
11-
the active `sessions/` copy wins.
10+
Both `.jsonl` and Zstandard-compressed `.jsonl.zst` files are supported. Compression
11+
does not change the session ID or token accounting. The plain copy wins when both
12+
encodings exist in one directory. For matching relative paths in both directories,
13+
the active `sessions/` copy wins regardless of encoding.
1214

1315
Relevant JSONL event:
1416

0 commit comments

Comments
 (0)