Skip to content

Commit 44a3aa8

Browse files
ryoppippicursoragentHenryTeo27
authored
fix(codex): read Zstandard-compressed rollouts (#1890)
* refactor(adapter): collect usage files with a path predicate Add collect_files_matching so an adapter can gather files that do not share one extension, such as plain and compressed rollouts, in a single directory walk. collect_files_with_extension now delegates to it. Co-authored-by: Henry Teo <[email protected]> * fix(codex): read Zstandard-compressed rollouts Since the 2026-10-14 Codex update, cold rollouts under sessions/ and archived_sessions/ are migrated to .jsonl.zst. The adapter only discovered .jsonl, so compressed sessions were skipped and totals such as codex monthly lost a large share of their tokens. Discover both encodings in one walk and stream .jsonl.zst through the zstd decoder into the existing parser, replay metadata probe, and rewritten-burst detection. Session IDs and the cross-directory dedupe key use the plain .jsonl path, so IDs stay stable and a rollout with both a plain and a compressed copy is counted once. The plain copy wins, matching Codex, which restores it before deleting the .zst when a compressed session is resumed. If a file switches encoding between discovery and open, the other encoding is read instead. Fixes #1888 Co-authored-by: Henry Teo <[email protected]> * docs(codex): document compressed rollout support Note that Codex session logs are read as plain .jsonl or .jsonl.zst, how duplicate encodings are resolved, and that session lookup accepts the .jsonl.zst filename. Co-authored-by: Henry Teo <[email protected]> * fix(codex): date headless usage from the rollout actually opened When a discovered rollout is compressed or restored before it is read, the reader opens the other encoding, but the fallback timestamp for timestamp-less headless usage still came from the vanished path and fell back to 1970. Return the opened path from open_rollout_reader and take the modification time from it. Reported by cubic on #1890. Co-authored-by: Henry Teo <[email protected]> * docs(codex): clarify rollout precedence and data paths Scope plain-over-compressed precedence to one directory and state that sessions/ beats archived_sessions/ regardless of encoding. List every recursive rollout path, and attribute ordered parallel reads to loader.rs rather than ccusage-adapter-common. Reported by cubic on #1890. Co-authored-by: Henry Teo <[email protected]> --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Henry Teo <[email protected]>
1 parent fc21e2c commit 44a3aa8

17 files changed

Lines changed: 713 additions & 28 deletions

File tree

‎docs/guide/codex/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Most users can start with unified reports such as `ccusage daily`. Add the `code
2323

2424
## Data Source
2525

26-
The CLI reads Codex session JSONL files located under `CODEX_HOME` (defaults to `~/.codex`). `CODEX_HOME` can be one directory or a comma-separated list of directories. For each entry, ccusage discovers `sessions/` and `archived_sessions/` independently, so an entry with only `archived_sessions/` still contributes archived Codex logs. When neither directory exists, the entry is read directly as a JSONL directory, which lets saved `codex exec --json` output live beside normal Codex homes. If the same relative JSONL path exists in both `sessions/` and `archived_sessions/` for one Codex home, the active `sessions/` copy wins so archived copies are not double counted.
26+
The CLI reads Codex session JSONL files, both plain `.jsonl` and Zstandard-compressed `.jsonl.zst`, located under `CODEX_HOME` (defaults to `~/.codex`). `CODEX_HOME` can be one directory or a comma-separated list of directories. For each entry, ccusage discovers `sessions/` and `archived_sessions/` independently, so an entry with only `archived_sessions/` still contributes archived Codex logs. When neither directory exists, the entry is read directly as a JSONL directory, which lets saved `codex exec --json` output live beside normal Codex homes. Compression does not change a session ID. Within one directory, when a plain and a compressed copy of the same rollout both exist, the plain copy is read and the compressed one is skipped, matching Codex. If the same relative rollout path exists in both `sessions/` and `archived_sessions/` for one Codex home, the active `sessions/` copy wins regardless of encoding, so archived copies are not double counted.
2727

2828
```bash
2929
CODEX_HOME="$HOME/.codex,$HOME/.codex-work,$HOME/codex-exec-logs" ccusage codex daily

‎docs/guide/session-reports.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ For Claude Code, session IDs are the actual filenames (without `.jsonl` extensio
5454

5555
You can find Claude session IDs by running `ccusage claude session` and looking for the files in your Claude data directory.
5656

57-
For Codex, use the `sessionId` from `ccusage codex session --json`. The lookup accepts that full ID, its JSONL filename with or without the extension, the trailing UUID, or a `codex://threads/<uuid>` URI. A shortened UUID is not accepted, and a UUID that matches more than one session is reported as ambiguous.
57+
For Codex, use the `sessionId` from `ccusage codex session --json`. The lookup accepts that full ID, its JSONL filename with or without the `.jsonl` or `.jsonl.zst` extension, the trailing UUID, or a `codex://threads/<uuid>` URI. A shortened UUID is not accepted, and a UUID that matches more than one session is reported as ambiguous.
5858

5959
## Example Output
6060

‎rust/Cargo.lock‎

Lines changed: 30 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎rust/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ terminal_size = "0.4.4"
6767
toml = { version = "1.1.8", default-features = false, features = ["parse", "serde", "std"] }
6868
unicode-width = "0.2.2"
6969
ureq = { version = "3.4.2", default-features = false, features = ["rustls", "gzip"] }
70+
zstd = { version = "0.14.2", default-features = false }
7071

7172
[profile.release]
7273
codegen-units = 1

‎rust/adapters/codex/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ rustc-hash.workspace = true
1414
serde.workspace = true
1515
serde_json.workspace = true
1616
toml.workspace = true
17+
zstd.workspace = true
1718

1819
[dev-dependencies]
1920
ccusage-test-support.workspace = true

‎rust/adapters/codex/README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ into the usage entries the reports render.
99
- `loader.rs` — reading the source, dedupe, and date filtering.
1010
- `parser.rs` — raw record parsing, token mapping, and model naming.
1111
- `paths.rs` — environment variables, default directories, and file discovery.
12+
- `reader.rs` — opening plain and Zstandard-compressed rollouts as one line stream.
1213
- `replay.rs` — detecting history that a forked or subagent session replays.
1314
- `report.rs` — the JSON and table shapes where they differ from the shared ones.
1415
- `speed.rs` — resolving the service tier that priced each request.
@@ -20,11 +21,15 @@ Anything that is not specific to this source belongs in `ccusage-core` or
2021
## Data source
2122

2223
- `~/.codex/sessions/**/*.jsonl`
24+
- `~/.codex/sessions/**/*.jsonl.zst`
25+
- `~/.codex/archived_sessions/**/*.jsonl`
26+
- `~/.codex/archived_sessions/**/*.jsonl.zst`
2327

2428
Record shapes, token mapping, and cost rules are documented in [`src/README.md`](src/README.md).
2529

26-
Reads plain files through `ccusage-adapter-common`, which handles walking, size-balanced
27-
chunking, and ordered parallel reads.
30+
Uses `ccusage-adapter-common` for walking and size-balanced chunking. `loader.rs` performs
31+
the ordered parallel reads, and `reader.rs` streams `.jsonl.zst` rollouts through `zstd`
32+
without unpacking them to disk.
2833

2934
## Public surface
3035

@@ -60,6 +65,7 @@ chunking, and ordered parallel reads.
6065
- `serde`
6166
- `serde_json`
6267
- `toml`
68+
- `zstd`
6369

6470
## Build layer
6571

‎rust/adapters/codex/src/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,10 @@ ${CODEX_HOME:-~/.codex}/sessions/
77
${CODEX_HOME:-~/.codex}/archived_sessions/
88
```
99

10-
When both directories contain the same relative JSONL path for one Codex home,
11-
the active `sessions/` copy wins.
10+
Rollouts are plain `.jsonl` or Zstandard-compressed `.jsonl.zst`. Compression does not
11+
change the session ID. When both encodings of a rollout exist in one directory, the
12+
plain copy wins, as it does in Codex. When both directories contain the same relative
13+
rollout path for one Codex home, the active `sessions/` copy wins regardless of encoding.
1214

1315
Relevant JSONL event:
1416

0 commit comments

Comments
 (0)