fix(codex): dedupe archived session files - #1176

@coderabbitai @cubic-dev-ai please review this PR.

This PR was auto-closed. Only contributors approved with lgtm can open PRs. Open an issue first.

Maintainers review auto-closed issues and reopen worthwhile ones. Issues that do not meet the quality bar in CONTRIBUTING.md may not be reopened or receive a reply.

If a maintainer replies lgtmi, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open.

See CONTRIBUTING.md.

sequenceDiagram
  participant codex_usage_sources as codex_usage_sources()
  participant collect_groups as collect_deduped_codex_usage_files
  participant loader as load_codex_events_from_sources
  participant aggregator as load_groups_from_sources
  participant fs as Filesystem

  codex_usage_sources->>fs: discover sessions/ and archived_sessions/ under homes
  codex_usage_sources->>collect_groups: build CodexUsageSource list
  collect_groups->>fs: collect and sort files per source, compute dedupe keys
  collect_groups->>loader: emit per-source deduped file groups
  loader->>fs: read session files (serial/parallel)
  loader->>loader: append group events and dedupe globally
  loader->>aggregator: provide per-source aggregated groups (shared dedupe state)
  aggregator->>aggregator: merge groups using shared dedupe shards
  aggregator->>caller: return deduplicated aggregated groups

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

@coderabbitai @cubic-dev-ai please review this PR.

@yashau I have started the AI code review. It will take a few minutes to complete.

@yashau Sure! I'll kick off a full review of this PR right away.

Opened the requested contribution issue first: #1177. This closed PR can serve as the ready implementation reference if maintainers approve/reopen.

@yashau looks reasonable. thank you for this pr.
i'll take a look in a couple of days!

@pullfrog review it. if we need to fix it just fix it

Open in StackBlitz

npx https://pkg.pr.new/ccusage@1176

npx https://pkg.pr.new/@ccusage/ccusage-darwin-arm64@1176

npx https://pkg.pr.new/@ccusage/ccusage-darwin-x64@1176

npx https://pkg.pr.new/@ccusage/ccusage-linux-arm64@1176

npx https://pkg.pr.new/@ccusage/ccusage-linux-x64@1176

npx https://pkg.pr.new/@ccusage/ccusage-win32-arm64@1176

npx https://pkg.pr.new/@ccusage/ccusage-win32-x64@1176

commit: 3327310

@pullfrog check cubic's response

Verified cubic's PR summary against the actual code diff (5 files, commit d29832d). All four claims are accurate:

• sessions/ + archived_sessions/ discovery — codex_usage_paths_from_homes in paths.rs now checks for both subdirectories and pushes whichever exist.
• Deduplication with sessions/ priority — collect_deduped_codex_usage_files deduplicates by relative JSONL path. Since sessions/ is pushed first by codex_usage_paths_from_homes, active copies always win in the dedup.
• Direct JSONL dirs unchanged — When neither sessions/ nor archived_sessions/ exists as a subdirectory, the path falls through to the original direct JSONL path handling.
• Shared file discovery — Both loader.rs (load_codex_events_from_directories) and aggregate.rs (load_groups_from_directories) use the same collect_deduped_codex_usage_files / collect_codex_usage_files helpers.

Notably, cubic's summary is more accurate than CodeRabbit's, which incorrectly claims "CODEX_HOME accepts multiple comma-separated roots" as a new feature — that was pre-existing. Cubic restricts itself to only what this PR actually introduces.

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using DeepSeek Pro (free via Pullfrog for OSS) ｜ 𝕏}

@pullfrog rebase main

Rebase of codex/include-archived-sessions onto main succeeded locally — no conflicts, all prepush hooks passed (clippy, oxlint, treefmt, gitleaks, cargo test, vitest). However, the push was rejected because this is a fork PR (yashau/ccusage) and the GitHub App pushing to the fork doesn't have workflows permission to modify workflow files (the merge brought in changes to .github/workflows/approve-contributor.yaml from main). The branch cannot be updated remotely.

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using DeepSeek Pro (free via Pullfrog for OSS) ｜ 𝕏}

@coderabbitai @cubic-dev-ai please review the rebased update at 3327310.\n\nThis keeps #1176 scoped to the remaining value after #1230: file-level dedupe between sessions/ and archived_sessions/ within the same CODEX_HOME. It also addresses the prior cubic findings by scoping the dedupe key per CODEX_HOME and using PathBuf relative keys instead of lossy string conversion.

@coderabbitai @cubic-dev-ai please review the rebased update at 3327310.\n\nThis keeps #1176 scoped to the remaining value after #1230: file-level dedupe between sessions/ and archived_sessions/ within the same CODEX_HOME. It also addresses the prior cubic findings by scoping the dedupe key per CODEX_HOME and using PathBuf relative keys instead of lossy string conversion.

@ryoppippi I have started the AI code review. It will take a few minutes to complete.

#!/bin/bash
# Get PR details and current diff
gh pr view 1176 --repo ryoppippi/ccusage --json title,state,headRefName,baseRefName,commits,body

echo "---COMMIT INFO---"
gh api repos/ryoppippi/ccusage/commits/3327310 --jq '{sha: .sha, message: .commit.message, files: [.files[].filename]}'

echo "---PR DIFF---"
gh pr diff 1176 --repo ryoppippi/ccusage

@ryoppippi The rebased implementation at 3327310 looks solid. Here's what I checked:

✅ Key design points verified

CodexUsageSource + dedupe_scope scoping
Dedupe is keyed by (dedupe_scope: PathBuf, relative_path: PathBuf). Since dedupe_scope is set to the parent CODEX_HOME per entry, two different roots (e.g. ~/.codex and ~/.codex-work) with the same relative session filename stay fully independent — exactly the correct behavior. The new keeps_same_relative_session_file_across_different_homes test covers this explicitly.

PathBuf-based key, no lossy conversion
codex_usage_file_key uses strip_prefix + to_path_buf() to produce the relative key — no to_string_lossy(). The #[cfg(unix)] test deduplicates_non_utf8_relative_session_paths_without_lossy_strings validates this for non-UTF-8 filenames. ✅

collect_codex_usage_files sort
.sort() on Vec<PathBuf> is lexicographic component-by-component comparison, which is functionally equivalent to the old sort_by_cached_key(|p| p.to_string_lossy().into_owned()) for normal paths, and correct for non-UTF-8 paths too. ✅

Single-source fast path
Both load_groups and load_codex_events_from_sources skip the multi-directory deduplication path when there's exactly one source — consistent with pre-existing behavior and safe since intra-directory dedupe is handled separately. ✅

Empty file groups
When archived_sessions/ holds only files that duplicate sessions/, its group ends up with an empty files vec. Both aggregate_files_with_dedupe and read_codex_session_files_parallel handle this gracefully (no-op iteration). Not a problem, though you could optionally filter them:

// optional cleanup — purely cosmetic
groups.retain(|g| !g.files.is_empty());

Test coverage

• paths.rs: same-home dedupe, cross-home independence, non-UTF8 key
• loader.rs: active copy wins over archived duplicate + archived-only file preserved
• aggregate.rs: mirrors loader test at the group aggregation level

Both single_thread = true and false paths are exercised in the loop in both tests. ✅

Minor observation

codex_usage_paths_from_homes is now #[cfg(test)] which is the right call since it's only needed by the existing path tests. No issues there.

Overall the implementation is clean, correctly scoped, and the prior findings (string-lossy keys, cross-home leakage) are properly addressed. LGTM. 🐇

Learnt from: ryoppippi
Repo: ryoppippi/ccusage PR: 0
File: :0-0
Timestamp: 2026-05-22T17:26:33.070Z
Learning: In `ryoppippi/ccusage`, `load_groups_from_directory` in `rust/crates/ccusage/src/adapter/codex.rs` no longer has a worker-local parallel aggregation path. As of commit 695a46e, the parallel branch always uses `create_dedupe_shards()` + `aggregate_files_parallel` (global dedupe in one pass). The previous `aggregate_files_parallel_local` function (with its cross-chunk reread fallback) was removed entirely.

Learnt from: ryoppippi
Repo: ryoppippi/ccusage PR: 0
File: :0-0
Timestamp: 2026-05-22T14:31:40.598Z
Learning: In `ryoppippi/ccusage`, the worker-local aggregation fallback in `aggregate_files_parallel_local` (rust/crates/ccusage/src/adapter/codex.rs) is intentionally left without observability instrumentation. The local-dedupe fast path is optimized for the common single-`CODEX_HOME` case where cross-chunk duplicates are rare; the fallback re-runs the full global-dedupe pass to preserve correctness without adding per-event retained state to the hot path. This is a deliberate design trade-off by the maintainer.

Learnt from: ryoppippi
Repo: ryoppippi/ccusage PR: 0
File: :0-0
Timestamp: 2026-05-22T15:24:46.102Z
Learning: In `ryoppippi/ccusage`, the `load_groups` function in `rust/crates/ccusage/src/adapter/codex.rs` has a single-CODEX_HOME fast path guarded by `paths.len() == 1 && !wants_json(shared)`. The worker-local aggregation fast path is intentionally restricted to human-table output only; JSON/jq output always uses the global-dedupe aggregation path to avoid cross-chunk fallback cost after all chunks are processed.

Learnt from: ryoppippi
Repo: ryoppippi/ccusage PR: 0
File: :0-0
Timestamp: 2026-05-22T16:02:17.988Z
Learning: In `ryoppippi/ccusage`, `visit_codex_session_file` in `rust/crates/ccusage/src/codex_loader.rs` uses bounded-memory streaming via `BufReader::with_capacity(128 * 1024, file)` + `read_until(b'\n', &mut line)` with `line.clear()` each iteration (restored in commit ebf035f). The `fs::read` / `byte_lines` (full-file-in-memory) approach was deliberately reverted to avoid memory spikes on large JSONL files.

Learnt from: ryoppippi
Repo: ryoppippi/ccusage PR: 0
File: :0-0
Timestamp: 2026-05-22T14:48:37.050Z
Learning: In `ryoppippi/ccusage`, the `codex_line_usage_kind` classifier in `rust/crates/ccusage/src/codex_loader.rs` uses compact JSON key-value finders (`"type":"turn_context"`, `"type":"event_msg"`, `"type":"token_count"`) rather than bare substring finders. A line is classified as a Codex session row only when `"type":"turn_context"` is present, or when both `"type":"event_msg"` AND `"type":"token_count"` appear together. This was tightened in commit 9b83895 to prevent headless rows with `token_count` in arbitrary content fields from being misrouted to the session parser.

Learnt from: ryoppippi
Repo: ryoppippi/ccusage PR: 0
File: :0-0
Timestamp: 2026-05-22T14:48:37.050Z
Learning: In `ryoppippi/ccusage`, session timestamps in `rust/crates/ccusage/src/codex_loader.rs` use a dedicated `codex_session_timestamp` helper (introduced in commit 9b83895) that fast-paths string timestamps (returned trimmed as-is, no normalization overhead) and only delegates numeric timestamps to `normalize_codex_timestamp`. This was added to avoid the normalization cost on the common string-timestamp case in the hot session parsing path.

Historical audit: this pull request was auto-closed by the legacy contributor gate. That closure did not assess technical importance.

Audit result: resolved. A later merged change or the current main implementation covers this request. This PR is kept for history and does not need to be revived.