Skip to content

refactor(ci): unify all Nix caching on one job+matrix-scoped action - #1296

Merged
ryoppippi merged 11 commits into
mainfrom
refactor/macos-cache-scope-by-matrix
Jun 12, 2026
Merged

ryoppippi merged 11 commits into
mainfrom
refactor/macos-cache-scope-by-matrix

Conversation

@ryoppippi

@ryoppippi ryoppippi commented Jun 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Unifies every Nix cache setup onto a single cross-platform action and removes the Blacksmith sticky disk entirely. Cache keys are now scoped by github.job + matrix key (matrix.name) instead of runner.os/runner.arch.

Why

  • github.job returns the workflow job id (e.g. build-native-packages), which is identical for every matrix leg, so it can't separate them alone.
  • The arm64 and x64 macOS builds share one arm64 runner, so runner.os/runner.arch are identical for both — they were clobbering one cache (the ci(cache): scope macOS Nix cache key by target arch #1294 bug). The Linux legs only stayed separate because they happen to run on different-arch runners.
  • nix-<github.job>-<matrix.name>-<hash> is unambiguous regardless of runner arch and generalises to any future matrix dimension.
  • The sticky disk was keyed per github.job, never shared a warm store across the jobs that mattered, and its weekly GC was a no-op. cache-nix-action's gc-max-store-size keeps every entry bounded without it.

What changed

  • New setup-nix-cache (renamed from setup-linux-nix-action-cache): installs Nix + cache-nix-action keyed by nix-${{ github.job }}-${{ inputs.cache-scope }}-<hash>, sets both gc-max-store-size-linux and -macos so one action serves both OSes, and fixes single-user store ownership.
  • Deleted setup-macos-nix-cache (folded in), setup-nix, and setup-linux-blacksmith-sticky-disk.
  • Every former sticky-disk consumer (npm-publish, perf comments, release publish, update-pricing, pullfrog) and both native-build actions now use setup-nix-cache, threading cache-scope from the matrix name.
  • pin-nix-deps rationale updated: the deps pin now exists to survive cache-nix-action's GC-root trim, not the sticky-disk trim.

Cache keys change once, so caches re-warm a single time.

Summary by CodeRabbit

  • Chores
    • Cache scoping unified to isolate Nix caches per matrix entry, reducing cross-build cache collisions.
    • Introduced a consolidated Nix cache setup with improved restore/save semantics and bounded GC to make caching more reliable.
    • Removed legacy Nix cache helpers and simplified native-package build steps; CI workflows updated to use the new cache setup.

Replaces the arch-derived key suffix with the matrix entry id. github.job
is identical across every matrix leg, so it cannot distinguish them on its
own; the arm64 and x64 macOS builds additionally share one arm64 runner
(same runner.os/runner.arch). Keying by `nix-<github.job>-<matrix.name>`
makes each leg's cache unambiguous without leaning on runner.arch, which
only happened to separate the Linux legs because they run on different-arch
runners.

The cache-scope is threaded from the matrix `name` through
build-macos-nix-native-package and setup-nix into setup-macos-nix-cache.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 12, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
ccusage-guide 9d3016c Commit Preview URL

Branch Preview URL
Jun 12 2026, 01:17 AM

@coderabbitai

coderabbitai Bot commented Jun 12, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a new composite action setup-nix-cache with a cache-scope input, replaces prior Nix setup/cache steps across actions and workflows with it, and threads required cache-scope through native build actions and setup-nix to namespace Nix caches per matrix entry.

Changes

Nix cache scoping mechanism

Layer / File(s) Summary
setup-nix-cache implementation
.github/actions/setup-nix-cache/action.yaml
New composite action installs Nix, restores/saves the Nix store via nix-community/cache-nix-action, and builds cache keys from github.workflow + github.job + optional inputs.cache-scope plus lock/build hashes; includes GC/purge settings and ownership-fix step.
setup-nix action input and macOS wiring
.github/actions/setup-nix/action.yaml
Replaces cache-key-suffix with inputs.cache-scope and forwards it into the macOS cache setup step.
Build actions accept cache-scope
.github/actions/build-macos-nix-native-package/action.yaml, .github/actions/build-linux-native-package/action.yaml
Adds required cache-scope inputs to native build actions and wires them into the setup-nix-cache step (replacing prior arch/runner-derived scoping).
Action description update
.github/actions/pin-nix-deps/action.yaml
Updates action description to reference cache-nix-action GC trimming behavior and clarify when unrooted intermediates are dropped.
Workflows: switch to setup-nix-cache and pass cache-scope
.github/workflows/ci.yaml, .github/workflows/release.yaml, .github/workflows/pullfrog.yml, .github/workflows/update-pricing.yaml
Workflows replace previous setup-nix/platform cache steps with ./.github/actions/setup-nix-cache and pass cache-scope: ${{ matrix.name }} or matrix-driven values for jobs that run per-matrix (including ccusage-perf-comment changes).

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant BuildAction as build-*-native-package
  participant SetupNix as setup-nix
  participant SetupNixCache as setup-nix-cache
  Workflow->>BuildAction: invoke build action (may pass cache-scope: matrix.name)
  BuildAction->>SetupNix: forward inputs.cache-scope
  SetupNix->>SetupNixCache: forward inputs.cache-scope
  SetupNixCache->>SetupNixCache: compute primary-key / restore-prefixes / purge-prefixes (using cache-scope and hashed inputs)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • ccusage/ccusage#1285: Related changes to Linux native build caching and setup-linux-nix-action-cache.
  • ccusage/ccusage#1294: Also modifies macOS Nix cache key scoping and related action wiring.
  • ccusage/ccusage#1257: Overlaps on the removed setup-linux-blacksmith-sticky-disk action and caching logic.

Suggested reviewers

  • pullfrog

Poem

A rabbit hops through actions bright,
It threads a cache-scope with nimble sight,
Each matrix leg now keeps its name,
Caches tidy, never the same,
Builds hum along — hooray, what a night! 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and concisely describes the main change: unifying Nix caching across CI workflows into a single, matrix-scoped action.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/macos-cache-scope-by-matrix

Comment @coderabbitai help to get the list of available commands and usage tips.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

The release workflow will fail on merge because .github/workflows/release.yaml calls build-macos-nix-native-package without the now-required cache-scope input.

Reviewed changes — This PR refactors the macOS Nix cache keying strategy from runner.os-runner.arch-key-suffix to github.job-cache-scope, and renames key-suffix/cache-key-suffix to cache-scope across the action chain, so each matrix leg's cache is properly isolated on the shared arm64 macOS runner.

  • Rename key-suffix → cache-scope across setup-macos-nix-cache, setup-nix, and build-macos-nix-native-package
  • Switch cache key prefix from runner.os-runner.arch to github.job-cache-scope in setup-macos-nix-cache
  • Update ci.yaml caller to pass cache-scope: ${{ matrix.name }}

🚨 Missing cache-scope in release.yaml

The release workflow at .github/workflows/release.yaml:58 calls build-macos-nix-native-package for the darwin matrix entries but does not pass cache-scope. The action now declares this input as required: true, so GitHub Actions will fail at workflow validation — the release won't run.

The matrix in release.yaml already uses matrix.name (e.g. build-mac-arm64, build-mac-x64), so the fix is the same one-liner as ci.yaml:

      - uses: ./.github/actions/build-macos-nix-native-package
        if: matrix.platform == 'darwin'
        with:
          arch: ${{ matrix.arch }}
          binary: ${{ matrix.binary }}
          cache-scope: ${{ matrix.name }}
Technical details
# Missing `cache-scope` in `release.yaml`

## Affected sites

- `.github/workflows/release.yaml:60-62` — caller of `build-macos-nix-native-package` that omits now-mandatory `cache-scope`

## Required outcome

The `release.yaml` build-native-packages job must pass `cache-scope: ${{ matrix.name }}` to `build-macos-nix-native-package`, matching the `ci.yaml` call site.

## Suggested approach

Append the line `cache-scope: ${{ matrix.name }}` to the `with:` block at `release.yaml:60`.

Pullfrog  | Fix it ➔ | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

actionlint flagged release.yaml for not supplying the new required
cache-scope input on the macOS native-package action. Mirror ci.yaml
and pass matrix.name.

Co-authored-by: Codesmith <[email protected]>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/actions/setup-nix/action.yaml (1)

4-8: Add macOS runtime validation for cache-scope in setup-nix to prevent empty-scope cache keys.

setup-nix forwards inputs.cache-scope (defaulting to '') to setup-macos-nix-cache, whose cache keys include nix-${{ github.job }}-${{ inputs.cache-scope }}-..., so an omitted value on macOS would produce ambiguous prefixes. Current workflows avoid this by passing cache-scope: ${{ matrix.name }} via build-macos-nix-native-package, but future direct macOS calls could regress.

🔧 Suggested fix: Validate `cache-scope` on macOS
+   - name: Validate cache-scope on macOS
+     if: runner.os == 'macOS'
+     shell: bash
+     run: |
+       if [ -z "${{ inputs.cache-scope }}" ]; then
+         echo "::error::cache-scope input is required when running on macOS"
+         exit 1
+       fi
+
    - uses: ./.github/actions/setup-macos-nix-cache
      if: runner.os == 'macOS'
      with:
        cache-scope: ${{ inputs.cache-scope }}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/actions/setup-nix/action.yaml around lines 4 - 8, Add a runtime
validation that fails early on macOS when inputs.cache-scope is empty: in the
action's runtime logic (the code path that forwards inputs.cache-scope to
setup-macos-nix-cache) check runner.os / process.env.RUNNER_OS (or the
equivalent runtime API) and if it is macOS and inputs.cache-scope is ''/falsy,
throw/exit with a clear error message instructing callers to provide cache-scope
(e.g., "cache-scope is required on macOS to avoid ambiguous cache keys"); ensure
this check runs before calling setup-macos-nix-cache so we never construct keys
like nix-${{ github.job }}- -... with an empty scope.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/actions/setup-nix/action.yaml:
- Around line 4-8: Add a runtime validation that fails early on macOS when
inputs.cache-scope is empty: in the action's runtime logic (the code path that
forwards inputs.cache-scope to setup-macos-nix-cache) check runner.os /
process.env.RUNNER_OS (or the equivalent runtime API) and if it is macOS and
inputs.cache-scope is ''/falsy, throw/exit with a clear error message
instructing callers to provide cache-scope (e.g., "cache-scope is required on
macOS to avoid ambiguous cache keys"); ensure this check runs before calling
setup-macos-nix-cache so we never construct keys like nix-${{ github.job }}-
-... with an empty scope.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: caa6b839-fc13-41bf-9c92-5a8899a06cb6

📥 Commits

Reviewing files that changed from the base of the PR and between 11c782f and 9a03c61.

📒 Files selected for processing (4)
  • .github/actions/build-macos-nix-native-package/action.yaml
  • .github/actions/setup-macos-nix-cache/action.yaml
  • .github/actions/setup-nix/action.yaml
  • .github/workflows/ci.yaml

@pkg-pr-new

pkg-pr-new Bot commented Jun 12, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

ccusage

npx https://pkg.pr.new/ccusage@1296

@ccusage/ccusage-darwin-arm64

npx https://pkg.pr.new/@ccusage/ccusage-darwin-arm64@1296

@ccusage/ccusage-darwin-x64

npx https://pkg.pr.new/@ccusage/ccusage-darwin-x64@1296

@ccusage/ccusage-linux-arm64

npx https://pkg.pr.new/@ccusage/ccusage-linux-arm64@1296

@ccusage/ccusage-linux-x64

npx https://pkg.pr.new/@ccusage/ccusage-linux-x64@1296

@ccusage/ccusage-win32-x64

npx https://pkg.pr.new/@ccusage/ccusage-win32-x64@1296

commit: 9d3016c

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/actions/build-macos-nix-native-package/action.yaml">

<violation number="1" location=".github/actions/build-macos-nix-native-package/action.yaml:15">
P1: New required `cache-scope` input breaks existing release workflow call that does not pass it.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Unique matrix-entry identifier (the matrix `name`) used to scope the
macOS Nix cache. arm64 and x64 share the same arm64 runner, so without a
per-leg scope the two builds clobber one cache.
required: true

@cubic-dev-ai cubic-dev-ai Bot Jun 12, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: New required cache-scope input breaks existing release workflow call that does not pass it.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/actions/build-macos-nix-native-package/action.yaml, line 15:

<comment>New required `cache-scope` input breaks existing release workflow call that does not pass it.</comment>

<file context>
@@ -7,14 +7,18 @@ inputs:
+      Unique matrix-entry identifier (the matrix `name`) used to scope the
+      macOS Nix cache. arm64 and x64 share the same arm64 runner, so without a
+      per-leg scope the two builds clobber one cache.
+    required: true
 runs:
   using: composite
</file context>
Fix with cubic

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 9a03c61373df
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 680.3ms 740.9ms 65.2ms 3
PR pkg.pr.new 9a03c61 801.1ms 782.7ms 58.2ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 9a03c61. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 865.3ms 849.4ms 1.02x 726.00 MiB 734.25 MiB 1.01x 1.16 GiB/s 1.19 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 183.0ms 180.5ms 1.01x 88.75 MiB 90.50 MiB 1.02x 5.50 GiB/s 5.58 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 985.4ms 1.02 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 813.1ms 1.24 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 163.1ms 6.17 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 125.2ms 8.04 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 42.7ms 47.2ms 0.91x 43.00 MiB 43.00 MiB 1.00x 0.04 MiB/s 0.03 MiB/s
claude session --offline --json 0.00 MiB 40.4ms 41.1ms 0.98x 43.25 MiB 43.00 MiB 0.99x 0.04 MiB/s 0.04 MiB/s
codex daily --offline --json 0.00 MiB 44.1ms 46.4ms 0.95x 43.25 MiB 43.00 MiB 0.99x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 46.7ms 53.6ms 0.87x 43.25 MiB 43.00 MiB 0.99x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 753.7ms 909.4ms 0.83x 732.25 MiB 726.75 MiB 0.99x 1.34 GiB/s 1.11 GiB/s
codex --offline --json 1.01 GiB 166.2ms 160.7ms 1.03x 92.00 MiB 90.50 MiB 0.98x 6.06 GiB/s 6.27 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — prior review flagged release.yaml would fail workflow validation at merge time because build-macos-nix-native-package now requires cache-scope but the release caller omitted it. Commit 7687f09 adds the missing cache-scope: ${{ matrix.name }} to the release.yaml call site, matching the ci.yaml pattern.

  • Add cache-scope to release.yaml — one-line addition at release.yaml:63 forwarding the matrix name as the cache scope

Pullfrog  | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 9a03c61373df
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 697.5ms 715.1ms 82.3ms 3
PR pkg.pr.new 9a03c61 1.094s 1.104s 96.4ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 9a03c61. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 963.7ms 863.6ms 1.12x 724.25 MiB 719.25 MiB 0.99x 1.04 GiB/s 1.17 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 290.9ms 299.2ms 0.97x 90.75 MiB 89.00 MiB 0.98x 3.46 GiB/s 3.37 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 983.0ms 1.02 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 965.3ms 1.04 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 257.1ms 3.92 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 246.6ms 4.08 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 76.5ms 10.8ms 7.07x 43.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s
claude session --offline --json 0.00 MiB 58.0ms 10.7ms 5.43x 43.25 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.14 MiB/s
codex daily --offline --json 0.00 MiB 72.1ms 12.6ms 5.71x 43.00 MiB 2.75 MiB 0.06x 0.01 MiB/s 0.07 MiB/s
codex session --offline --json 0.00 MiB 67.1ms 13.9ms 4.84x 43.00 MiB 2.75 MiB 0.06x 0.01 MiB/s 0.06 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 930.4ms 971.8ms 0.96x 738.25 MiB 721.25 MiB 0.98x 1.08 GiB/s 1.04 GiB/s
codex --offline --json 1.01 GiB 292.7ms 224.3ms 1.30x 91.00 MiB 89.50 MiB 0.98x 3.44 GiB/s 4.49 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@pullfrog

pullfrog Bot commented Jun 12, 2026 •

Copy link
Copy Markdown
Contributor

Diagnosis: The check suite failed on commit 9a03c61 — the treefmt-check step of nix flake check failed because actionlint detected that build-macos-nix-native-package (which requires the new cache-scope input) was called from release.yaml without passing cache-scope.

Outcome: No fix needed — the issue was already resolved by subsequent commits on the PR branch. Commit 7687f09 ("ci(release): pass cache-scope to build-macos-nix-native-package") added cache-scope: ${{ matrix.name }} to both build-linux-native-package and build-macos-nix-native-package in release.yaml. The check suite simply ran before these fixes were pushed. The ccusage-clippy and ccusage-rust-fmt failures in the same check suite are pre-existing (this PR touches no Rust code).

Task list (5/5 completed)
  • Checkout PR branch and fetch check suite logs
  • Diagnose the CI failure
  • Verify failure was introduced by this PR
  • Fix the issue and verify
  • Commit, push, and report progress

Pullfrog  | View workflow run | via Pullfrog | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

Collapses the per-platform cache setups into a single cross-platform
`setup-nix-cache` action and drops the Blacksmith sticky disk entirely.

- `setup-nix-cache` (renamed from setup-linux-nix-action-cache) now installs
  Nix, restores/saves with cache-nix-action keyed by
  `nix-<github.job>-<cache-scope>`, sets both gc-max-store-size-linux and
  -macos so one action serves both OSes, and fixes single-user store
  ownership. cache-scope is the matrix `name`; github.job alone cannot
  separate matrix legs and runner.arch collides for the two macOS legs that
  share one arm64 runner.
- Deletes setup-macos-nix-cache (folded in), setup-nix, and
  setup-linux-blacksmith-sticky-disk.
- Routes every former sticky-disk consumer (npm-publish, perf comments,
  release publish, update-pricing, pullfrog) and both native-build actions
  through setup-nix-cache, threading cache-scope from the matrix name.

The sticky disk was keyed per github.job, never shared a warm store across
the jobs that mattered, and its weekly GC was a no-op; cache-nix-action's
gc-max-store-size keeps every entry bounded without it.
Sticky disks are gone; the deps pin now exists to survive cache-nix-action's
gc-max-store-size trim (which keeps only GC roots), not the sticky-disk trim.
@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 7687f09f43a4
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 875.9ms 829.2ms 57.0ms 3
PR pkg.pr.new 7687f09 889.5ms 829.8ms 57.6ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 7687f09. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 863.4ms 893.0ms 0.97x 752.25 MiB 737.50 MiB 0.98x 1.17 GiB/s 1.13 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 174.9ms 177.0ms 0.99x 93.25 MiB 91.50 MiB 0.98x 5.76 GiB/s 5.69 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 893.8ms 1.13 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 868.2ms 1.16 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 162.7ms 6.19 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 125.6ms 8.01 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 39.3ms 4.6ms 8.55x 43.00 MiB 3.00 MiB 0.07x 0.04 MiB/s 0.34 MiB/s
claude session --offline --json 0.00 MiB 47.6ms 4.4ms 10.73x 43.00 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.35 MiB/s
codex daily --offline --json 0.00 MiB 39.3ms 4.2ms 9.34x 43.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.20 MiB/s
codex session --offline --json 0.00 MiB 38.7ms 4.2ms 9.13x 42.75 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.20 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 783.6ms 749.4ms 1.05x 739.50 MiB 742.50 MiB 1.00x 1.28 GiB/s 1.34 GiB/s
codex --offline --json 1.01 GiB 156.9ms 122.8ms 1.28x 89.75 MiB 93.75 MiB 1.04x 6.42 GiB/s 8.20 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@ryoppippi ryoppippi changed the title refactor(ci): scope macOS Nix cache by job + matrix key refactor(ci): unify all Nix caching on one job+matrix-scoped action Jun 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 7687f09f43a4
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 982.1ms 970.6ms 58.6ms 3
PR pkg.pr.new 7687f09 2.266s 884.3ms 56.3ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 7687f09. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 903.6ms 908.0ms 1.00x 739.25 MiB 738.75 MiB 1.00x 1.11 GiB/s 1.11 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 197.1ms 188.8ms 1.04x 90.50 MiB 87.75 MiB 0.97x 5.11 GiB/s 5.33 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 901.2ms 1.12 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 919.0ms 1.10 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 174.4ms 5.77 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 130.5ms 7.71 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 43.9ms 43.7ms 1.01x 43.00 MiB - - 0.04 MiB/s 0.04 MiB/s
claude session --offline --json 0.00 MiB 38.2ms 38.4ms 0.99x 43.00 MiB 43.00 MiB 1.00x 0.04 MiB/s 0.04 MiB/s
codex daily --offline --json 0.00 MiB 37.2ms 37.3ms 1.00x 43.00 MiB 43.00 MiB 1.00x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 37.3ms 37.2ms 1.01x 43.00 MiB 43.00 MiB 1.00x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 861.8ms 835.0ms 1.03x 733.00 MiB 727.75 MiB 0.99x 1.17 GiB/s 1.21 GiB/s
codex --offline --json 1.01 GiB 169.3ms 167.8ms 1.01x 90.00 MiB 89.50 MiB 0.99x 5.95 GiB/s 6.00 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — prior pullfrog review confirmed the cache-scope fix in release.yaml. Since then, the PR goes further: unifying all Nix caching into a single setup-nix-cache action, removing 4 legacy actions and the Blacksmith sticky disk, and threading cache-scope through the Linux native build path alongside macOS.

  • Unify Nix cache setup into setup-nix-cache — replaces setup-linux-nix-action-cache, setup-macos-nix-cache, setup-nix, and setup-linux-blacksmith-sticky-disk with a single cross-platform action keyed by github.job + cache-scope
  • Add cache-scope to Linux build path — build-linux-native-package now requires and forwards cache-scope, matching the macOS pattern
  • Remove Blacksmith sticky disk — every former consumer now uses setup-nix-cache whose gc-max-store-size bounds each cache entry
  • Update pin-nix-deps rationale — references cache-nix-action GC instead of the removed sticky disk

Pullfrog  | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 04ef555151a7
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 834.3ms 851.5ms 54.8ms 3
PR pkg.pr.new 04ef555 1.287s 805.1ms 53.8ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 04ef555. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 821.2ms 824.4ms 1.00x 738.00 MiB 734.75 MiB 1.00x 1.23 GiB/s 1.22 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 175.4ms 174.8ms 1.00x 88.75 MiB 86.75 MiB 0.98x 5.74 GiB/s 5.76 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 768.7ms 1.31 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 847.1ms 1.19 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 158.5ms 6.35 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 122.3ms 8.23 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 47.2ms 37.9ms 1.25x 44.25 MiB 44.00 MiB 0.99x 0.03 MiB/s 0.04 MiB/s
claude session --offline --json 0.00 MiB 39.3ms 36.7ms 1.07x 44.25 MiB 44.25 MiB 1.00x 0.04 MiB/s 0.04 MiB/s
codex daily --offline --json 0.00 MiB 36.5ms 37.1ms 0.99x 44.00 MiB 44.25 MiB 1.01x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 36.8ms 37.6ms 0.98x 44.00 MiB 44.00 MiB 1.00x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 807.8ms 760.3ms 1.06x 744.00 MiB 742.25 MiB 1.00x 1.25 GiB/s 1.32 GiB/s
codex --offline --json 1.01 GiB 155.0ms 159.5ms 0.97x 91.50 MiB 88.75 MiB 0.97x 6.49 GiB/s 6.31 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 04ef555151a7
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 942.8ms 954.2ms 69.2ms 3
PR pkg.pr.new 04ef555 1.031s 1.137s 67.4ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 04ef555. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 768.1ms 977.4ms 0.79x 738.75 MiB 744.50 MiB 1.01x 1.31 GiB/s 1.03 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 240.0ms 225.9ms 1.06x 93.00 MiB 91.50 MiB 0.98x 4.20 GiB/s 4.46 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 946.4ms 1.06 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 854.6ms 1.18 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 222.3ms 4.53 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 161.2ms 6.25 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 48.7ms 8.5ms 5.76x 44.00 MiB 3.00 MiB 0.07x 0.03 MiB/s 0.18 MiB/s
claude session --offline --json 0.00 MiB 50.8ms 12.1ms 4.20x 44.00 MiB 3.00 MiB 0.07x 0.03 MiB/s 0.13 MiB/s
codex daily --offline --json 0.00 MiB 45.2ms 7.8ms 5.77x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.11 MiB/s
codex session --offline --json 0.00 MiB 47.3ms 10.7ms 4.43x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.08 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 852.7ms 837.8ms 1.02x 742.75 MiB 727.00 MiB 0.98x 1.18 GiB/s 1.20 GiB/s
codex --offline --json 1.01 GiB 213.5ms 167.0ms 1.28x 95.00 MiB 92.25 MiB 0.97x 4.71 GiB/s 6.03 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/actions/setup-nix-cache/action.yaml (1)

30-31: Document/guard against cache restore/purge prefix collisions via cache-scope

Current workflow matrix name values (build-linux-arm64, build-linux-x64, build-mac-arm64, build-mac-x64, build-windows-arm64, build-windows-x64) don’t introduce any obvious substring/prefix collisions with the restore/purge prefix nix-${{ github.job }}-${{ inputs.cache-scope }}-.

Because restore-prefixes-first-match (and purge-prefixes) rely on that prefix, adding future scopes with prefix relationships could cause ambiguous restores. Document that cache-scope must not be a prefix/sub-prefix of another scope (or switch to a more unambiguous delimiter/encoding between scope and hash).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/actions/setup-nix-cache/action.yaml around lines 30 - 31, The
restore/purge prefix using restore-prefixes-first-match and inputs.cache-scope
can collide if cache-scope values are prefixes of one another; update
documentation and add a guard: require/validate that inputs.cache-scope values
are not a prefix of any other scope (or normalize/encode the scope with an
unambiguous delimiter like "::" or base64) and switch
restore-prefixes-first-match/purge-prefixes to use the encoded delimiter form;
specifically update the action input validation and README to enforce the rule
for inputs.cache-scope and update primary-key/restore-prefixes-first-match usage
to concatenate a clearly delimited scope (e.g., nix-${{ github.job
}}-<encoded-cache-scope>-${{ hashFiles(...) }}) so future scope names cannot
create prefix collisions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/actions/setup-nix-cache/action.yaml:
- Around line 30-31: The restore/purge prefix using restore-prefixes-first-match
and inputs.cache-scope can collide if cache-scope values are prefixes of one
another; update documentation and add a guard: require/validate that
inputs.cache-scope values are not a prefix of any other scope (or
normalize/encode the scope with an unambiguous delimiter like "::" or base64)
and switch restore-prefixes-first-match/purge-prefixes to use the encoded
delimiter form; specifically update the action input validation and README to
enforce the rule for inputs.cache-scope and update
primary-key/restore-prefixes-first-match usage to concatenate a clearly
delimited scope (e.g., nix-${{ github.job }}-<encoded-cache-scope>-${{
hashFiles(...) }}) so future scope names cannot create prefix collisions.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ef396dc7-5612-41df-98b6-76393beaafcc

📥 Commits

Reviewing files that changed from the base of the PR and between 7687f09 and af12a3c.

📒 Files selected for processing (12)
  • .github/actions/build-linux-native-package/action.yaml
  • .github/actions/build-macos-nix-native-package/action.yaml
  • .github/actions/pin-nix-deps/action.yaml
  • .github/actions/setup-linux-blacksmith-sticky-disk/action.yaml
  • .github/actions/setup-linux-nix-action-cache/action.yaml
  • .github/actions/setup-macos-nix-cache/action.yaml
  • .github/actions/setup-nix-cache/action.yaml
  • .github/actions/setup-nix/action.yaml
  • .github/workflows/ci.yaml
  • .github/workflows/pullfrog.yml
  • .github/workflows/release.yaml
  • .github/workflows/update-pricing.yaml
💤 Files with no reviewable changes (4)
  • .github/actions/setup-linux-blacksmith-sticky-disk/action.yaml
  • .github/actions/setup-linux-nix-action-cache/action.yaml
  • .github/actions/setup-macos-nix-cache/action.yaml
  • .github/actions/setup-nix/action.yaml
✅ Files skipped from review due to trivial changes (2)
  • .github/workflows/pullfrog.yml
  • .github/actions/pin-nix-deps/action.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/actions/build-macos-nix-native-package/action.yaml

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: af12a3c94eab
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 1.048s 1.064s 68.2ms 3
PR pkg.pr.new af12a3c 784.1ms 1.013s 70.5ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: af12a3c. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 865.5ms 974.6ms 0.89x 724.50 MiB 734.25 MiB 1.01x 1.16 GiB/s 1.03 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 227.4ms 215.5ms 1.06x 92.00 MiB 91.25 MiB 0.99x 4.43 GiB/s 4.67 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 914.6ms 1.10 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 888.0ms 1.13 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 196.9ms 5.11 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 140.3ms 7.18 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 52.9ms 7.3ms 7.25x 44.00 MiB 3.00 MiB 0.07x 0.03 MiB/s 0.21 MiB/s
claude session --offline --json 0.00 MiB 50.8ms 7.5ms 6.80x 44.25 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.21 MiB/s
codex daily --offline --json 0.00 MiB 49.6ms 7.3ms 6.79x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.12 MiB/s
codex session --offline --json 0.00 MiB 46.8ms 5.6ms 8.43x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.15 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 850.3ms 863.4ms 0.98x 724.00 MiB 743.25 MiB 1.03x 1.18 GiB/s 1.17 GiB/s
codex --offline --json 1.01 GiB 192.0ms 147.5ms 1.30x 90.50 MiB 89.25 MiB 0.99x 5.24 GiB/s 6.83 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: af12a3c94eab
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 917.8ms 911.3ms 77.8ms 3
PR pkg.pr.new af12a3c 1.091s 1.058s 95.4ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: af12a3c. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 876.7ms 1.231s 0.71x 730.25 MiB 720.50 MiB 0.99x 1.15 GiB/s 837.38 MiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 425.2ms 445.3ms 0.96x 95.75 MiB 90.00 MiB 0.94x 2.37 GiB/s 2.26 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 953.8ms 1.06 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 1.229s 838.62 MiB/s 1
codex --offline --json Package wrapper 1.01 GiB 499.7ms 2.01 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 449.3ms 2.24 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 59.0ms 62.5ms 0.94x 44.00 MiB 44.00 MiB 1.00x 0.03 MiB/s 0.02 MiB/s
claude session --offline --json 0.00 MiB 57.7ms 63.5ms 0.91x 44.25 MiB 44.25 MiB 1.00x 0.03 MiB/s 0.02 MiB/s
codex daily --offline --json 0.00 MiB 66.4ms 60.9ms 1.09x 44.25 MiB 44.00 MiB 0.99x 0.01 MiB/s 0.01 MiB/s
codex session --offline --json 0.00 MiB 61.5ms 66.6ms 0.92x 44.00 MiB 44.00 MiB 1.00x 0.01 MiB/s 0.01 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 1.020s 953.7ms 1.07x 708.25 MiB 728.75 MiB 1.03x 1010.31 MiB/s 1.06 GiB/s
codex --offline --json 1.01 GiB 743.0ms 508.3ms 1.46x 91.50 MiB 93.00 MiB 1.02x 1.35 GiB/s 1.98 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 12 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/actions/build-macos-nix-native-package/action.yaml">

<violation number="1" location=".github/actions/build-macos-nix-native-package/action.yaml:15">
P1: New required `cache-scope` input breaks existing release workflow call that does not pass it.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

ryoppippi and others added 2 commits June 12, 2026 00:43
ci.yaml and release.yaml both define a job named build-native-packages
with identical matrix `name` values, so the previous
`nix-<github.job>-<scope>-<hash>` key collided across the two
workflows. The release-tag run could purge ci.yaml's entries via
`purge-prefixes` (and vice versa). Prefix with github.workflow so each
workflow keeps its own namespace.

Co-authored-by: Codesmith <[email protected]>
ccusage-perf-comment and ccusage-rust-perf-comment were near-identical;
only --head-runtime, the output filename, and the comment marker differed.
Fold them into one job with a `runtime: [js, rust]` matrix, parameterised by
matrix.label/slug/head-runtime-arg. The displayed check names ("ccusage perf
comment" / "ccusage rust perf comment") are unchanged, and the cache is scoped
per runtime so the two legs do not share the job's cache key.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — one new commit (75fe0da) on top of the prior clean review adds github.workflow to the setup-nix-cache cache key prefix so workflows sharing a job id (e.g. build-native-packages in both ci.yaml and release.yaml) don't collide on cache entries.

  • Add github.workflow to cache key prefix — primary-key, restore-prefixes-first-match, and purge-prefixes in setup-nix-cache/action.yaml now use nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-... instead of just nix-${{ github.job }}-...

Pullfrog  | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — new commit eb3a34d collapses the two previously separate perf-comment jobs (ccusage-perf-comment and ccusage-rust-perf-comment) into a single matrix job with js and rust legs, scoping each leg's Nix cache via matrix.runtime.

  • Collapse perf-comment jobs into a runtime matrix — ccusage-perf-comment now runs two matrix legs (js/rust) instead of two standalone jobs, eliminating ~85 lines of duplicated step definitions while preserving the same behavior and output file names
  • Scope perf cache by runtime — passes cache-scope: ${{ matrix.runtime }} to setup-nix-cache so the Nix store caches for js and rust legs don't collide

Pullfrog  | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 75fe0daf7700
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 631.1ms 826.7ms 55.2ms 3
PR pkg.pr.new 75fe0da 797.1ms 882.1ms 57.5ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 75fe0da. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 875.8ms 880.5ms 0.99x 735.75 MiB 731.25 MiB 0.99x 1.15 GiB/s 1.14 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 173.0ms 173.7ms 1.00x 86.75 MiB 89.75 MiB 1.03x 5.82 GiB/s 5.80 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 858.9ms 1.17 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 770.1ms 1.31 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 154.0ms 6.54 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 117.8ms 8.54 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 49.5ms 37.0ms 1.34x 44.00 MiB 44.25 MiB 1.01x 0.03 MiB/s 0.04 MiB/s
claude session --offline --json 0.00 MiB 37.9ms 47.6ms 0.80x 44.00 MiB 44.25 MiB 1.01x 0.04 MiB/s 0.03 MiB/s
codex daily --offline --json 0.00 MiB 36.4ms 36.9ms 0.99x 44.00 MiB 44.00 MiB 1.00x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 37.8ms 37.3ms 1.02x 44.25 MiB 44.00 MiB 0.99x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 782.1ms 802.3ms 0.97x 727.50 MiB 740.50 MiB 1.02x 1.29 GiB/s 1.25 GiB/s
codex --offline --json 1.01 GiB 154.8ms 154.5ms 1.00x 92.00 MiB 91.50 MiB 0.99x 6.50 GiB/s 6.52 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 75fe0daf7700
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 814.7ms 847.9ms 62.3ms 3
PR pkg.pr.new 75fe0da 990.2ms 1.039s 64.7ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 75fe0da. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 905.6ms 951.2ms 0.95x 727.00 MiB 724.25 MiB 1.00x 1.11 GiB/s 1.06 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 203.8ms 192.1ms 1.06x 94.00 MiB 89.25 MiB 0.95x 4.94 GiB/s 5.24 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 904.6ms 1.11 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 860.6ms 1.17 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 177.0ms 5.69 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 131.1ms 7.68 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 48.4ms 6.7ms 7.23x 44.00 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.23 MiB/s
claude session --offline --json 0.00 MiB 41.9ms 5.8ms 7.18x 44.00 MiB 2.75 MiB 0.06x 0.04 MiB/s 0.26 MiB/s
codex daily --offline --json 0.00 MiB 40.0ms 5.6ms 7.10x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.15 MiB/s
codex session --offline --json 0.00 MiB 43.8ms 6.1ms 7.21x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 886.1ms 831.0ms 1.07x 733.25 MiB 737.75 MiB 1.01x 1.14 GiB/s 1.21 GiB/s
codex --offline --json 1.01 GiB 170.0ms 137.0ms 1.24x 92.00 MiB 92.25 MiB 1.00x 5.92 GiB/s 7.35 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: eb3a34d1c3bd
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 844.5ms 784.3ms 67.4ms 3
PR pkg.pr.new eb3a34d 888.2ms 898.8ms 64.6ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: eb3a34d. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 960.8ms 1.016s 0.95x 740.50 MiB 731.00 MiB 0.99x 1.05 GiB/s 1014.65 MiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 225.1ms 214.0ms 1.05x 93.50 MiB 93.75 MiB 1.00x 4.47 GiB/s 4.70 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 938.2ms 1.07 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 948.2ms 1.06 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 186.7ms 5.39 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 142.7ms 7.06 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 47.5ms 43.8ms 1.09x 44.25 MiB 44.25 MiB 1.00x 0.03 MiB/s 0.04 MiB/s
claude session --offline --json 0.00 MiB 42.7ms 48.7ms 0.88x 44.00 MiB 44.00 MiB 1.00x 0.04 MiB/s 0.03 MiB/s
codex daily --offline --json 0.00 MiB 43.4ms 44.8ms 0.97x 44.25 MiB 44.25 MiB 1.00x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 45.0ms 44.7ms 1.01x 44.00 MiB 44.00 MiB 1.00x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 878.7ms 929.9ms 0.94x 734.25 MiB 730.25 MiB 0.99x 1.15 GiB/s 1.08 GiB/s
codex --offline --json 1.01 GiB 178.4ms 186.4ms 0.96x 91.50 MiB 89.25 MiB 0.98x 5.64 GiB/s 5.40 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: eb3a34d1c3bd
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 778.1ms 782.7ms 64.2ms 3
PR pkg.pr.new eb3a34d 784.0ms 758.7ms 69.5ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: eb3a34d. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 990.5ms 1.053s 0.94x 728.75 MiB 741.00 MiB 1.02x 1.02 GiB/s 978.69 MiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 213.7ms 218.2ms 0.98x 91.25 MiB 92.75 MiB 1.02x 4.71 GiB/s 4.61 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 925.0ms 1.09 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 833.9ms 1.21 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 178.3ms 5.65 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 134.2ms 7.50 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 52.5ms 6.5ms 8.08x 44.25 MiB 3.00 MiB 0.07x 0.03 MiB/s 0.24 MiB/s
claude session --offline --json 0.00 MiB 51.4ms 6.7ms 7.64x 44.25 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.23 MiB/s
codex daily --offline --json 0.00 MiB 47.3ms 6.1ms 7.76x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s
codex session --offline --json 0.00 MiB 46.2ms 6.3ms 7.33x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 893.8ms 939.8ms 0.95x 733.50 MiB 737.00 MiB 1.00x 1.13 GiB/s 1.07 GiB/s
codex --offline --json 1.01 GiB 169.0ms 136.5ms 1.24x 89.75 MiB 93.00 MiB 1.04x 5.96 GiB/s 7.38 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

The cache hit but every job still recompiled ~130 crates: gc-max-store-size
makes cache-nix-action run `nix store gc` before saving, which dropped the
unrooted crane cargoArtifacts so they were never cached. pin-nix-deps was a
band-aid to root them, but it did not actually survive the trim.

These runners are ephemeral (restore -> build -> save), so the store never
accumulates across runs the way the old sticky disk did — there is nothing for
gc-max-store-size to bound. Removing it lets cache-nix-action save the whole
non-substitutable set (incl. the deps) as-is, so the next run restores them and
skips the recompile. The pin is then redundant and is removed with its action.
Stale entries from changed deps hashes still roll off via `purge`.
On an exact primary-key hit cache-nix-action skips saving, so the stale
deps-less entry from the gc-max-store-size era would persist while the deps
hash is unchanged. A version salt forces one miss, after which the gc-free
save stores the deps and subsequent runs restore them without recompiling.
cache-nix-action skips saving on an exact primary-key hit, so a key based only
on the deps hash kept the first run's (deps-less) store forever. Append
github.sha to the primary key so every commit writes a fresh entry, and restore
via the sha-less prefix to warm the deps from the newest prior entry. The cache
backend is effectively unlimited, so one entry per commit is fine and `purge`
rolls off the old ones. Replaces the v2 salt bust.
@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 6a62711bd3cc
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 734.8ms 663.3ms 63.0ms 3
PR pkg.pr.new 6a62711 684.4ms 734.0ms 55.9ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 6a62711. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 893.0ms 1.011s 0.88x 735.00 MiB 738.75 MiB 1.01x 1.13 GiB/s 1019.81 MiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 206.0ms 194.7ms 1.06x 90.50 MiB 94.50 MiB 1.04x 4.89 GiB/s 5.17 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 848.3ms 1.19 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 917.4ms 1.10 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 177.5ms 5.67 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 152.2ms 6.62 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 43.3ms 6.5ms 6.67x 44.00 MiB 3.00 MiB 0.07x 0.04 MiB/s 0.24 MiB/s
claude session --offline --json 0.00 MiB 45.6ms 6.0ms 7.57x 44.25 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.26 MiB/s
codex daily --offline --json 0.00 MiB 41.4ms 5.9ms 7.08x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.15 MiB/s
codex session --offline --json 0.00 MiB 38.8ms 5.3ms 7.32x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.16 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 867.2ms 875.4ms 0.99x 725.75 MiB 728.75 MiB 1.00x 1.16 GiB/s 1.15 GiB/s
codex --offline --json 1.01 GiB 168.5ms 139.9ms 1.20x 93.25 MiB 91.75 MiB 0.98x 5.97 GiB/s 7.20 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB -0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 6a62711bd3cc
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 816.6ms 917.8ms 66.1ms 3
PR pkg.pr.new 6a62711 745.1ms 856.2ms 61.2ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 6a62711. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 950.7ms 1.001s 0.95x 728.75 MiB 729.75 MiB 1.00x 1.06 GiB/s 1.01 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 226.5ms 229.0ms 0.99x 96.75 MiB 91.25 MiB 0.94x 4.44 GiB/s 4.40 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 912.7ms 1.10 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 861.6ms 1.17 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 168.3ms 5.98 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 137.0ms 7.35 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 38.7ms 43.9ms 0.88x 44.25 MiB 44.00 MiB 0.99x 0.04 MiB/s 0.04 MiB/s
claude session --offline --json 0.00 MiB 40.5ms 41.0ms 0.99x 44.00 MiB 44.25 MiB 1.01x 0.04 MiB/s 0.04 MiB/s
codex daily --offline --json 0.00 MiB 41.2ms 41.2ms 1.00x 44.25 MiB 44.25 MiB 1.00x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 38.2ms 44.1ms 0.87x 44.00 MiB 44.00 MiB 1.00x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 830.8ms 926.8ms 0.90x 729.25 MiB 739.25 MiB 1.01x 1.21 GiB/s 1.09 GiB/s
codex --offline --json 1.01 GiB 179.0ms 174.4ms 1.03x 87.75 MiB 89.75 MiB 1.02x 5.63 GiB/s 5.77 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB -0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

Mirror the nix cache: append github.sha so every commit saves a fresh entry
(actions/cache skips saving on an exact key hit) and restore-keys warms from
the newest prior entry.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — two new commits since the prior clean review at eb3a34d: dropping gc-max-store-size from setup-nix-cache and removing the vestigial pin-nix-deps action entirely, then busting cache keys with a nix-v2 salt so the new gc-free save takes effect on ephemeral runners.

  • Drop gc-max-store-size and delete pin-nix-deps — ephemeral runners don't accumulate store cruft across runs the way the old Blacksmith sticky disk did; gc-max-store-size before save only deletes unrooted crane deps and forces recompiles, so it's removed. pin-nix-deps (which existed to root deps against that GC) is deleted along with all 3 callers in build-linux-native-package and ci.yaml
  • Bust cache with nix-v2 prefix — without a cache key change, stale entries with gc-max-store-size behavior would persist as exact primary-key hits and never be re-saved; the v2 salt forces fresh entries

Pullfrog  | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — two new commits on top of the prior clean review replace the nix-v2 version-salt approach with a github.sha-based key strategy and add a dedicated Windows cargo cache action.

  • Replace nix-v2 salt with -${{ github.sha }} suffix — every commit now gets a unique primary key, ensuring cache-nix-action always saves a fresh entry; restore-prefixes-first-match uses the hash prefix to warm from any prior commit with the same deps lockfiles
  • Add setup-native-build-cache for Windows cargo artifacts — new composite action caching ~/.cargo/registry, ~/.cargo/git, and rust/target via actions/cache, keyed by runner.os/runner.arch/hashFiles/github.sha with hash-based and runner-based restore fallbacks; used only by build-windows-native-package

Pullfrog  | View workflow run | Using Big Pickle (free via Pullfrog for OSS) | 𝕏

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 9a35974b5b70
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 806.2ms 753.4ms 54.6ms 3
PR pkg.pr.new 9a35974 942.1ms 887.5ms 54.9ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 9a35974. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 925.4ms 930.7ms 0.99x 735.75 MiB 722.25 MiB 0.98x 1.09 GiB/s 1.08 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 190.2ms 188.3ms 1.01x 92.00 MiB 92.75 MiB 1.01x 5.29 GiB/s 5.35 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 979.9ms 1.03 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 899.5ms 1.12 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 181.1ms 5.56 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 128.6ms 7.83 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 37.1ms 5.5ms 6.70x 44.00 MiB 3.00 MiB 0.07x 0.04 MiB/s 0.28 MiB/s
claude session --offline --json 0.00 MiB 41.5ms 5.6ms 7.36x 44.00 MiB 2.75 MiB 0.06x 0.04 MiB/s 0.27 MiB/s
codex daily --offline --json 0.00 MiB 35.6ms 5.1ms 6.95x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.17 MiB/s
codex session --offline --json 0.00 MiB 43.1ms 5.1ms 8.43x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.17 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 910.7ms 857.0ms 1.06x 729.50 MiB 727.50 MiB 1.00x 1.11 GiB/s 1.17 GiB/s
codex --offline --json 1.01 GiB 165.6ms 130.5ms 1.27x 93.50 MiB 88.00 MiB 0.94x 6.08 GiB/s 7.72 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 9a35974b5b70
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 736.7ms 865.2ms 58.5ms 3
PR pkg.pr.new 9a35974 934.2ms 1.102s 59.9ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 9a35974. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 1.004s 889.7ms 1.13x 732.75 MiB 741.50 MiB 1.01x 1.00 GiB/s 1.13 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 194.5ms 189.9ms 1.02x 92.75 MiB 88.75 MiB 0.96x 5.18 GiB/s 5.30 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 891.7ms 1.13 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 956.3ms 1.05 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 167.5ms 6.01 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 136.4ms 7.38 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 35.8ms 37.8ms 0.95x 44.25 MiB 44.00 MiB 0.99x 0.04 MiB/s 0.04 MiB/s
claude session --offline --json 0.00 MiB 38.8ms 37.3ms 1.04x 44.25 MiB 44.00 MiB 0.99x 0.04 MiB/s 0.04 MiB/s
codex daily --offline --json 0.00 MiB 40.8ms 42.2ms 0.97x 44.25 MiB 44.25 MiB 1.00x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 41.3ms 39.8ms 1.04x 44.25 MiB 44.00 MiB 0.99x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 838.2ms 869.3ms 0.96x 732.75 MiB 744.25 MiB 1.02x 1.20 GiB/s 1.16 GiB/s
codex --offline --json 1.01 GiB 172.0ms 171.3ms 1.00x 91.25 MiB 90.75 MiB 0.99x 5.85 GiB/s 5.88 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: d71a9a7fef91
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 791.6ms 1.000s 78.6ms 3
PR pkg.pr.new d71a9a7 732.5ms 879.1ms 73.4ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: d71a9a7. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 899.8ms 1.162s 0.77x 741.00 MiB 729.00 MiB 0.98x 1.12 GiB/s 887.19 MiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 221.5ms 236.0ms 0.94x 92.50 MiB 93.75 MiB 1.01x 4.54 GiB/s 4.27 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 1.060s 972.34 MiB/s 1
claude --offline --json Installed native binary 1.01 GiB 987.1ms 1.02 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 205.0ms 4.91 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 145.9ms 6.90 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 47.7ms 7.0ms 6.85x 44.00 MiB 3.00 MiB 0.07x 0.03 MiB/s 0.22 MiB/s
claude session --offline --json 0.00 MiB 57.5ms 7.1ms 8.13x 44.25 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.22 MiB/s
codex daily --offline --json 0.00 MiB 48.7ms 6.1ms 8.03x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s
codex session --offline --json 0.00 MiB 45.1ms 6.2ms 7.25x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 808.6ms 926.2ms 0.87x 740.75 MiB 732.25 MiB 0.99x 1.25 GiB/s 1.09 GiB/s
codex --offline --json 1.01 GiB 202.3ms 151.1ms 1.34x 90.25 MiB 89.50 MiB 0.99x 4.98 GiB/s 6.66 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: d71a9a7fef91
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 933.8ms 729.4ms 70.0ms 3
PR pkg.pr.new d71a9a7 829.8ms 859.3ms 72.0ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: d71a9a7. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 971.0ms 1.039s 0.93x 733.00 MiB 733.00 MiB 1.00x 1.04 GiB/s 992.22 MiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 246.3ms 239.7ms 1.03x 91.50 MiB 94.00 MiB 1.03x 4.09 GiB/s 4.20 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 1.084s 950.87 MiB/s 1
claude --offline --json Installed native binary 1.01 GiB 1.025s 1005.90 MiB/s 1
codex --offline --json Package wrapper 1.01 GiB 202.6ms 4.97 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 149.2ms 6.75 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 57.5ms 51.8ms 1.11x 44.25 MiB 44.00 MiB 0.99x 0.03 MiB/s 0.03 MiB/s
claude session --offline --json 0.00 MiB 49.7ms 49.4ms 1.01x 44.00 MiB 44.25 MiB 1.01x 0.03 MiB/s 0.03 MiB/s
codex daily --offline --json 0.00 MiB 50.3ms 47.3ms 1.06x 44.25 MiB 44.25 MiB 1.00x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 47.9ms 47.9ms 1.00x 44.00 MiB 44.25 MiB 1.01x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 987.5ms 999.9ms 0.99x 736.75 MiB 737.50 MiB 1.00x 1.02 GiB/s 1.01 GiB/s
codex --offline --json 1.01 GiB 187.7ms 198.1ms 0.95x 91.25 MiB 92.50 MiB 1.01x 5.36 GiB/s 5.08 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: c92af09f534f
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 754.5ms 825.6ms 68.8ms 3
PR pkg.pr.new c92af09 795.1ms 1.056s 59.0ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: c92af09. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 968.8ms 946.8ms 1.02x 742.25 MiB 735.50 MiB 0.99x 1.04 GiB/s 1.06 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 196.3ms 190.2ms 1.03x 91.50 MiB 89.25 MiB 0.98x 5.13 GiB/s 5.29 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 927.0ms 1.09 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 880.9ms 1.14 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 166.9ms 6.03 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 128.0ms 7.86 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 37.1ms 5.7ms 6.55x 44.00 MiB 3.00 MiB 0.07x 0.04 MiB/s 0.27 MiB/s
claude session --offline --json 0.00 MiB 36.3ms 5.4ms 6.71x 44.25 MiB 2.75 MiB 0.06x 0.04 MiB/s 0.29 MiB/s
codex daily --offline --json 0.00 MiB 37.1ms 5.1ms 7.22x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.17 MiB/s
codex session --offline --json 0.00 MiB 36.7ms 5.4ms 6.80x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.16 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 856.4ms 805.1ms 1.06x 735.25 MiB 738.00 MiB 1.00x 1.18 GiB/s 1.25 GiB/s
codex --offline --json 1.01 GiB 165.8ms 135.0ms 1.23x 86.50 MiB 89.75 MiB 1.04x 6.07 GiB/s 7.46 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: c92af09f534f
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 791.7ms 593.1ms 60.8ms 3
PR pkg.pr.new c92af09 948.7ms 759.9ms 68.1ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: c92af09. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 998.6ms 984.8ms 1.01x 724.50 MiB 704.00 MiB 0.97x 1.01 GiB/s 1.02 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 206.3ms 209.0ms 0.99x 94.00 MiB 88.25 MiB 0.94x 4.88 GiB/s 4.82 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 1.006s 1.00 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 917.4ms 1.10 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 178.3ms 5.64 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 141.5ms 7.11 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 45.2ms 50.6ms 0.89x 44.00 MiB 44.25 MiB 1.01x 0.03 MiB/s 0.03 MiB/s
claude session --offline --json 0.00 MiB 46.3ms 46.4ms 1.00x 44.25 MiB 44.25 MiB 1.00x 0.03 MiB/s 0.03 MiB/s
codex daily --offline --json 0.00 MiB 46.4ms 43.2ms 1.07x 44.25 MiB 44.00 MiB 0.99x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 42.4ms 45.1ms 0.94x 44.00 MiB 44.25 MiB 1.01x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 906.4ms 954.2ms 0.95x 730.50 MiB 739.50 MiB 1.01x 1.11 GiB/s 1.06 GiB/s
codex --offline --json 1.01 GiB 183.3ms 183.2ms 1.00x 93.25 MiB 93.50 MiB 1.00x 5.49 GiB/s 5.50 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 9d3016c148ed
Base SHA: 11c782fd92f4

This compares the PR package against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 855.2ms 859.7ms 53.2ms 3
PR pkg.pr.new 9d3016c 940.8ms 874.1ms 53.4ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 9d3016c. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 847.8ms 815.2ms 1.04x 730.00 MiB 738.75 MiB 1.01x 1.19 GiB/s 1.24 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 182.3ms 174.6ms 1.04x 90.25 MiB 89.75 MiB 0.99x 5.52 GiB/s 5.77 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 863.6ms 1.17 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 741.9ms 1.36 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 150.5ms 6.69 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 121.3ms 8.30 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 46.6ms 38.3ms 1.22x 44.25 MiB 44.00 MiB 0.99x 0.03 MiB/s 0.04 MiB/s
claude session --offline --json 0.00 MiB 36.9ms 45.8ms 0.81x 44.25 MiB 44.00 MiB 0.99x 0.04 MiB/s 0.03 MiB/s
codex daily --offline --json 0.00 MiB 37.2ms 36.2ms 1.03x 44.25 MiB 44.00 MiB 0.99x 0.02 MiB/s 0.02 MiB/s
codex session --offline --json 0.00 MiB 37.0ms 36.8ms 1.01x 44.00 MiB 44.50 MiB 1.01x 0.02 MiB/s 0.02 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs the published ccusage package from pkg.pr.new, installed before measurement. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 778.4ms 712.9ms 1.09x 752.25 MiB 738.00 MiB 0.98x 1.29 GiB/s 1.41 GiB/s
codex --offline --json 1.01 GiB 155.6ms 154.7ms 1.01x 91.75 MiB 90.50 MiB 0.99x 6.47 GiB/s 6.51 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@github-actions

Copy link
Copy Markdown
Contributor

ccusage performance comparison

PR SHA: 9d3016c148ed
Base SHA: 11c782fd92f4

This compares the Rust PR release binary against the configured base package on the same CI runner.

Package runner startup

Execution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one bunx -p <url> ccusage --version run with an empty Bun install cache. Warm reuses that cache and reports the median of repeated runs.

Package SHA Execution setup Bunx temp cache Bunx warm median Warm samples
Base pkg.pr.new 11c782fd92f4 834.5ms 683.6ms 55.5ms 3
PR pkg.pr.new 9d3016c 875.7ms 798.4ms 54.4ms 3

Cached bunx execution performance

Runs the same large fixture through bunx -p <pkg.pr.new URL> ccusage after the Bun install cache has already been populated by the startup measurement. This separates cached package-runner execution from first-fetch package materialization.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base package: 11c782fd92f4; PR package: 9d3016c. Both run through bunx -p <pkg.pr.new URL> ccusage using the warmed Bun install cache from package runner startup, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
bunx -p <pkg> ccusage claude --offline --json 1.01 GiB 903.1ms 947.2ms 0.95x 719.25 MiB 728.75 MiB 1.01x 1.11 GiB/s 1.06 GiB/s
bunx -p <pkg> ccusage codex --offline --json 1.01 GiB 189.6ms 198.7ms 0.95x 88.75 MiB 89.25 MiB 1.01x 5.31 GiB/s 5.07 GiB/s

Package runtime diagnostics

Compares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
All rows run --offline --json, measured by hyperfine with 0 warmups and 1 runs. This isolates wrapper overhead from the installed native optional dependency and the workspace release binary built on the runner.

Command Runtime Input Median Throughput Samples
claude --offline --json Package wrapper 1.01 GiB 889.3ms 1.13 GiB/s 1
claude --offline --json Installed native binary 1.01 GiB 835.9ms 1.20 GiB/s 1
codex --offline --json Package wrapper 1.01 GiB 168.6ms 5.97 GiB/s 1
codex --offline --json Installed native binary 1.01 GiB 130.3ms 7.73 GiB/s 1

Committed fixture performance

Committed small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage.

Fixtures: Claude apps/ccusage/test/fixtures/claude (0.00 MiB, 2 files), Codex apps/ccusage/test/fixtures/codex (0.00 MiB, 1 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 2 warmups and 7 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude daily --offline --json 0.00 MiB 49.5ms 5.8ms 8.52x 44.00 MiB 2.75 MiB 0.06x 0.03 MiB/s 0.27 MiB/s
claude session --offline --json 0.00 MiB 39.7ms 6.0ms 6.62x 44.00 MiB 2.75 MiB 0.06x 0.04 MiB/s 0.26 MiB/s
codex daily --offline --json 0.00 MiB 42.1ms 6.1ms 6.93x 44.00 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s
codex session --offline --json 0.00 MiB 42.8ms 6.0ms 7.14x 44.25 MiB 2.75 MiB 0.06x 0.02 MiB/s 0.14 MiB/s

Large real-world-shaped fixture performance

Generated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures.

Fixtures: Claude /home/runner/_work/_temp/ccusage-large-fixture (1.01 GiB, 2,597 files), Codex /home/runner/_work/_temp/ccusage-large-codex-fixture (1.01 GiB, 2,597 files)
Base runs the published ccusage package from pkg.pr.new, installed before measurement; PR runs rust/target/release/ccusage directly. Both run --offline --json, measured by hyperfine with 0 warmups and 1 runs.
Peak RSS is measured separately with /usr/bin/time using 1 runs. Lower RSS ratios are better.

Command Input Base median PR median PR vs base Base peak RSS PR peak RSS PR/base RSS Base throughput PR throughput
claude --offline --json 1.01 GiB 817.9ms 887.3ms 0.92x 733.75 MiB 730.75 MiB 1.00x 1.23 GiB/s 1.13 GiB/s
codex --offline --json 1.01 GiB 176.5ms 142.9ms 1.24x 87.50 MiB 91.50 MiB 1.05x 5.70 GiB/s 7.05 GiB/s

Artifact size

Artifact Base PR Delta Ratio
packed ccusage-*.tgz 17.32 KiB 17.32 KiB +0.00 KiB 1.00x
installed native package binary 3324.84 KiB 3324.84 KiB +0.00 KiB 1.00x

Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees.

@ryoppippi
ryoppippi merged commit 31c7b00 into main Jun 12, 2026
35 checks passed
@ryoppippi
ryoppippi deleted the refactor/macos-cache-scope-by-matrix branch June 12, 2026 01:25
ryoppippi added a commit that referenced this pull request Oct 14, 2026
* ci(release): shallow-clone the tag in native build jobs

The native build jobs checked out the release tag with fetch-depth: 0,
but nothing in the build path reads git history: CI builds the same
outputs from a default shallow checkout. The full clone cost up to 41s
on build-windows-x64 in run 38470996208, which finished within 20s of
the slowest leg.

* ci: keep the Linux Nix store on a Blacksmith sticky disk

The cache-nix-action store cache misses most of the time. Across ~25
CI pushes to main between 2026-09-25 and 2026-10-13, the native build
legs restored an entry in only 3-6 of ~19 runs each (macOS and Linux
alike), and a hit roughly halves the build step (linux-arm64: 82s vs
213s). Blacksmith's Actions cache allows 25 GB per repository per week
with LRU eviction, while the per-commit keys introduced in #1296 write
a fresh 1.3-3 GB entry for every job on every run: ~12 GB per push to
main, ~6 GB per PR run, up to 2.5 GB per hourly update-pricing run.
#1296 assumed the backend was effectively unlimited; it is not. Even
on a hit, restoring takes ~60s and the post-job save another ~40s.

On Blacksmith Linux runners, mount /nix from a sticky disk instead. A
job gets a copy-on-write clone of the last committed snapshot, so the
whole store is present at mount time with no restore or save step and
nothing counted against the cache quota. macOS and GitHub-hosted
runners have no sticky disks and keep using cache-nix-action.

#1296 dropped an earlier sticky-disk setup. This one avoids its
problems:

- Keys are per job and matrix leg but carry no workflow name, so
  release.yaml's build-native-packages mounts the disk ci.yaml keeps
  warm on every push to main instead of starting cold. autofix, which
  only runs on pull requests and so would never commit, borrows
  preflight's disk via the new disk-job input; both build the treefmt
  closure.
- The old trim dropped Crane's cargoArtifacts because nothing rooted
  them (#1274, #1279). The Linux build now roots them inside
  /nix/var/nix/gcroots, which persists with the disk, and
  keep-derivations/keep-outputs make GC keep their build-time closure
  (toolchain, vendored sources) too.
- The weekly GC workflow (#1295) mounted its own disk and never touched
  the real ones. Trimming now happens in the job that commits: once the
  disk is over half full, `nix-collect-garbage --delete-old` runs before
  the job adds to the store. Jobs without persistent roots go cold once
  per trim.

Only runs on main that are not pull_request(_target) events commit,
whatever `save` says, so pull request code (including forks) can read
main's store but never write the store that main and releases build
from. cache-nix-action keeps its existing behaviour, now also gated by
`save`.

Not confirmed from Blacksmith's documentation: the disk's capacity
(hence the relative threshold and the `df` log on every run) and which
snapshot wins when two runs commit the same key concurrently. Either
outcome is a valid store for a cache.

* ci(release): stop persisting the Nix store from release builds

Release builds now mount the Linux sticky disk that CI commits on every
push to main. Committing it from the release as well would only race
those commits with an equivalent store.

On macOS, the release's own cache-nix-action entries never survive to
the next release under Blacksmith's cache eviction, yet each save took
~40s (build-mac-arm64 and build-mac-x64 in run 38470996208) before the
job could finish and let the npm job start.

Add a save-nix-cache input to both native build actions, forwarded to
setup-nix-cache, skip rooting the Crane artifacts when nothing will be
saved, and pass save-nix-cache: false from release.yaml.

* ci: mount test's Nix sticky disk in the perf comment legs

ccusage-perf-comment only runs on pull requests, and pull requests
never commit a sticky disk, so its own per-leg keys would stay freshly
formatted on every run. Mount the disk test commits on every push to
main instead: it installs `just` and resolves the same flake inputs.

Both legs mount it read-only, so the per-runtime cache-scope that kept
their saves apart is no longer needed.

Reported by pullfrog in review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant