Repository navigation
ci: keep the Linux Nix store on a Blacksmith sticky disk - #1882
Conversation
The native build jobs checked out the release tag with fetch-depth: 0, but nothing in the build path reads git history: CI builds the same outputs from a default shallow checkout. The full clone cost up to 41s on build-windows-x64 in run 38470996208, which finished within 20s of the slowest leg.
The cache-nix-action store cache misses most of the time. Across ~25 CI pushes to main between 2026-09-25 and 2026-10-13, the native build legs restored an entry in only 3-6 of ~19 runs each (macOS and Linux alike), and a hit roughly halves the build step (linux-arm64: 82s vs 213s). Blacksmith's Actions cache allows 25 GB per repository per week with LRU eviction, while the per-commit keys introduced in #1296 write a fresh 1.3-3 GB entry for every job on every run: ~12 GB per push to main, ~6 GB per PR run, up to 2.5 GB per hourly update-pricing run. #1296 assumed the backend was effectively unlimited; it is not. Even on a hit, restoring takes ~60s and the post-job save another ~40s. On Blacksmith Linux runners, mount /nix from a sticky disk instead. A job gets a copy-on-write clone of the last committed snapshot, so the whole store is present at mount time with no restore or save step and nothing counted against the cache quota. macOS and GitHub-hosted runners have no sticky disks and keep using cache-nix-action. #1296 dropped an earlier sticky-disk setup. This one avoids its problems: - Keys are per job and matrix leg but carry no workflow name, so release.yaml's build-native-packages mounts the disk ci.yaml keeps warm on every push to main instead of starting cold. autofix, which only runs on pull requests and so would never commit, borrows preflight's disk via the new disk-job input; both build the treefmt closure. - The old trim dropped Crane's cargoArtifacts because nothing rooted them (#1274, #1279). The Linux build now roots them inside /nix/var/nix/gcroots, which persists with the disk, and keep-derivations/keep-outputs make GC keep their build-time closure (toolchain, vendored sources) too. - The weekly GC workflow (#1295) mounted its own disk and never touched the real ones. Trimming now happens in the job that commits: once the disk is over half full, `nix-collect-garbage --delete-old` runs before the job adds to the store. Jobs without persistent roots go cold once per trim. Only runs on main that are not pull_request(_target) events commit, whatever `save` says, so pull request code (including forks) can read main's store but never write the store that main and releases build from. cache-nix-action keeps its existing behaviour, now also gated by `save`. Not confirmed from Blacksmith's documentation: the disk's capacity (hence the relative threshold and the `df` log on every run) and which snapshot wins when two runs commit the same key concurrently. Either outcome is a valid store for a cache.
Release builds now mount the Linux sticky disk that CI commits on every push to main. Committing it from the release as well would only race those commits with an equivalent store. On macOS, the release's own cache-nix-action entries never survive to the next release under Blacksmith's cache eviction, yet each save took ~40s (build-mac-arm64 and build-mac-x64 in run 38470996208) before the job could finish and let the npm job start. Add a save-nix-cache input to both native build actions, forwarded to setup-nix-cache, skip rooting the Crane artifacts when nothing will be saved, and pass save-nix-cache: false from release.yaml.
|
@coderabbitai review |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Docstring Coverage |  |
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Description Check | |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | |
The title clearly and concisely describes the primary change: keeping the Linux Nix store on a Blacksmith sticky disk for CI. |
✨ Finishing Touches: 💡 1
-
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
🛠️ Fix failing CI checks: 💡
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | dd2a2cc | Commit Preview URL Branch Preview URL |
Oct 13 2026, 11:18 PM |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
ccusage
@ccusage/ccusage-darwin-arm64
@ccusage/ccusage-darwin-x64
@ccusage/ccusage-linux-arm64
@ccusage/ccusage-linux-x64
@ccusage/ccusage-win32-x64
commit: |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
|
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
Confirm the sticky-disk trust boundary and two persistence gaps before merging.
Reviewed changes
- Sticky-disk backend on Blacksmith Linux:
setup-nix-cachemounts auseblacksmith/stickydiskvolume at/nixwhen the Blacksmith agent variable is present, falling back tocache-nix-actionon macOS and GitHub-hosted runners. - Trust-gated commits: disks commit only when
saveis true and the run is onmainoutsidepull_request(_target)events. - Warmer Crane closures: Linux builds root artifacts under
/nix/var/nix/gcroots, enablekeep-derivations/keep-outputs, and committing runs trim past 50% capacity. - Release slim-down: native builds pass
save-nix-cache: falseand check out the tag shallowly;autofixborrowspreflight's disk viadisk-job.
muse-spark-1.3-contributor (free via Pullfrog for OSS) ｜ 𝕏
ccusage-perf-comment only runs on pull requests, and pull requests never commit a sticky disk, so its own per-leg keys would stay freshly formatted on every run. Mount the disk test commits on every push to main instead: it installs `just` and resolves the same flake inputs. Both legs mount it read-only, so the per-runtime cache-scope that kept their saves apart is no longer needed. Reported by pullfrog in review.
|
@coderabbitai review |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
ℹ️ Perf-leg fix looks good — one prior confirmation still open.
Reviewed changes since the prior pullfrog review (
868991e3) — one new commit (dd2a2cc4) addressing the cold perf-comment disk feedback.
- Pointed perf legs at
test's sticky disk: bothccusage-perf-commentlegs now passdisk-job: testand drop the per-runtimecache-scope, so their key matches the donortestkey on the sameblacksmith-32vcpu-ubuntu-2404runner.- Kept the share read-only: the perf job is
pull_request-only, so its mounts staycommit: falsewhiletestkeeps committing onmainpushes with thejustclosure the perf legs need.No new issues in this delta. Merging is still gated on the open thread: please confirm Blacksmith dashboard Branch Protection for sticky disks is enabled, since the in-action
commitgate alone cannot stop a fork PR from mounting the shared disk with its owncommit: true.
muse-spark-1.3-contributor (free via Pullfrog for OSS) ｜ 𝕏
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
|
… layers (#1921) * ci(nix): drop the workflow name from the Nix cache key release.yaml builds the macOS packages with save-nix-cache: false, so its keys (nix-tagpr-build-native-packages-...) never had an entry to restore: the v20.0.35 release (run 39053076844, build-mac-arm64) found no cache, built 161 derivations including the Rust toolchain, fetched 185 paths, and spent 139s in the build step. Key the store by job and matrix scope only, like the sticky disk key. Release builds run on pushes to main, so they now restore the newest entry CI saved on main, often for the very commit being released, and still never write one. The commit sha keeps workflows that share a job id from blocking each other's saves; only runs for the same commit can collide, and their stores hold the same build. The key is computed once in a step and exposed through its outputs so a later change can reuse it outside cache-nix-action's combined step. Existing nix-CI-* and nix-tagpr-* entries stop matching, so each cached job restores cold once. * ci(nix): save the macOS Nix store explicitly after rooting its closure No CI run has saved a macOS Nix store since #1882 (6ca2776). The build-mac-arm64 leg of every push to main from 2026-10-14T11:14Z to 2026-10-18 (24 runs) restored the same entry, nix-CI-build-native-packages-build-mac-arm64-05c2a29d...-75d48e98, written by the last saving run (38547180232), and its post-job "Cache Nix store" step finished in ~100 ms without a line of output. In cache-nix-action v7.0.2 the only silent exit from the save is `save` not being "true": #1882 started passing `save: ${{ inputs.save }}`, and the post step of a cache-nix-action nested in build-macos-nix-native-package -> setup-nix-cache evidently reads it as empty. Before #1882 it fell back to the "true" default and saved; the single-level tagpr npm job still saves today. So every macOS build restored a four-day-old 1.4 GB entry (~50 s, 47 s of it gtar extraction), found that the deps hash had moved on, and rebuilt the whole Crane chain: deps ~20 s, foundation ~9 s, adapters ~5 s, ccusage ~25 s. setup-nix-cache gains defer-save: it only restores, and exposes the primary key and whether this run may save. The macOS build saves with cache-nix-action/save as an ordinary step, so nothing depends on how post steps evaluate inputs. Saving needs a push to main as well as save-nix-cache, matching the sticky disks: a pull request would only spend ~40 s per leg writing an entry main cannot read, adding eviction pressure on Blacksmith's cache. Before saving, the build roots what the next run needs in the store's own gcroots, like the Linux build: the Crane cacheRoot (keep-outputs holds its toolchain and vendored sources) and the flake inputs this build fetched, found with `nix flake archive --dry-run` so unused pins are not downloaded. The save then collects every other path (gc-max-store-size-macos: 0), so an entry is one build closure instead of accumulating every commit's layers and binaries. * build(rust): install the minimal rustup profile plus explicit components rust-toolchain.toml named no profile, so rustup and rust-overlay used "default", which adds rust-docs: 728 MB and 65,585 files in rust-docs-1.97.1-aarch64-apple-darwin, roughly half the files of the macOS build closure. Every cold build downloads and unpacks it (the v20.0.35 release built rust-docs-1.97.1-aarch64-apple-darwin.tar.xz), and every cache-nix-action entry tars it, a large share of the work in the ~47 s gtar extraction on a macOS restore. Nothing builds or reads the docs. The minimal profile keeps rustc, cargo, and rust-std, and the existing components list still adds clippy, llvm-tools-preview, and rustfmt; rust-overlay also stops pulling libsecret into the Linux toolchain for non-minimal profiles. The toolchain derivation changes, so every Nix store and the Windows cargo cache rebuild once. * ci: correct which entry release macOS builds restore Release starts before CI saves the entry for the release commit, so it restores the previous main commit's store, which still matches because release commits only bump package.json files.
Summary
Move the Nix store on Blacksmith Linux runners from
cache-nix-actionto a Blacksmith sticky disk mounted at/nix, and stop release builds from persisting the store. macOS and GitHub-hosted runners keep usingcache-nix-action; reducing its write volume for macOS is left for a follow-up PR.The goal is a faster release workflow. After #1878 the release took ~8.4 minutes (run 38470996208), and its critical path was
build-linux-arm64(284s) →npm(120s) →release(49s). The arm64 leg spent ~90s fetching the toolchain and ~50s rebuilding the Crane deps because it never had a warm cache, then ~41s saving a cache entry that would be evicted again before the next release.Why the current cache misses
github.shato every key on the assumption that the backend was effectively unlimited, so every job writes a fresh 1.3–3 GB entry on every run:update-pricingrunnix-tagpr-*entries never survive until the next release.Why sticky disks, and why they failed before
A sticky disk is an ext4 block device. Each job mounts a copy-on-write clone of the last committed snapshot, so the whole store is there at mount time, with no tarball restore, no save step, and nothing counted against the cache quota (docs, action). The
stickydiskREADME has an official example that mounts/nixbefore runningnix-quick-install-action, which is the setup used here.#1296 removed an earlier sticky-disk setup. This PR addresses each of the problems it ran into:
github.job, so no job shared a warm store with another.<repo>-nix-<arch>-<job>-<matrix leg>, without a workflow name.release.yaml'sbuild-native-packagestherefore mounts the disk thatci.yamlcommits on every push to main.autofixruns only on pull requests, so it would never commit; it borrowspreflight's disk through the newdisk-jobinput, since both build the treefmt closure.cargoArtifactsbecause nothing rooted them (#1274 reverted, #1279)./nix/var/nix/gcroots/ci/ccusage-cargo-artifacts. That path is a real GC root and persists with the disk, unlike the previous$RUNNER_TEMPout-link.keep-derivationsandkeep-outputsmake GC keep their build-time closure (toolchain, vendored sources) as well.nix-collect-garbage --delete-oldbefore adding to the store. Jobs without persistent roots go cold once per trim.Trust model
Note
Blacksmith's "Sticky Disks: branch protection" is enabled (dashboard → Settings → Caching). Blacksmith only commits snapshots from default-branch
push,schedule, andworkflow_dispatchruns, and discards changes from every other run on the server side. Verified on this PR's CI: the mount step logsSticky disk changes will not be committed for this job (branch protection: event "pull_request" is not a trusted trigger).setup-nix-cachealso setscommit: falseunless the run is onmainand is not apull_requestorpull_request_targetevent. That gate is computed from workflow and action content, which a fork PR controls: apull_requestrun uses the PR's version of the workflow files, so a fork could add its ownstickydiskstep withcommit: true. The in-action gate only keeps same-repository PR runs and release builds from committing by accident. The server-side setting is the actual trust boundary for the store that main and releases build from (raised by Cubic and pullfrog in review).The
cache-nix-actionpath keeps its existing behaviour, now also gated bysave. Actions cache entries from PRs are scoped to the PR ref, so they never reach main.What Changed
.github/actions/setup-nix-cacheBLACKSMITH_STICKY_DISK_GRPC_PORTvariable thatstickydiskitself uses;cache-nix-actioneverywhere else.useblacksmith/stickydiskv1.7.2 (pinned by SHA) at/nixbefore installing Nix.keep-derivationsandkeep-outputssettings.disk-jobandsaveinputs.df -h /nixon every sticky-disk run and trims committing runs at 50% usage.hashFileslist once, in its own step..github/actions/build-linux-native-package: persistent GC root for the Crane artifacts, plus asave-nix-cacheinput..github/actions/build-macos-nix-native-package:save-nix-cacheinput..github/workflows/release.yamlsave-nix-cache: false. On Linux a commit would only race CI's equivalent commits; on macOS the ~40s save was pure overhead because the entry never survives until the next release.build-windows-x64..github/workflows/autofix.yaml:disk-job: preflight..github/workflows/ci.yaml: bothccusage-perf-commentlegs usedisk-job: test. They run only on pull requests and would otherwise mount a freshly formatted disk every time. The per-runtimecache-scopeis dropped because read-only mounts never save.Expected impact
tirith-scan,preflight,test, the Linux native builds,npm-publish-dry-run-and-upload-pkg-pr-now,update-pricing,pullfrog,release, plusautofixand the perf comments through their donor disks): no ~60s restore and no ~35–40s save, and the deps stay warm instead of being rebuilt on most runs.Not confirmed
/dev/vdb 590G 111M 590G 1% /nix), against 2–5 GB used per job. The trim threshold is relative (50%) and every run logsdf -h /nix; in practice the trim is a safety net. A scheduled GC was considered and rejected: a GC-only job sees only persistent roots, so it would wipe every job's tool closures on every schedule, and it would have to enumerate every disk key, which is the failure mode of chore(ci): remove the dead weekly sticky-disk GC workflow #1295.commit: false. These runs only exercise the read-only path: pull requests never commit, and the new keys start freshly formatted. The warm path can only be observed on main after merge.Testing
nix build .#checks.aarch64-darwin.treefmtpasses. It covers actionlint and zizmor with the repository config.nix run nixpkgs#actionlinton the changed workflows reports only the existingparallel:steps inci.yamlandupdate-pricing.yaml.cache-nix-actionignores the blank lines an unset input leaves behind (getInputAsArrayfilters empty strings).nix-quick-install-actionv35 keeps an existing writable/nix: it extracts with--skip-old-filesand registers its own paths with--load-db.