Skip to content

Commit 6ca2776

Browse files
authored
ci: keep the Linux Nix store on a Blacksmith sticky disk (#1882)
* ci(release): shallow-clone the tag in native build jobs The native build jobs checked out the release tag with fetch-depth: 0, but nothing in the build path reads git history: CI builds the same outputs from a default shallow checkout. The full clone cost up to 41s on build-windows-x64 in run 38470996208, which finished within 20s of the slowest leg. * ci: keep the Linux Nix store on a Blacksmith sticky disk The cache-nix-action store cache misses most of the time. Across ~25 CI pushes to main between 2026-09-25 and 2026-10-13, the native build legs restored an entry in only 3-6 of ~19 runs each (macOS and Linux alike), and a hit roughly halves the build step (linux-arm64: 82s vs 213s). Blacksmith's Actions cache allows 25 GB per repository per week with LRU eviction, while the per-commit keys introduced in #1296 write a fresh 1.3-3 GB entry for every job on every run: ~12 GB per push to main, ~6 GB per PR run, up to 2.5 GB per hourly update-pricing run. #1296 assumed the backend was effectively unlimited; it is not. Even on a hit, restoring takes ~60s and the post-job save another ~40s. On Blacksmith Linux runners, mount /nix from a sticky disk instead. A job gets a copy-on-write clone of the last committed snapshot, so the whole store is present at mount time with no restore or save step and nothing counted against the cache quota. macOS and GitHub-hosted runners have no sticky disks and keep using cache-nix-action. #1296 dropped an earlier sticky-disk setup. This one avoids its problems: - Keys are per job and matrix leg but carry no workflow name, so release.yaml's build-native-packages mounts the disk ci.yaml keeps warm on every push to main instead of starting cold. autofix, which only runs on pull requests and so would never commit, borrows preflight's disk via the new disk-job input; both build the treefmt closure. - The old trim dropped Crane's cargoArtifacts because nothing rooted them (#1274, #1279). The Linux build now roots them inside /nix/var/nix/gcroots, which persists with the disk, and keep-derivations/keep-outputs make GC keep their build-time closure (toolchain, vendored sources) too. - The weekly GC workflow (#1295) mounted its own disk and never touched the real ones. Trimming now happens in the job that commits: once the disk is over half full, `nix-collect-garbage --delete-old` runs before the job adds to the store. Jobs without persistent roots go cold once per trim. Only runs on main that are not pull_request(_target) events commit, whatever `save` says, so pull request code (including forks) can read main's store but never write the store that main and releases build from. cache-nix-action keeps its existing behaviour, now also gated by `save`. Not confirmed from Blacksmith's documentation: the disk's capacity (hence the relative threshold and the `df` log on every run) and which snapshot wins when two runs commit the same key concurrently. Either outcome is a valid store for a cache. * ci(release): stop persisting the Nix store from release builds Release builds now mount the Linux sticky disk that CI commits on every push to main. Committing it from the release as well would only race those commits with an equivalent store. On macOS, the release's own cache-nix-action entries never survive to the next release under Blacksmith's cache eviction, yet each save took ~40s (build-mac-arm64 and build-mac-x64 in run 38470996208) before the job could finish and let the npm job start. Add a save-nix-cache input to both native build actions, forwarded to setup-nix-cache, skip rooting the Crane artifacts when nothing will be saved, and pass save-nix-cache: false from release.yaml. * ci: mount test's Nix sticky disk in the perf comment legs ccusage-perf-comment only runs on pull requests, and pull requests never commit a sticky disk, so its own per-leg keys would stay freshly formatted on every run. Mount the disk test commits on every push to main instead: it installs `just` and resolves the same flake inputs. Both legs mount it read-only, so the per-runtime cache-scope that kept their saves apart is no longer needed. Reported by pullfrog in review.
1 parent 74ba67b commit 6ca2776

6 files changed

Lines changed: 122 additions & 20 deletions

File tree

‎.github/actions/build-linux-native-package/action.yaml‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,16 @@ inputs:
1515
Unique matrix-entry identifier (the matrix `name`) used to scope the Nix
1616
cache so the arm64 and x64 legs do not share one key.
1717
required: true
18+
save-nix-cache:
19+
description: Whether this job may persist the Nix store; forwarded to setup-nix-cache as `save`.
20+
default: 'true'
1821
runs:
1922
using: composite
2023
steps:
2124
- uses: ./.github/actions/setup-nix-cache
2225
with:
2326
cache-scope: ${{ inputs.cache-scope }}
27+
save: ${{ inputs.save-nix-cache }}
2428
gc-max-store-size-linux: 4G
2529

2630
- name: Build Linux static binary
@@ -34,12 +38,18 @@ runs:
3438
nix build .#ccusage-static --print-build-logs
3539
fi
3640
41+
# The link lives inside the store's own gcroots, so it persists with a
42+
# sticky disk or a cache-nix-action entry and protects these artifacts
43+
# (and, through keep-outputs, their toolchain) from the next trim until a
44+
# later run replaces it.
3745
- name: Root current Crane artifacts for cache save
46+
if: inputs.save-nix-cache == 'true'
3847
shell: bash
3948
run: |
4049
system="$(nix eval --impure --raw --expr builtins.currentSystem)"
50+
mkdir -p /nix/var/nix/gcroots/ci
4151
nix build ".#packages.$system.ccusage-static.passthru.workspaceArtifacts.cacheRoot" \
42-
--out-link "$RUNNER_TEMP/ccusage-cargo-artifacts"
52+
--out-link /nix/var/nix/gcroots/ci/ccusage-cargo-artifacts
4353
4454
- name: Verify Linux binary is static
4555
shell: bash

‎.github/actions/build-macos-nix-native-package/action.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,16 @@ inputs:
1313
macOS Nix cache. arm64 and x64 share the same arm64 runner, so without a
1414
per-leg scope the two builds clobber one cache.
1515
required: true
16+
save-nix-cache:
17+
description: Whether this job may persist the Nix store; forwarded to setup-nix-cache as `save`.
18+
default: 'true'
1619
runs:
1720
using: composite
1821
steps:
1922
- uses: ./.github/actions/setup-nix-cache
2023
with:
2124
cache-scope: ${{ inputs.cache-scope }}
25+
save: ${{ inputs.save-nix-cache }}
2226

2327
- name: Build macOS binary
2428
shell: bash
Lines changed: 93 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
1-
name: Setup Nix with action cache
1+
name: Setup Nix with a persistent store
22
description: >
3-
Install Nix and restore/save the store with cache-nix-action, shared by Linux
4-
and macOS. cache-nix-action only caches non-substitutable paths (the crane
5-
deps, not the cache.nixos.org toolchain). Linux jobs root the current Crane
6-
artifacts after building; garbage collection then removes superseded
7-
generations before saving so restored caches do not grow on every commit.
8-
Blacksmith evicts stale entries, so no purge step is needed.
3+
Install Nix with a store that persists across runs. On Blacksmith Linux
4+
runners /nix is a sticky disk: a block device cloned from the last committed
5+
snapshot, so the whole store is there at mount time without a restore or save
6+
step and without counting against the Actions cache quota. Runners without
7+
sticky disks (macOS, GitHub-hosted) fall back to cache-nix-action, which
8+
restores and saves the store through the Actions cache.
99
inputs:
1010
cache-scope:
1111
description: >
@@ -14,14 +14,57 @@ inputs:
1414
each leg separate without leaning on runner.arch. Leave empty for
1515
single-leg jobs, which github.job alone already distinguishes.
1616
default: ''
17+
disk-job:
18+
description: >
19+
Job id whose sticky disk to mount instead of this job's own. Lets a job
20+
that never commits (e.g. one that only runs on pull requests) start from
21+
the store another job with the same closure keeps warm.
22+
default: ''
23+
save:
24+
description: >
25+
Whether this job may persist the store for later runs. Sticky disks only
26+
commit from runs on main that are not pull_request(_target) events, so
27+
pull request code can never write a store that main and releases read.
28+
default: 'true'
1729
gc-max-store-size-linux:
1830
description: >
19-
Optional Linux store size limit applied before cache save. Callers must
20-
root every artifact they want the garbage collector to retain.
31+
Optional Linux store size limit applied before a cache-nix-action save.
32+
Callers must root every artifact they want the garbage collector to retain.
2133
default: ''
2234
runs:
2335
using: composite
2436
steps:
37+
- name: Select Nix store backend
38+
id: backend
39+
shell: bash
40+
env:
41+
SAVE: ${{ inputs.save }}
42+
TRUSTED_REF: ${{ github.ref == 'refs/heads/main' && github.event_name != 'pull_request' && github.event_name != 'pull_request_target' }}
43+
run: |
44+
# stickydisk reads the same variable to find the Blacksmith agent.
45+
if [ "$RUNNER_OS" = Linux ] && [ -n "${BLACKSMITH_STICKY_DISK_GRPC_PORT:-}" ]; then
46+
echo "sticky=true" >> "$GITHUB_OUTPUT"
47+
else
48+
echo "sticky=false" >> "$GITHUB_OUTPUT"
49+
fi
50+
if [ "$SAVE" = true ] && [ "$TRUSTED_REF" = true ]; then
51+
echo "commit=true" >> "$GITHUB_OUTPUT"
52+
else
53+
echo "commit=false" >> "$GITHUB_OUTPUT"
54+
fi
55+
56+
# The key has no workflow name, so release.yaml's build-native-packages
57+
# mounts the disk ci.yaml keeps warm on every push to main. It is mounted
58+
# before Nix is installed; nix-quick-install-action keeps an existing
59+
# writable /nix and only adds its own paths to the database.
60+
- name: Mount Nix store sticky disk
61+
if: steps.backend.outputs.sticky == 'true'
62+
uses: useblacksmith/stickydisk@3f6be1451e3cad893b778bc93495c80452a5da26 # v1.7.2
63+
with:
64+
key: ${{ github.repository }}-nix-${{ runner.arch }}-${{ inputs.disk-job || github.job }}-${{ inputs.cache-scope }}
65+
path: /nix
66+
commit: ${{ steps.backend.outputs.commit }}
67+
2568
- name: Install Nix
2669
uses: nixbuild/nix-quick-install-action@9f63be77f412a248c9d9a65a4c82cf066cdf8f0c # v35
2770
with:
@@ -30,15 +73,33 @@ runs:
3073
# otherwise keep its documented default of max-jobs = 1 and build the
3174
# independent Crane artifact layers one at a time. cores stays at the
3275
# default 0 so the single dependency derivation, which compiles the whole
33-
# third-party graph, still gets every core.
76+
# third-party graph, still gets every core. keep-derivations and
77+
# keep-outputs make garbage collection keep the build-time closure
78+
# (toolchain, vendored sources, Crane deps) of every rooted artifact,
79+
# not only its runtime closure.
3480
nix_conf: |
3581
access-tokens = github.com=${{ github.token }}
3682
max-jobs = auto
83+
keep-derivations = true
84+
keep-outputs = true
85+
86+
# package.json only supplies the final binary version. It is excluded
87+
# because release bumps do not change the deps-only Crane derivation.
88+
# Likewise, checks and developer-tooling modules do not affect Cargo
89+
# artifacts; only package modules participate in this hash.
90+
- name: Hash Nix cache inputs
91+
if: steps.backend.outputs.sticky != 'true'
92+
id: deps
93+
shell: bash
94+
env:
95+
DEPS_HASH: ${{ hashFiles('flake.lock', 'flake.nix', 'default.nix', 'package.nix', 'nix/packages.nix', 'nix/static-package.nix', 'nix/darwin-x64-package.nix', 'rust-toolchain.toml', 'rust/Cargo.lock', 'rust/**/Cargo.toml') }}
96+
run: echo "hash=$DEPS_HASH" >> "$GITHUB_OUTPUT"
3797

3898
# github.workflow is included so workflows that share a job id (e.g.
3999
# build-native-packages in both ci.yaml and release.yaml) do not collide on
40100
# the same primary-key and block each other from saving updated entries.
41101
- name: Cache Nix store
102+
if: steps.backend.outputs.sticky != 'true'
42103
uses: nix-community/cache-nix-action@7df957e333c1e5da7721f60227dbba6d06080569 # v7.0.2
43104
with:
44105
# The commit sha is appended so every commit writes a fresh entry:
@@ -47,17 +108,14 @@ runs:
47108
# forever. restore-prefixes-first-match (without the sha) then restores
48109
# the newest prior entry to warm the deps. Blacksmith handles stale cache
49110
# eviction without the GitHub REST API used by this action's purge mode.
50-
# package.json only supplies the final binary version. It is excluded
51-
# because release bumps do not change the deps-only Crane derivation.
52-
# Likewise, checks and developer-tooling modules do not affect Cargo
53-
# artifacts; only package modules participate in this key.
54-
primary-key: nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-${{ hashFiles('flake.lock', 'flake.nix', 'default.nix', 'package.nix', 'nix/packages.nix', 'nix/static-package.nix', 'nix/darwin-x64-package.nix', 'rust-toolchain.toml', 'rust/Cargo.lock', 'rust/**/Cargo.toml') }}-${{ github.sha }}
111+
primary-key: nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-${{ steps.deps.outputs.hash }}-${{ github.sha }}
55112
# First prefix prefers the current deps hash. The second drops the hash so
56113
# a flake change that only adds developer tooling still restores the newest
57114
# prior store instead of starting from a cold cache.
58115
restore-prefixes-first-match: |
59-
nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-${{ hashFiles('flake.lock', 'flake.nix', 'default.nix', 'package.nix', 'nix/packages.nix', 'nix/static-package.nix', 'nix/darwin-x64-package.nix', 'rust-toolchain.toml', 'rust/Cargo.lock', 'rust/**/Cargo.toml') }}-
116+
nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-${{ steps.deps.outputs.hash }}-
60117
nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-
118+
save: ${{ inputs.save }}
61119
gc-max-store-size-linux: ${{ inputs.gc-max-store-size-linux }}
62120

63121
- name: Fix single-user Nix store ownership
@@ -69,3 +127,22 @@ runs:
69127
sudo chown -R "$USER:$(id -gn)" "$path"
70128
fi
71129
done
130+
131+
# A committed disk keeps everything every earlier run added, so trim it
132+
# once it passes half its capacity. Only committing runs trim: elsewhere the
133+
# changes are discarded and every later run would pay for the same GC.
134+
# Persistent roots under /nix/var/nix/gcroots (see build-linux-native-package)
135+
# and the current profile generations survive; out-links from earlier runs
136+
# pointed into their own workspaces and no longer count.
137+
- name: Trim Nix store sticky disk
138+
if: steps.backend.outputs.sticky == 'true'
139+
shell: bash
140+
env:
141+
COMMIT: ${{ steps.backend.outputs.commit }}
142+
run: |
143+
df -h /nix
144+
read -r size used < <(df -B1 --output=size,used /nix | tail -n1)
145+
if [ "$COMMIT" = true ] && [ "$used" -gt $((size / 2)) ]; then
146+
nix-collect-garbage --delete-old
147+
df -h /nix
148+
fi

‎.github/workflows/autofix.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,11 @@ jobs:
1919
with:
2020
ref: ${{ github.event.pull_request.head.sha }}
2121
persist-credentials: false
22+
# This workflow only runs on pull requests, which never commit a sticky
23+
# disk, so borrow preflight's: it builds the same treefmt closure on main.
2224
- uses: ./.github/actions/setup-nix-cache
25+
with:
26+
disk-job: preflight
2327
- name: Apply formatting and lint fixes
2428
run: |
2529
nix fmt -- --no-cache

‎.github/workflows/ci.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -469,8 +469,10 @@ jobs:
469469
- name: Setup Nix
470470
uses: ./.github/actions/setup-nix-cache
471471
with:
472-
# js and rust legs share this job id, so scope the cache per runtime.
473-
cache-scope: ${{ matrix.runtime }}
472+
# This job only runs on pull requests, which never commit a sticky
473+
# disk, so both legs read test's instead: it installs `just` and the
474+
# same flake inputs on every push to main.
475+
disk-job: test
474476
# This job only benchmarks the prebuilt ccusage binary bundled in the
475477
# installed pkg.pr.new package (no Rust build happens here), so the full
476478
# `nix develop` dev shell is unnecessary. Install just the tools the perf

‎.github/workflows/release.yaml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,20 +109,25 @@ jobs:
109109
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
110110
with:
111111
ref: ${{ needs.tagpr.outputs.tag }}
112-
fetch-depth: 0
113112
persist-credentials: false
113+
# Linux mounts the sticky disk CI commits on every push to main. Saving
114+
# here would only race those commits on Linux, and on macOS the release's
115+
# own cache entries are evicted long before the next release, while the
116+
# save delays the npm job.
114117
- uses: ./.github/actions/build-linux-native-package
115118
if: matrix.platform == 'linux'
116119
with:
117120
arch: ${{ matrix.arch }}
118121
binary: ${{ matrix.binary }}
119122
cache-scope: ${{ matrix.name }}
123+
save-nix-cache: false
120124
- uses: ./.github/actions/build-macos-nix-native-package
121125
if: matrix.platform == 'darwin'
122126
with:
123127
arch: ${{ matrix.arch }}
124128
binary: ${{ matrix.binary }}
125129
cache-scope: ${{ matrix.name }}
130+
save-nix-cache: false
126131
- uses: ./.github/actions/build-windows-native-package
127132
if: matrix.platform == 'win32'
128133
with:

0 commit comments

Comments
 (0)