Skip to content

Commit 4f1f470

Browse files
authored
perf: speed up CI and local builds (ccusage#1984)
* fix(test): isolate the copilot CLI snapshot from host XDG data The unified report also scans opencode under XDG_DATA_HOME, so a developer's real opencode database leaked into the snapshot. Point it at an empty fixture directory like the other home directories. * perf(nix): fetch only the LiteLLM pricing file instead of the repository Every Nix CI job unpacked the whole 2.7 GB BerriAI/litellm repository (7-25s per job), and pricing bumps re-lock it every few commits. The input is now a single-file URL pinned to a commit; update-litellm-lock.nu rewrites the commit in that URL before re-locking, and non-Nix cargo builds read the locked URL directly. * perf(nix): give tests, clippy and hawk a cached dev-profile dependency layer The test derivation ran cargo test in the release profile on top of the release layers, which hold neither dev-dependencies nor the test-snapshots feature variants, so ~50s of dependencies recompiled on every run; hawk compiled the whole workspace uncached (~25s). A dev-profile deps-only layer now serves tests and clippy, hawk gets its own layer, and main's test job roots both on the sticky disk. * perf(nix): skip crane's check pass in the cross dependency layers buildDepsOnly runs cargo check before cargo build, but only the host layer's check artifacts are used (by clippy and hawk). The static Linux and darwin-x64 layers only need the build output, which saves ~30s whenever dependencies change. * ci: start build and test jobs without waiting on preflight preflight (~45s) sat ahead of the slowest build on the critical path. ci-gate still requires it, and the pkg.pr.new publish now waits for it so a leaked secret is never published. * ci: build the Windows native package on a Dev Drive The rustup install (44s) and Cargo cache extraction (36s) were dominated by Defender and NTFS small-file costs. CARGO_HOME, RUSTUP_HOME and the target directory now live on a ReFS Dev Drive. The cache paths change, so the first run after merge starts cold. * perf(dev): link with lld on macOS and share dependencies via sccache lld links the test binaries an edit relinks faster than the Nix cctools ld64 (about 7-10% on leaf and mid-depth rebuilds). sccache shares compiled dependencies, C build scripts included, across worktrees: a fresh worktree's cargo test --no-run went from ~60s to ~45s wall (CPU 315s to 200s) with a warm cache. Set RUSTC_WRAPPER, even empty, to opt out. * perf(nix): drop the redundant check phase from the test deps layer The build command already compiles the test binaries; crane's check phase only added another cargo test --no-run for the ccusage bin. * ci: keep the Windows arm64 build off the Dev Drive setup-dev-drive needs the Hyper-V VHD cmdlets, which the release workflow's windows-11-arm runner is not known to provide, so arm64 falls back to the default Cargo and target locations. * ci: exclude Windows build directories from Defender instead of a Dev Drive Blacksmith's Windows image lacks the Hyper-V cmdlets setup-dev-drive needs (New-VHD), so the Dev Drive step failed. Defender path exclusions for the Cargo home, rustup home and workspace target the same small-file scan cost without Hyper-V, and the step is a no-op where Defender is absent. * ci: benchmark against the nearest base commit with a preview package Pricing snapshot commits are pushed to main with GITHUB_TOKEN, so they trigger no CI and never get a pkg.pr.new package. A PR based on one made both perf comment jobs wait out the full 5-minute package timeout and then skip the base comparison. Walk back over pricing-only commits to the nearest one with a package; any other commit without one keeps the old wait. * ci: drop the Windows Defender exclusion step Blacksmith's Windows image has no Defender (Add-MpPreference fails with 'Invalid class'), so the step only cost ~7s; the slow cache extraction is not a Defender scan cost there.
1 parent 0c76c3e commit 4f1f470

15 files changed

Lines changed: 199 additions & 41 deletions

File tree

‎.github/scripts/update-litellm-lock.nu‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,27 +5,38 @@ use ./pricing-lock.nu report
55

66
# Bump the pinned LiteLLM input and report whether it moved the pricing JSON that
77
# the build embeds, so the workflow can skip validating lock-only churn.
8+
#
9+
# The input is a single-file URL (not the 2.7 GB repository), and that URL embeds
10+
# the commit, so `nix flake update` alone cannot move it: rewrite the commit in
11+
# flake.nix first, then let Nix re-lock the new narHash.
812
def main [] {
913
let before = (locked-pricing)
14+
let rev = (
15+
^git ls-remote https://github.com/BerriAI/litellm refs/heads/main
16+
| split row "\t"
17+
| first
18+
)
19+
open --raw flake.nix
20+
| str replace --regex 'BerriAI/litellm/[0-9a-f]{40}/' $"BerriAI/litellm/($rev)/"
21+
| save --force flake.nix
1022
^nix flake update litellm
1123

1224
let changed = match ((locked-pricing) == $before) {
1325
false => true
1426
true => {
1527
print 'LiteLLM pricing JSON is unchanged; dropping the lock-only bump.'
16-
^git checkout -- flake.lock
28+
^git checkout -- flake.nix flake.lock
1729
false
1830
}
1931
}
2032

2133
report {
2234
changed: $changed
23-
paths: [flake.lock]
35+
paths: [flake.nix flake.lock]
2436
}
2537
}
2638

27-
# Fetch model_prices_and_context_window.json at the revision flake.lock pins.
39+
# Fetch the pricing JSON from the URL flake.lock pins.
2840
def locked-pricing []: nothing -> record {
29-
let rev = open --raw flake.lock | from json | get nodes.litellm.locked.rev
30-
http get $"https://raw.githubusercontent.com/BerriAI/litellm/($rev)/model_prices_and_context_window.json"
41+
http get (open --raw flake.lock | from json | get nodes.litellm.locked.url)
3142
}

‎.github/workflows/ci.yaml‎

Lines changed: 49 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -138,7 +138,9 @@ jobs:
138138
# local git hooks never run). Both are existing flake checks, so this reuses
139139
# their exact behavior instead of re-specifying flags here. The heavy test job
140140
# re-runs them inside `nix flake check`; that is intentional duplication so the
141-
# gate can fail fast before any build/test work starts.
141+
# gate can still fail the run early. Build and test jobs start alongside it
142+
# instead of waiting on it: it takes ~45s and sat on the critical path ahead
143+
# of the slowest build, and `ci-gate` still requires it to pass.
142144
preflight:
143145
name: security & lint preflight
144146
runs-on: blacksmith-32vcpu-ubuntu-2404
@@ -155,7 +157,7 @@ jobs:
155157
run: nix build --print-build-logs .#checks.x86_64-linux.gitleaks .#checks.x86_64-linux.treefmt
156158

157159
test:
158-
needs: [changes, tirith-scan, preflight]
160+
needs: [changes, tirith-scan]
159161
if: needs.changes.outputs.code-changed == 'true'
160162
runs-on: blacksmith-32vcpu-ubuntu-2404
161163
timeout-minutes: 30
@@ -185,9 +187,21 @@ jobs:
185187
run: .github/scripts/contribution-gate.test.nu
186188
- name: Babashka performance harness test
187189
run: apps/ccusage/scripts/compare-pr-performance_test.clj
190+
# The dev-profile test and hawk dependency layers are only reused if the
191+
# sticky disk keeps them, and this job roots nothing else. Rooting them in
192+
# the store's own gcroots protects them from the next trim (see
193+
# build-linux-native-package); only main commits the disk, so PRs skip it.
194+
- name: Root test dependency layers for the sticky disk
195+
if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
196+
run: |
197+
mkdir -p /nix/var/nix/gcroots/ci
198+
nix build .#ccusage-tests.passthru.cargoArtifacts \
199+
--out-link /nix/var/nix/gcroots/ci/ccusage-test-deps
200+
nix build .#ccusage-tests.passthru.hawkArtifacts \
201+
--out-link /nix/var/nix/gcroots/ci/ccusage-hawk-deps
188202
189203
build-native-packages:
190-
needs: [changes, tirith-scan, preflight]
204+
needs: [changes, tirith-scan]
191205
if: needs.changes.outputs.code-changed == 'true'
192206
name: ${{ matrix.name }}
193207
permissions:
@@ -247,8 +261,11 @@ jobs:
247261
binary: ${{ matrix.binary }}
248262

249263
npm-publish-dry-run-and-upload-pkg-pr-now:
264+
# preflight gates the publish rather than the builds: a leaked secret must
265+
# never reach pkg.pr.new, and it finishes long before the builds do.
250266
needs:
251267
- changes
268+
- preflight
252269
- build-native-packages
253270
if: needs.changes.outputs.code-changed == 'true'
254271
permissions:
@@ -501,6 +518,33 @@ jobs:
501518
fi
502519
echo "CCUSAGE_PREVIEW_URL=$preview_url" >> "$GITHUB_ENV"
503520
521+
# Pricing snapshot commits are pushed to main with GITHUB_TOKEN, which
522+
# triggers no CI, so they never get a preview package and the benchmark
523+
# would wait out its whole package timeout for one. They do not change
524+
# code, so walk back over them to the nearest commit that has a package.
525+
# Any other commit without one (e.g. its CI is still running) stops the
526+
# walk and keeps the real base, which is then waited for as before.
527+
- name: Resolve base preview package
528+
env:
529+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
530+
REPOSITORY: ${{ github.repository }}
531+
run: |
532+
git fetch --quiet --no-tags --depth=50 origin "$BASE_SHA"
533+
base="$BASE_SHA"
534+
for sha in $(git rev-list --first-parent -n 50 "$BASE_SHA"); do
535+
if curl --fail --head --location --silent --output /dev/null \
536+
"https://pkg.pr.new/$REPOSITORY/ccusage@$sha"; then
537+
base="$sha"
538+
break
539+
fi
540+
if git diff --name-only "$sha^" "$sha" \
541+
| grep -qvE '^(flake\.(lock|nix)|rust/crates/ccusage-pricing-snapshots/src/[^/]+\.json)$'; then
542+
break
543+
fi
544+
done
545+
echo "Comparing against base package ${base}"
546+
echo "BASE_PACKAGE_SHA=$base" >> "$GITHUB_ENV"
547+
504548
- name: Generate large ccusage fixture
505549
run: |
506550
just generate-large-fixture \
@@ -511,8 +555,8 @@ jobs:
511555
- name: Compare performance
512556
run: |
513557
apps/ccusage/scripts/compare-pr-performance.bb \
514-
--base-package-url "https://pkg.pr.new/${{ github.repository }}/ccusage@${{ github.event.pull_request.base.sha }}" \
515-
--base-sha "${{ github.event.pull_request.base.sha }}" \
558+
--base-package-url "https://pkg.pr.new/${{ github.repository }}/ccusage@$BASE_PACKAGE_SHA" \
559+
--base-sha "$BASE_PACKAGE_SHA" \
516560
--head-dir . \
517561
--head-package-url "$CCUSAGE_PREVIEW_URL" \
518562
${{ matrix.head-runtime-arg }} \

‎apps/ccusage/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -221,9 +221,9 @@ nix run github:ccusage/ccusage -- codex daily --offline
221221
nix build github:ccusage/ccusage
222222
```
223223

224-
Nix builds embed the LiteLLM pricing file from the locked `litellm` flake input, so sandboxed builds do not fetch pricing at build time. To update the locked pricing snapshot:
224+
Nix builds embed the LiteLLM pricing file from the locked `litellm` flake input (a single-file URL pinned to a LiteLLM commit), so sandboxed builds do not fetch pricing at build time. To update the locked pricing snapshot:
225225

226-
Non-Nix Cargo builds read the same locked LiteLLM revision from `flake.lock` and fetch the pricing file from that revision at build time.
226+
Non-Nix Cargo builds read the same locked LiteLLM revision URL from `flake.lock` and fetch the pricing file from that revision at build time.
227227

228228
```bash
229229
just update-litellm-pricing

‎flake.lock‎

Lines changed: 5 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎flake.nix‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,10 @@
2222
url = "github:cachix/git-hooks.nix";
2323
inputs.nixpkgs.follows = "nixpkgs";
2424
};
25+
# Only the pricing JSON is needed; a file input avoids unpacking the 2.7 GB
26+
# repository. The commit lives in the URL, so update-litellm-lock.nu rewrites it.
2527
litellm = {
26-
url = "github:BerriAI/litellm";
28+
url = "file+https://raw.githubusercontent.com/BerriAI/litellm/aa33b052d2292c9fcdc3d34523a968f1b41d0bd4/model_prices_and_context_window.json";
2729
flake = false;
2830
};
2931
models-dev = {

‎justfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ gen-bun-nix:
8383
8484
# Update the locked LiteLLM pricing snapshot and validate the result
8585
update-litellm-pricing:
86-
nix flake update litellm
86+
GITHUB_OUTPUT=/dev/null .github/scripts/update-litellm-lock.nu
8787
just check
8888

8989
# Regenerate committed models.dev snapshots from the pinned input

‎nix/checks.nix‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,16 @@ in
1717
rustToolchain = pkgs.rust-bin.fromRustupToolchainFile (root + /rust-toolchain.toml);
1818
craneLib = (inputs.crane.mkLib pkgs).overrideToolchain rustToolchain;
1919
inherit (config.packages.ccusage.passthru)
20-
cargoArtifacts
2120
workspaceArtifacts
2221
commonArgs
22+
depsOnlyArgs
2323
version
2424
;
25+
testArtifacts = import ./test-artifacts.nix {
26+
inherit craneLib depsOnlyArgs;
27+
cargoHawk = config.packages.cargo-hawk;
28+
};
29+
inherit (testArtifacts) cargoArtifacts;
2530
rustSrc = import ./rust-src.nix { inherit lib root; };
2631
repoSrc = lib.cleanSourceWith {
2732
name = "source";
@@ -40,6 +45,7 @@ in
4045
};
4146
ccusage-clippy = craneLib.cargoClippy (
4247
commonArgs
48+
// testArtifacts.consumerArgs
4349
// {
4450
src = rustSrc;
4551
sourceRoot = "source/rust";
@@ -51,18 +57,21 @@ in
5157
);
5258
# hawk belongs here rather than in treefmt: it analyses the whole workspace as a
5359
# closed world instead of a file at a time, and narrowing a visibility is a
54-
# semantic change, not formatting. It reuses the same cargo artifacts as clippy,
60+
# semantic change, not formatting. It reuses the same dev-profile test artifacts as clippy,
5561
# so the cost is its own analysis rather than another workspace build.
5662
ccusage-hawk = craneLib.mkCargoDerivation (
5763
commonArgs
64+
// testArtifacts.consumerArgs
5865
// {
5966
pname = "ccusage-hawk";
6067
src = rustSrc;
6168
sourceRoot = "source/rust";
6269
cargoLock = root + /rust/Cargo.lock;
63-
inherit cargoArtifacts;
70+
cargoArtifacts = testArtifacts.hawkArtifacts;
6471
nativeBuildInputs = commonArgs.nativeBuildInputs or [ ] ++ [ config.packages.cargo-hawk ];
65-
buildPhaseCargoCommand = "cargo hawk check --manifest-path Cargo.toml -D warnings";
72+
buildPhaseCargoCommand = ''
73+
cargo hawk check --manifest-path Cargo.toml --target-dir "''${CARGO_TARGET_DIR:-target}" -D warnings
74+
'';
6675
}
6776
);
6877
ccusage-fmt = craneLib.cargoFmt {

‎nix/darwin-x64-package.nix‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,10 @@ in
5252
};
5353
crossDepsOnlyArgs = config.packages.ccusage.passthru.depsOnlyArgs // {
5454
cargoExtraArgs = "-p ccusage --bin ccusage --target ${target}";
55+
# Clippy and hawk use the host layer, so this target only needs the
56+
# build output; Crane's default `cargo check` pass would add a
57+
# second, unused compile of every dependency.
58+
buildPhaseCargoCommand = "cargoWithProfile build -p ccusage --bin ccusage --target ${target}";
5559
buildInputs = crossBuildInputs;
5660
};
5761
crossDependencyArtifacts = crossCraneLib.buildDepsOnly crossDepsOnlyArgs;

‎nix/dev-shell.nix‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ in
2222
# Hand cargo the pinned LiteLLM snapshot the way the Nix packages do, so
2323
# `cargo build` in the dev shell stays offline and does not need the
2424
# fetch-litellm-pricing feature's rustls stack.
25-
CCUSAGE_PRICING_JSON_PATH = "${inputs.litellm}/model_prices_and_context_window.json";
25+
CCUSAGE_PRICING_JSON_PATH = "${inputs.litellm}";
2626

2727
buildInputs =
2828
(with pkgs; [
@@ -73,6 +73,7 @@ in
7373
]
7474
++ pkgs.lib.optionals pkgs.stdenv.hostPlatform.isDarwin [
7575
pkgs.apple-sdk_15
76+
pkgs.llvmPackages.lld
7677
]
7778
++ config.pre-commit.settings.enabledPackages;
7879

@@ -83,6 +84,23 @@ in
8384
*) export RUSTFLAGS="''${RUSTFLAGS:+$RUSTFLAGS }-C link-arg=-fuse-ld=mold" ;;
8485
esac
8586
fi
87+
# A test run relinks every test binary downstream of an edit, and
88+
# lld's Mach-O port links them faster than the cctools ld64 the Nix
89+
# clang wrapper defaults to, the same reason Linux uses mold above.
90+
if [ "$(uname -s)" = "Darwin" ]; then
91+
case " ''${RUSTFLAGS:-} " in
92+
*" -C link-arg=-fuse-ld=lld "*) ;;
93+
*) export RUSTFLAGS="''${RUSTFLAGS:+$RUSTFLAGS }-C link-arg=-fuse-ld=lld" ;;
94+
esac
95+
fi
96+
97+
# Every worktree starts with an empty rust/target, so without a shared
98+
# cache each one recompiles all dependencies, C build scripts included
99+
# (cc-rs routes those through sccache too). Cached dependencies cut a
100+
# fresh worktree's `cargo test --no-run` by about a quarter; workspace
101+
# crates are incremental, so sccache passes them straight to rustc.
102+
# An existing RUSTC_WRAPPER, even an empty one, wins as an opt-out.
103+
export RUSTC_WRAPPER="''${RUSTC_WRAPPER-${lib.getExe pkgs.sccache}}"
86104
87105
# Install whenever a lockfile is newer than the last `just install`, so
88106
# a `git pull` that bumps dependencies does not leave a stale tree. The

‎nix/static-package.nix‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,10 @@ in
6161
# Share the same deps-only cache key, then add static target settings.
6262
staticDepsOnlyArgs = config.packages.ccusage.passthru.depsOnlyArgs // {
6363
cargoExtraArgs = "-p ccusage --bin ccusage --target ${linuxStaticTarget}";
64+
# Clippy and hawk use the host layer, so this target only needs the
65+
# build output; Crane's default `cargo check` pass would add a
66+
# second, unused compile of every dependency.
67+
buildPhaseCargoCommand = "cargoWithProfile build -p ccusage --bin ccusage --target ${linuxStaticTarget}";
6468
nativeBuildInputs = with staticPkgs; [
6569
mold
6670
pkg-config

0 commit comments

Comments
 (0)