Skip to content

Commit df89eb7

Browse files
authored
refactor(nix): manage Nix-built JS tooling with bun2nix (#1489)
* ci(nix-cache): fall back to the newest prior store `restore-prefixes-first-match` only carried the prefix that embeds the deps hash, so any change to flake.nix, flake.lock or nix/packages.nix sent the job to a completely cold Nix store even when the change merely added developer tooling and left every Cargo artefact untouched. Add a second, hashless prefix after it. The hashed prefix still wins when it matches, and otherwise the newest prior entry warms the deps instead of forcing a full rebuild. * refactor(nix): build publint with bun2nix nix/publint.nix carried a full pnpm-lock.yaml as a heredoc inside a Nix string, fed to fetchPnpmDeps. Bumping publint meant hand-editing that embedded lockfile and its fetcherVersion/hash pair, which is how the lockfile drifted from the workspace catalogue in the first place. Move the dependency set into nix/tools/publint as real files instead: package.json, the bun.lock that `bun install` resolved from it, and the bun.nix that `bun2nix` derived from that lockfile. The Nix expression lives next to them as default.nix, so the tool is self-contained. nix/bun-node-modules.nix is the shared piece: it turns one of those manifests into a node_modules tree via bun2nix's setup hook, scoping src to just package.json and bun.lock so neither the sibling Nix expression nor a tool's own scripts force the dependency tree to rebuild. Two wrinkles worth recording: - bun2nix stages each dependency through its own derivation and patches shebangs there against bun's `bun-with-fake-node` shim, so publint arrives bound to Bun's Node emulation. dontPatchShebangs cannot undo that, since the path is already baked into the cached package, so the install phase repoints it at real Node. publint is a Node packaging linter, and the baked-in path would otherwise keep bun in the runtime closure, which is now just bash and nodejs. - The hook only honours a wholesale replacement of its default install flags, so the linker and backend are restated alongside --frozen-lockfile. The publint flake check now skips nix/tools/*/package.json: those are dependency manifests for Nix-built tooling, not publishable packages. * fix(treefmt): tolerate oxlint batches that are wholly ignored treefmt matches files against a formatter's `includes` and passes them as arguments, so a batch can consist entirely of paths that nix/oxlint-check.json ignores. oxlint exits non-zero with "No files found to lint" in that case, which treefmt reports as a formatting failure for a file that was deliberately excluded. Pass --no-error-on-unmatched-pattern, oxlint's own switch for exactly this situation, mirroring the flag oxfmt is already given here. * refactor(nix): resolve models-dev-gen deps with bun2nix The generator's two runtime dependencies were vendored by hand: two pkgs.fetchurl calls with npm integrity strings copied out of the pinned models-dev input's bun.lock, extracted into a flat node_modules with tar. Bumping the input meant transcribing hashes between two lockfiles. Resolve them through bun2nix instead, from a package.json that pins the same remeda 2.33.7 and zod 3.24.2 that input's lockfile records. The generated bun.nix carries hashes identical to the fetchurl calls it replaces, and both snapshots the build emits are unchanged. Also move the tool into nix/tools/models-dev-gen so its Nix expression, the scripts it copies into the workspace and its dependency manifests sit together: models-dev-pricing.nix becomes default.nix, models-dev-gen.ts becomes gen.ts, and models-dev-compact.ts drops its now-redundant prefix. The remaining references follow: the node test invocation in the justfile and in the git hook, the tsconfig exclude and the oxlint ignore pattern. * build(nix): guard bun.nix drift and add gen-bun-nix Each tool under nix/tools now carries a bun.lock and the bun.nix that bun2nix derived from it. Nothing tied the two together: Renovate bumps bun.lock without knowing bun.nix exists, and a stale pair only shows up as a confusing "install wants the network" failure inside the sandbox. Add a bun-nix flake check that re-derives bun.nix from each committed bun.lock and diffs the result. bun2nix only parses the lockfile, so this needs no network access and runs in the sandbox like the other checks. `just gen-bun-nix` is the fix the check points at: it re-resolves every tool and regenerates its bun.nix. Both bun and the bun2nix CLI join the dev shell so that recipe can run. * fix(oxlint): follow the models-dev-gen rename 23b7d24 moved nix/models-dev-gen.ts to nix/tools/models-dev-gen/gen.ts and updated nix/oxlint-check.json, but .oxlintrc.json carries its own copy of the same ignore list and was left pointing at the old path. `just typecheck` therefore started type-checking the generator, which imports `./packages/core/src/generate.ts` from the upstream models.dev worktree it is copied into at build time. That module does not exist in this repository, so the file cannot type-check here by construction. * refactor(nix): extract a bun CLI wrapper builder nix/tools/publint/default.nix held the whole recipe for turning a bun2nix-resolved node_modules tree into a wrapped CLI, including the non-obvious part: bun2nix bakes `bun-with-fake-node` into the shebangs of the dependencies it stages, so a Node CLI has to be repointed at real Node or it runs under Bun's emulation and drags bun into the runtime closure. That is exactly the kind of detail the next tool would silently omit, so move it into nix/bun-cli.nix. The tool's own expression shrinks to the package name and its metadata. `runtimeInputs` is there for CLIs that shell out to other programs; publint does not need it. * refactor(release): build changelogithub with bun2nix changelogithub was a root devDependency purely so the release workflow could reach it through `nix develop --command pnpm changelogithub`: a full dev shell plus a pnpm workspace install, to run one CLI that touches nothing in this repository's source. Move it to nix/tools/changelogithub and run it as `nix run .#changelogithub`. It shells out to git to resolve the release tag, so git is wired into the wrapper instead of being inherited from the runner. This also retires the `changelogithub>c12` pnpm override. That existed only to keep chokidar 4.0.3 out of pnpm-lock.yaml, which pnpm rejected under `trustPolicy: no-downgrade` (see f7f7be5). bun resolves c12 3.3.4 and chokidar 5.0.0 on its own, so the constraint is satisfied by plain resolution rather than by an override. Dropping the dependency removes 751 lines from pnpm-lock.yaml. * build(deps): update nixpkgs for zizmor 1.27 zizmor 1.26.1 predates the parallel-step keys GitHub Actions shipped on 2026-06-25, so it failed schema validation on ci.yaml and audited nothing there. That gap was invisible: treefmt hands the matched files to zizmor in batches, and a batch containing any other workflow still exits zero, so the warning about ci.yaml scrolled past while its findings went uncollected. A batch consisting of ci.yaml alone exits non-zero instead, which is how it finally surfaced. zizmor 1.27.0 added experimental support for parallel steps, and nixpkgs now carries it. ci.yaml audits clean with no findings, so this turns an unaudited workflow into an audited one rather than merely quieting an error. * ci(pkg-pr-new): fetch on demand with pnpm dlx pkg-pr-new was a root devDependency, so every workspace install carried it even though the preview-release CI job is the only thing that runs it and it resolves nothing against the workspace. Fetch it with `pnpm dlx` at the point of use instead. The version is pinned in the workflow because Renovate does not update `dlx` specifiers, which is noted next to the call so the pin does not quietly rot. * refactor(types): move @types/node to the tool that needs it The root package.json carried @types/node solely so the root tsconfig could set `types: ["node"]`, and that tsconfig existed solely to cover `nix/**/*.ts`. With the last of those files now living under nix/tools/models-dev-gen, both belonged there rather than at the root: no `*.ts` file sits at the repository root, and apps/ccusage and docs each declare @types/node themselves. So the tool directory gets its own tsconfig, standalone like the other two in this repository, and pins @types/node next to the dependencies it already pins. gen.ts stays excluded for the same reason the oxlint config ignores it. The root package.json is left with no devDependencies at all. `just typecheck` still resolves node types for compact.test.ts, now out of nix/tools/models-dev-gen/node_modules, which `just gen-bun-nix` populates. * docs(nix): fix stale paths in the tool manifests Both nix/tools manifests described themselves in terms of the file layout that existed before the move: models-dev-gen pointed at `nix/models-dev-gen.ts` and publint at `nix/publint.nix`. Neither path exists any more; each tool's Nix expression and scripts are siblings of the manifest now. Reported on #1489 by pullfrog for the first of the two; the second is the same mistake, found by grepping for the pre-rename paths. * build(just): add an install recipe covering the nix/tools deps CONTRIBUTING tells contributors to run `just typecheck`, and moving @types/node into nix/tools/models-dev-gen broke that on a fresh checkout: oxlint resolves that directory's types out of its own node_modules, which nothing created. `pnpm install` does not cover it, since the tool directories sit outside the pnpm workspace. Consolidate setup into `just install`: the workspace install plus a `bun install --frozen-lockfile` for every tool directory carrying a tsconfig. Nix builds each tool's dependencies itself, so the ones with no TypeScript in them need nothing locally and are skipped. `git wt`'s hook now calls the recipe instead of running pnpm directly, so new worktrees stay usable without a manual step, and CONTRIBUTING lists it ahead of the checks it is a prerequisite for.
1 parent 0d968b9 commit df89eb7

37 files changed

Lines changed: 1272 additions & 987 deletions

‎.github/actions/setup-nix-cache/action.yaml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,12 @@ runs:
4343
# Likewise, checks and developer-tooling modules do not affect Cargo
4444
# artifacts; only package modules participate in this key.
4545
primary-key: nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-${{ hashFiles('flake.lock', 'flake.nix', 'default.nix', 'package.nix', 'nix/packages.nix', 'nix/static-package.nix', 'nix/darwin-x64-package.nix', 'rust-toolchain.toml', 'rust/Cargo.lock', 'rust/**/Cargo.toml') }}-${{ github.sha }}
46-
restore-prefixes-first-match: nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-${{ hashFiles('flake.lock', 'flake.nix', 'default.nix', 'package.nix', 'nix/packages.nix', 'nix/static-package.nix', 'nix/darwin-x64-package.nix', 'rust-toolchain.toml', 'rust/Cargo.lock', 'rust/**/Cargo.toml') }}-
46+
# First prefix prefers the current deps hash. The second drops the hash so
47+
# a flake change that only adds developer tooling still restores the newest
48+
# prior store instead of starting from a cold cache.
49+
restore-prefixes-first-match: |
50+
nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-${{ hashFiles('flake.lock', 'flake.nix', 'default.nix', 'package.nix', 'nix/packages.nix', 'nix/static-package.nix', 'nix/darwin-x64-package.nix', 'rust-toolchain.toml', 'rust/Cargo.lock', 'rust/**/Cargo.toml') }}-
51+
nix-${{ github.workflow }}-${{ github.job }}-${{ inputs.cache-scope }}-
4752
4853
- name: Fix single-user Nix store ownership
4954
if: runner.os != 'Windows'

‎.github/workflows/ci.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -275,7 +275,11 @@ jobs:
275275
tar -xf "$archive"
276276
done
277277
- id: publish
278-
run: pnpm pkg-pr-new publish --pnpm --bin './apps/ccusage' './packages/ccusage-linux-arm64' './packages/ccusage-linux-x64' './packages/ccusage-darwin-arm64' './packages/ccusage-darwin-x64' './packages/ccusage-win32-x64'
278+
# Fetched on demand rather than carried as a root devDependency: this
279+
# job is the only thing that runs it, and it never needs to resolve
280+
# against the workspace. Renovate does not update `dlx` specifiers, so
281+
# bump the pin here by hand.
282+
run: pnpm dlx [email protected] publish --pnpm --bin './apps/ccusage' './packages/ccusage-linux-arm64' './packages/ccusage-linux-x64' './packages/ccusage-darwin-arm64' './packages/ccusage-darwin-x64' './packages/ccusage-win32-x64'
279283
# Resolve the ccusage CLI preview URL and build the E2E fixture here (this
280284
# job has the checkout); the E2E jobs then run without a checkout.
281285
- id: preview-url

‎.github/workflows/release.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,6 @@ jobs:
166166
fetch-depth: 0
167167
persist-credentials: false
168168
- uses: ./.github/actions/setup-nix-cache
169-
- run: nix develop --command pnpm changelogithub
169+
- run: nix run .#changelogithub
170170
env:
171171
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

‎.oxlintrc.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"coverage",
88
".git",
99
".claude/settings.local.json",
10-
"nix/models-dev-gen.ts"
10+
"nix/tools/models-dev-gen/gen.ts"
1111
],
1212
"options": {
1313
"typeAware": true,

‎CONTRIBUTING.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,15 @@ Do not open a PR unless you have already been approved with `lgtm`.
4444
Before submitting a PR, run:
4545

4646
```bash
47+
just install
4748
just fmt
4849
just typecheck
4950
just test
5051
```
5152

53+
`just install` is only needed once per checkout (and after a lockfile change);
54+
`git wt` runs it for you when it creates a worktree.
55+
5256
Use the canonical `ccusage` command in docs and tests. Standalone wrapper packages such as `ccusage-codex`, `ccusage-opencode`, `ccusage-amp`, and `ccusage-pi` have been removed and should not be reintroduced.
5357

5458
Do not proactively create documentation files unless the change requires user-facing documentation.

‎flake.lock‎

Lines changed: 86 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎flake.nix‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,10 @@
33

44
inputs = {
55
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
6+
bun2nix = {
7+
url = "github:nix-community/bun2nix";
8+
inputs.nixpkgs.follows = "nixpkgs";
9+
};
610
crane.url = "github:ipetkov/crane";
711
flake-parts.url = "github:hercules-ci/flake-parts";
812
agent-skills = {

‎justfile‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,19 @@ mod rust
1919
default:
2020
@just --list
2121

22+
# Install the JS dependencies needed to work in this repository
23+
install:
24+
#!/usr/bin/env bash
25+
set -euo pipefail
26+
pnpm install --frozen-lockfile
27+
# Tool directories under nix/tools are outside the pnpm workspace. Nix builds
28+
# their dependencies itself, so only the ones carrying a tsconfig need a local
29+
# node_modules — `just typecheck` resolves their types out of it.
30+
for tsconfig in nix/tools/*/tsconfig.json; do
31+
toolDir="$(dirname "$tsconfig")"
32+
(cd "$toolDir" && bun install --frozen-lockfile)
33+
done
34+
2235
# Build every workspace package
2336
build: ccusage::build docs::build
2437

@@ -32,7 +45,7 @@ test: rust::test test-node
3245

3346
# Run Node's built-in test runner for TypeScript package and tooling tests
3447
test-node:
35-
TZ=UTC node --test apps/ccusage/src/cli.test.ts nix/models-dev-compact.test.ts
48+
TZ=UTC node --test apps/ccusage/src/cli.test.ts nix/tools/models-dev-gen/compact.test.ts
3649

3750
# Generate a large benchmark fixture for PR performance comparisons
3851
generate-large-fixture output_dir codex_output_dir size_mib="1024":
@@ -50,6 +63,16 @@ check:
5063
schema:
5164
nix run .#generate-schema
5265

66+
# Re-resolve the Nix-built JS tools under nix/tools and regenerate their bun.nix
67+
gen-bun-nix:
68+
#!/usr/bin/env bash
69+
set -euo pipefail
70+
for lockfile in nix/tools/*/bun.lock; do
71+
toolDir="$(dirname "$lockfile")"
72+
echo "Regenerating $toolDir"
73+
(cd "$toolDir" && bun install && bun2nix -o bun.nix)
74+
done
75+
5376
# Update the locked LiteLLM pricing snapshot and validate the result
5477
update-litellm-pricing:
5578
nix flake update litellm

‎nix/bun-cli.nix‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Wrap a CLI published on npm as a Nix package, from one of the `nix/tools/*`
2+
# dependency manifests.
3+
#
4+
# The manifest pins the CLI (and nothing else) as a dependency, so `bun.lock`
5+
# and `bun.nix` next to it describe the whole closure; `just gen-bun-nix`
6+
# regenerates them.
7+
{
8+
bunNodeModules,
9+
lib,
10+
makeWrapper,
11+
nodejs,
12+
stdenvNoCC,
13+
}:
14+
{
15+
# Directory containing package.json, bun.lock and bun.nix.
16+
toolDir,
17+
# npm package name; also the name of the wrapper this produces.
18+
pname,
19+
# Extra packages the CLI shells out to at runtime.
20+
runtimeInputs ? [ ],
21+
meta ? { },
22+
}:
23+
let
24+
manifest = lib.importJSON (toolDir + "/package.json");
25+
nodeModules = bunNodeModules { inherit toolDir; };
26+
in
27+
stdenvNoCC.mkDerivation {
28+
inherit pname;
29+
inherit (manifest) version;
30+
31+
dontUnpack = true;
32+
33+
nativeBuildInputs = [ makeWrapper ];
34+
35+
# The shebangs rewritten below are already absolute store paths, so the
36+
# default fixupPhase would leave them alone; skipping it keeps the intent
37+
# explicit and the phase from walking the whole dependency tree again.
38+
dontPatchShebangs = true;
39+
40+
installPhase = ''
41+
runHook preInstall
42+
43+
toolRoot="$out/lib/${pname}"
44+
mkdir -p "$toolRoot" "$out/bin"
45+
cp -R ${nodeModules}/node_modules "$toolRoot/node_modules"
46+
chmod -R u+w "$toolRoot/node_modules"
47+
48+
# bun2nix stages each dependency through its own derivation and patches
49+
# shebangs there against bun's `bun-with-fake-node` shim, so CLIs arrive
50+
# already pointing at Bun's Node emulation. Repoint them at real Node:
51+
# these are Node programs, and the baked-in path would otherwise keep bun
52+
# in the runtime closure.
53+
grep -rlE '^#!.*bun-with-fake-node/bin/node' "$toolRoot" | while IFS= read -r script; do
54+
sed -i "1s|^#!.*bun-with-fake-node/bin/node|#!${lib.getExe nodejs}|" "$script"
55+
done
56+
57+
# Invoke Node directly rather than relying on the shebang, so the wrapper
58+
# keeps working whatever bun2nix decides to write there.
59+
makeWrapper ${lib.getExe nodejs} "$out/bin/${pname}" \
60+
--add-flags "$toolRoot/node_modules/.bin/${pname}" \
61+
${lib.optionalString (runtimeInputs != [ ]) ''
62+
--prefix PATH : ${lib.makeBinPath runtimeInputs}
63+
''}
64+
65+
runHook postInstall
66+
'';
67+
68+
meta = {
69+
mainProgram = pname;
70+
platforms = lib.platforms.all;
71+
}
72+
// meta;
73+
}

‎nix/bun-node-modules.nix‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Turn one of the `nix/tools/*` dependency manifests into a `node_modules` tree.
2+
#
3+
# Each tool directory holds a `package.json`, the `bun.lock` that `bun install`
4+
# resolved from it, and the `bun.nix` that `bun2nix` derived from that lockfile
5+
# (regenerate both with `just gen-bun-nix`). bun2nix fetches every tarball as a
6+
# fixed-output derivation, so the install below needs no network access.
7+
{
8+
bun2nix,
9+
lib,
10+
stdenvNoCC,
11+
}:
12+
{
13+
# Directory containing package.json, bun.lock and bun.nix.
14+
toolDir,
15+
}:
16+
let
17+
manifest = lib.importJSON (toolDir + "/package.json");
18+
in
19+
stdenvNoCC.mkDerivation {
20+
pname = "${manifest.name}-node-modules";
21+
inherit (manifest) version;
22+
23+
# Only the two files `bun install` reads. A tool directory also holds its Nix
24+
# expression and any scripts the tool runs, and none of those should force the
25+
# dependency tree to rebuild. (`bun.nix` is read at evaluation time, so it does
26+
# not belong in the build inputs either.)
27+
src = lib.fileset.toSource {
28+
root = toolDir;
29+
fileset = lib.fileset.unions [
30+
(toolDir + "/package.json")
31+
(toolDir + "/bun.lock")
32+
];
33+
};
34+
35+
nativeBuildInputs = [ bun2nix.hook ];
36+
37+
bunDeps = bun2nix.fetchBunDeps {
38+
bunNix = toolDir + "/bun.nix";
39+
};
40+
41+
# The hook only honors a wholesale replacement of its default flags, so the
42+
# linker and backend are restated here alongside `--frozen-lockfile`.
43+
#
44+
# `--frozen-lockfile`: fail loudly if bun wants to resolve anything the
45+
# committed lockfile does not already pin, instead of reaching for the network.
46+
# `--linker=isolated`: what the hook defaults to on every platform.
47+
# `--backend=copyfile`: one backend on every platform. The hook picks
48+
# `symlink` on Darwin and bun's default elsewhere, which suits its own
49+
# `bun build --compile` flow where `node_modules` is transient; here the tree
50+
# *is* the output, so it should not depend on backend semantics.
51+
bunInstallFlags = "--linker=isolated --backend=copyfile --frozen-lockfile";
52+
53+
# These manifests pin plain library packages, so there are no lifecycle
54+
# scripts to run and nothing for bun to build, check or install.
55+
dontRunLifecycleScripts = true;
56+
dontUseBunBuild = true;
57+
dontUseBunCheck = true;
58+
dontUseBunInstall = true;
59+
60+
installPhase = ''
61+
runHook preInstall
62+
63+
mkdir -p "$out"
64+
cp -R node_modules "$out/node_modules"
65+
# The staged dependencies come out of the store read-only and `cp -R` keeps
66+
# those modes; consumers copy this tree into a writable build directory.
67+
chmod -R u+w "$out/node_modules"
68+
69+
runHook postInstall
70+
'';
71+
}

0 commit comments

Comments
 (0)