Skip to content

Commit dad1b5d

Browse files
build(perf): migrate benchmark harness to Babashka (#1432)
* build(perf): migrate benchmark harness to Babashka Replace the large Nushell PR benchmark script with a Babashka implementation split by data, system, benchmark, report, and orchestration responsibilities. The new process boundary keeps argv, environment, and working-directory data explicit while preserving hyperfine, package installation, memory, size, and Markdown behavior. Move the CI caller and profiling guidance to the executable Babashka entry point. Add focused tests behind their own Nix shebang so contributors can run the harness suite without adding Babashka to the full development shell. * docs(agents): document implementation language choices Route small command-oriented automation to Nushell and data-heavy, testable automation to Babashka. Keep production binaries in Rust and npm-integrated APIs in TypeScript so future tooling changes follow the same criteria used by the benchmark migration. * test(ci): run Babashka harness tests Execute the self-contained benchmark harness test entry point in the CI test job so changes to CLI parsing, normalization, fallback decisions, and report rendering cannot bypass pull request validation. * fix(perf): harden platform and tarball paths Normalize version-qualified Windows os.name values to win32 so native executable and package paths use the expected suffixes. Resolve relative pnpm pack filenames against the temporary destination while preserving the absolute paths emitted by current pnpm versions. Add regression coverage for both platform normalization and relative or absolute tarball filenames. * fix(perf): size local package fallbacks Use remote tarball sizing only after the corresponding preview package was installed successfully. When either package URL times out, benchmark and size the available local checkout so fallback runs can still produce a complete report. Cover base and head source selection and verify both unavailable URLs through a committed-fixture smoke run. * fix(perf): bound harness child processes and skip RSS on unsupported platforms Add a cancellable timeout to run-process and thread --package-runner-timeout-ms through the package URL probe, install, pnpm pack, and git rev-parse flows so a stalled child cannot outlive the deadline; give the curl probe and download explicit connect and read limits. measure-memory now warns once and skips gracefully when /usr/bin/time is unavailable (unsupported platforms) instead of throwing and aborting the entire benchmark run. Co-authored-by: Codesmith <[email protected]> * Revert "fix(perf): bound harness child processes and skip RSS on unsupported platforms" This reverts commit 9140a99. --------- Co-authored-by: Codesmith <[email protected]>
1 parent 997ad7f commit dad1b5d

13 files changed

Lines changed: 696 additions & 1057 deletions

File tree

‎.agents/skills/profile/references/rust.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,13 +72,13 @@ When CI performance comments are relevant, inspect options with `--help`, but do
7272
not treat help output as a profiling workload:
7373

7474
```sh
75-
apps/ccusage/scripts/compare-pr-performance.nu --head-runtime rust --help
75+
apps/ccusage/scripts/compare-pr-performance.bb --head-runtime rust --help
7676
```
7777

7878
To reproduce the workflow shape locally, pass real fixture and worktree inputs:
7979

8080
```sh
81-
apps/ccusage/scripts/compare-pr-performance.nu \
81+
apps/ccusage/scripts/compare-pr-performance.bb \
8282
--base-dir /tmp/ccusage-main \
8383
--head-dir "$PWD" \
8484
--head-runtime rust \

‎.github/workflows/ci.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,8 @@ jobs:
179179
mkdir -p "$HOME/.claude/projects"
180180
mkdir -p "$HOME/.config/claude/projects"
181181
just test-node
182+
- name: Babashka performance harness test
183+
run: apps/ccusage/scripts/compare-pr-performance_test.clj
182184

183185
build-native-packages:
184186
needs: [changes, tirith-scan, preflight]
@@ -492,7 +494,7 @@ jobs:
492494
493495
- name: Compare performance
494496
run: |
495-
apps/ccusage/scripts/compare-pr-performance.nu \
497+
apps/ccusage/scripts/compare-pr-performance.bb \
496498
--base-package-url "https://pkg.pr.new/${{ github.repository }}/ccusage@${{ github.event.pull_request.base.sha }}" \
497499
--base-sha "${{ github.event.pull_request.base.sha }}" \
498500
--head-dir . \

‎AGENTS.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,10 @@ Check the nearest package-specific `AGENTS.md` before editing package code:
3333
## Always-On Reminders
3434

3535
- The canonical user-facing CLI is `ccusage` with agent subcommands such as `ccusage amp`, `ccusage codex`, `ccusage opencode`, and `ccusage pi`.
36+
- Use Nushell for small structured automation that primarily orchestrates external commands, files, HTTP, and JSON.
37+
- Use Babashka when automation develops a reusable data model, multiple transformation stages, substantial reporting logic, or focused unit tests.
38+
- Use Rust for production binaries and CPU- or memory-intensive tooling.
39+
- Use TypeScript when the implementation directly benefits from the npm and JavaScript ecosystem or exposes a JavaScript API.
3640
- Standalone agent wrapper packages have been removed. Do not add docs, tests, or features that promote `ccusage-amp`, `ccusage-codex`, `ccusage-opencode`, or `ccusage-pi`.
3741
- Runtime libraries for bundled packages belong in `devDependencies` unless explicitly requested otherwise.
3842
- When initializing this repository environment, run `direnv allow` so the pinned Nix dev shell is activated.

‎apps/ccusage/AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,6 @@ This is the published `ccusage` npm package. The CLI implementation lives in Rus
1313

1414
- Published bin launcher: `src/cli.js`
1515
- Rust CLI implementation: `../../rust/crates/ccusage`
16-
- PR benchmark scripts: `scripts/compare-pr-performance.nu` and `scripts/generate-large-fixture.nu`
16+
- PR benchmark scripts: `scripts/compare-pr-performance.bb` and `scripts/generate-large-fixture.nu`
1717

1818
The package is distributed as the canonical native CLI. Keep the public surface centered on `ccusage`, agent subcommands such as `ccusage amp`, and stable `--json` output instead of library-style TypeScript exports.

‎apps/ccusage/justfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@ start *args:
3232
test-statusline:
3333
just test-statusline-file test/statusline-test.json
3434

35+
# Run focused tests for the Babashka performance harness.
36+
test-performance-harness:
37+
scripts/compare-pr-performance_test.clj
38+
3539
test-statusline-all:
3640
@echo 'Testing Sonnet 4:'
3741
just test-statusline-sonnet4
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env nix
2+
#! nix shell --inputs-from ../../.. nixpkgs#babashka-unwrapped nixpkgs#bun nixpkgs#curl nixpkgs#git nixpkgs#hyperfine nixpkgs#nodejs nixpkgs#pnpm --command bb
3+
4+
(require '[babashka.fs :as fs])
5+
6+
(def script-dir (fs/parent (fs/file *file*)))
7+
8+
(doseq [source ["data.clj" "system.clj" "benchmark.clj" "report.clj" "main.clj"]]
9+
(load-file (str (fs/path script-dir "compare_pr_performance" source))))
10+
11+
(when (= *file* (System/getProperty "babashka.file"))
12+
(try
13+
(apply -main *command-line-args*)
14+
(catch Exception error
15+
(binding [*out* *err*]
16+
(println (ex-message error)))
17+
(System/exit 1))))

0 commit comments

Comments
 (0)