Skip to content

Commit 86707db

Browse files
authored
build(perf): make the PR performance comparison catch real regressions (ccusage#2061)
* build(perf): shape the large fixture like current Claude Code and Codex logs The CI performance comparison missed the double parse that ccusage#1719 added to every Claude line containing `null`, because the generated fixture never contained `null`: each line was a minimal assistant usage record padded with a single text block of `x` characters. Real Claude Code lines now carry `stop_sequence`, `container`, `stop_details` and other null-valued fields, nested `usage.iterations` and cache-creation TTL splits, and most lines are not usage records at all. On the old fixture v20.0.20 -> v20.0.21 moved CPU time by +8%; on real-shaped lines it is +55%. The generator now writes sessions whose line types, per-type byte share, and field shapes follow aggregate statistics taken from real logs (structure only; every value is synthesized from a seeded PRNG, so the output is deterministic): - Claude: user/tool-result, attachment, api-request(-blob) and metadata lines around assistant responses written one line per content block with repeated usage, null-valued fields, `usage.iterations` including occasional advisor iterations, 5m/1h cache-creation split, escaped and non-ASCII text, 48 projects, sidechain `subagents/agent-*.jsonl` files that replay parent message IDs, and legacy files with `progress` lines wrapping AgentProgress assistant messages. - Codex: dated `sessions/YYYY/MM/DD/rollout-*.jsonl` files with session_meta, turn_context, reasoning, tool call/output and item_completed lines, plus token_count (with null-heavy rate limits) and token_usage_record per response. As in real Codex homes, rollouts older than two weeks are zstd-compressed (`.jsonl.zst`) and some sit in `archived_sessions/`; on real data about 90% of Codex bytes are now read through zstd, a path the old fixture never touched. The file-size distribution (of uncompressed bytes) and the CLI are unchanged. * build(perf): build base and PR from source and judge them by CPU time The PR performance comment could not show a regression or a real improvement: - Unequal artifacts. Base ran the published `ccusage` npm package (Node launcher spawning the native binary) while the Rust leg ran the bare native binary, so every PR showed a ~2.6x "win" on the committed fixtures that was only launcher startup. - One sample. The large fixture ran with 0 warmups and 1 run, and the base and PR runs were not interleaved, so runner noise dominated. - The committed fixtures are a few KiB and measure process startup. - No verdict. Nothing said whether a number was a regression. The job now builds both binaries with `nix build .#ccusage-static`, the derivation the Linux x64 native package ships, on the same runner: the PR head from the checkout and the base from `github.event.pull_request.base.sha`, the same base commit as before. The walk back over pricing-only commits only existed to find a commit with a pkg.pr.new package, so it goes. The job reuses the build-native-packages sticky disk for dependency artifacts and no longer waits for pkg.pr.new, so the JS launcher leg, the committed-fixture section, and the package diagnostics are gone too. The harness runs `claude daily`, `claude session`, `codex daily` and the unified `daily` over the realistic generated fixture in interleaved rounds (each round one hyperfine invocation with both binaries, order rotated), recording wall time, CPU time, peak RSS and, where the platform exposes it, instructions. It reports medians with min-max spread and per-round PR/base ratios. A command is flagged when the median CPU-time or peak-RSS ratio moves past 10% and at least four in five rounds agree; the verdict line and a GitHub warning annotation say so. Wall time is shown but not judged: it moves with runner load, while CPU time paired within a round does not. The check warns rather than fails: a failing check needs a calibrated false-positive rate across many runs on these runners and an escape hatch for intentional trade-offs, and neither exists yet. * build(perf): add `just bench-history` to compare refs Finding which release introduced a slowdown meant building each tag by hand and timing it ad hoc. `just bench-history <ref|binary>...` builds each ref in release mode into `rust/target/bench-history` (one shared cargo target dir, binaries cached by commit), generates the realistic fixture once per generator revision and size, and runs every report command over all refs in the same interleaved rounds as the PR comparison. Each ref is judged against the previous one with the same CPU-time/peak-RSS rule, so a regression shows up at the step that introduced it. `--real` adds a section over the logs the current environment points at. It reuses the PR harness modules instead of adding a parallel tool, and the profile skill now points at it. * fix(perf): print the regression annotation on stdout GitHub Actions reads workflow commands only from stdout, but the `::warning` line was printed while `*out*` was bound to `*err*`, so a flagged regression never produced an annotation. Print it on stdout and pass the GitHub Actions check in, so a test can assert the stream. Reported by Pullfrog on ccusage#2061. * fix(perf): reject bench-history runs below the minimum `parse-history-cli` validated a copy with `:runs` raised to 3 but returned the original options, so `--runs 2` passed validation and then measured two rounds, too few for the four-in-five agreement rule. Validate the options that are returned, so fewer than three rounds is rejected as in the PR comparison, and say so in both `--runs` helps. Reported by Pullfrog on ccusage#2061. * fix(perf): run fixture benchmarks in an empty environment The fixture runs pointed Claude and Codex at the fixture and moved HOME and XDG_* to an empty directory, but every other agent has its own data-dir variable (AMP_DATA_DIR, OPENCODE_DATA_DIR, PI_AGENT_DIR, GEMINI_DATA_DIR and a dozen more), and a value set in the caller's shell still pointed the unified `daily` report at real logs. Listing every variable would rot as agents are added, so the fixture commands now run under `env -i` with only the variables the run needs; with XDG_* unset the agents fall back to the empty HOME. The local-logs mode keeps the caller's environment on purpose. Reported by Cubic on ccusage#2061. * fix(perf): make bench-history safe to point at binaries and run twice Four ways `just bench-history` could measure the wrong thing or fail: - Binary-path variants were labelled by basename, so two builds at `a/ccusage` and `b/ccusage` collided in hyperfine's results. Label them by the argument as given, and reject a ref or path passed twice. - `--fixture-dir` alone silently reused the Claude logs as the Codex fixture. Require both directories or neither. - The cached binary was copied without attributes; copy them so the executable bit is kept on every platform. - Every run checked refs out into the shared `<cache>/src` worktree and removed it on exit, so a second run sharing the cache deleted the first run's checkout mid-build. Each run now uses its own temporary worktree; the shared cargo target is safe because cargo locks it. The first three were reported by Cubic on ccusage#2061; the fourth showed up while bisecting with two runs at once. * fix(perf): report memory-only improvements and history instruction ratios The overall verdict counted only CPU improvements, so a PR that cut peak RSS read "no significant change" above a table saying "less memory". Count CPU and RSS improvements alike and name what improved per command. The history tables also dropped the instruction-count ratio the PR comparison shows; add it as a column whenever hardware counters are available. Reported by Cubic on ccusage#2061. * test(perf): restore duplicate-name and time-unit hyperfine cases The hyperfine reader rewrite kept the missing-name and malformed-output cases but lost the duplicate-name and non-second-unit ones, so a change that accepted ambiguous or mis-scaled samples would pass. Restore both against the new per-run reader. Reported by Cubic on ccusage#2061. * fix(perf): print the generated fixture's size on disk With zstd-compressed Codex rollouts the generator's `Size` line showed the uncompressed bytes, about four times what it wrote. Print the size on disk from `Bun.write` and the uncompressed size separately. Reported by Cubic on ccusage#2061. * fix(ci): measure the PR merged onto its base commit The job built the PR side from `pull_request.head.sha` and the base from `pull_request.base.sha`. When a branch is behind its base, the two binaries differ by the base's newer commits as well as the PR's own change, and those commits show up as PR regressions or improvements. Build the checked-out merge commit (`github.sha`) instead. It is the PR applied on top of the base commit, so the diff measured is exactly the PR. A step fails the job if the base is not an ancestor of the built commit, so the report's "merged onto Base:" line always names what was actually built. BASE_SHA/HEAD_SHA fall back to `merge_group.base_sha` and `github.sha`, and the PR-comment step is limited to pull_request events, so the job also works if a merge_group trigger is added. Reported by CodeRabbit on ccusage#2061. * fix(perf): cache built bench-history binaries atomically `build-commit!` copied the release binary straight to the cache path that later runs trust when it exists, so an interrupted copy left a truncated binary that every later run reused. Copy to a `.partial` sibling and rename it into place with an atomic move. Reported by CodeRabbit on ccusage#2061. * fix(perf): start fixture generation from an empty directory A cached fixture without its `.complete` marker was regenerated into the same directory, so files from an interrupted run could remain next to the new output. Delete the directory first, so the marker always describes exactly what the generator wrote. Reported by CodeRabbit on ccusage#2061. * fix(perf): order Codex usage records like real rollouts and add compactions The generator wrote each response's `token_count` before its `token_usage_record`. Real rollouts write the record first, and the parser settles a pending record only when a later `token_count` advances the cumulative total, so every synthetic record stayed in the compaction pending map until the end of its file. That inflated peak RSS in a way real logs never do: on the corrected fixture `codex daily` peaks at 56 MiB on main instead of 69 MiB. Write the record before the `token_count`, as Codex does, and add the remote compactions real rollouts contain: about one turn in 30 ends with a usage record whose usage is not in the cumulative total, followed by a `compacted` marker naming it and carrying a large replacement history. In real logs every marker names an earlier record. Reported by Cubic on ccusage#2061. * fix(ci): gate the perf job on preflight and keep its token read-only The old perf job depended on the pkg.pr.new publish job and so, through it, on `preflight` (secret scan and workflow lint); building from source dropped that dependency, letting a PR that fails the scan run the benchmark anyway. Depend on `preflight` directly. The job also ran PR-built binaries, the fixture generator and the harness with `pull-requests: write` and `issues: write`. Split it: `ccusage-perf` builds and measures with a read-only token and uploads the report, and `ccusage-perf-comment` downloads it and only runs the comment upsert with the write token. Reported by Cubic on ccusage#2061. * fix(perf): lock bench-history builds and fixture generation Runs sharing the cache could interleave writes: one build could replace `target/release/ccusage` in the shared cargo target before another run copied it into its commit's cache slot, and two runs could delete and regenerate the same fixture under each other. Build-and-copy now runs under a cache-wide file lock, and each fixture key under its own, both re-checking the cache once they hold the lock. A per-commit target dir would avoid the lock but rebuild every dependency for every ref. The lock is a FileChannel lock released by closing the channel, and a test runs three processes against it and checks their critical sections never overlap. Reported by Cubic on ccusage#2061. * fix(ci): drop the perf job's unreachable merge_group fallbacks The perf job only runs on `pull_request`, and ccusage#2062, which adds the `merge_group` trigger, keeps the perf comment PR-only, so the `merge_group.base_sha` and `github.sha` fallbacks and the comment promising merge_group support could never apply. Remove them. Reported by Cubic on ccusage#2061. * fix(perf): give bench-history a Codex fixture every compared ref reads The generated Codex fixture is mostly zstd-compressed rollouts, some in `archived_sessions/`. Releases before ccusage#1890 skip `.jsonl.zst` files (and before ccusage#1230 skip archived sessions), so a history run across that change compared an old ref reading a fraction of the input with a new ref reading all of it, and reported the difference as a CPU and RSS regression. The generator gains `--codex-layout legacy`, which writes the same rollouts uncompressed under `sessions/`. bench-history keeps the current layout when every ref is a commit containing ccusage#1890, and otherwise generates the legacy layout and says so at the top of the report; binary paths count as unknown. With a custom Codex fixture that holds `.zst` files and a ref that cannot read them, and for `--real` runs across the change, the report leads with a warning instead. The PR comparison always builds refs that read the current layout, so CI keeps compressed coverage. Reported by CodeRabbit on ccusage#2061. * fix(perf): check every Codex input class a compared ref may skip The compatibility check only looked for `.jsonl.zst` files, but releases before ccusage#1230 also skip `archived_sessions/`, so a custom fixture with archived plain rollouts got no warning and a history run across ccusage#1230 compared different input. Replace the single zstd commit with a table of Codex input classes taken from the Codex loader's history (`paths.rs`, `reader.rs`): archived sessions (a9b12c4, ccusage#1230, v20.0.7) and zstd-compressed rollouts (44a3aa8, ccusage#1890, v20.0.33). Each ref is checked per class by git ancestry; the generated fixture switches to the legacy layout when any ref skips any class, and a custom fixture or `--real` run warns per class with the refs that skip it. Claude discovery has scanned `projects/` recursively throughout v20 and the date-window skips (ccusage#1665, ccusage#1794) do not apply to unbounded reports, so they are not classes. Probing each ref's source for the reader code was rejected: the same strings appear in tests and comments. Reported by Cubic on ccusage#2061. * fix(ci): run the perf comment upsert from the base commit The comment job holds `pull-requests: write` and `issues: write`, but it checked out the PR merge commit and ran that tree's setup action and `upsert-pr-comment.nu`, so a PR that edited either ran them with the write token. Check out `pull_request.base.sha` instead, still without persisted credentials. The PR's report reaches the job only as an artifact the script reads as text. This PR does not change `upsert-pr-comment.nu`, and the base version reads the same inputs (COMMENT_FILE as raw markdown, COMMENT_MARKER, PR_NUMBER, GITHUB_REPOSITORY, GITHUB_TOKEN), so the first runs after merge need no fallback. The workflow file itself still comes from the PR on `pull_request` events, so this removes the PR's scripts from the write job rather than making the job a hard trust boundary. Reported by CodeRabbit on ccusage#2061. * fix(perf): match Codex input classes to the rollouts the loader reads The input-class matchers looked at absolute paths: any `.zst` file counted as a compressed rollout, and any path containing `/archived_sessions/`, including a parent of the fixture, counted as an archived session. Match paths relative to the Codex home against what the loader reads: `*.jsonl` and `*.jsonl.zst` (reader.rs) under `sessions/` and `archived_sessions/` (paths.rs). Tests cover non-rollout `.zst` files, rollouts outside those directories, non-rollout files in `archived_sessions/`, and a fixture under a directory named `archived_sessions`. Reported by Cubic on ccusage#2061.
1 parent 2f6d0b3 commit 86707db

14 files changed

Lines changed: 1595 additions & 743 deletions

File tree

‎.agents/skills/profile/references/rust.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -45,10 +45,14 @@ affect rendering — before believing a speedup.
4545
- Parallelism only when it improves end-to-end time on real fixture shapes.
4646
- Binary size when adding a dependency or feature (`rust-binary-size` skill).
4747

48-
## Reproducing The CI Performance Comment
48+
## Comparing Refs And Reproducing The CI Comment
4949

50-
`apps/ccusage/scripts/compare-pr-performance.bb --help` lists the options; the
51-
`compare-pr-performance.bb` step in `.github/workflows/ci.yaml` is the reference
52-
for a realistic argument set. Locally, point `--base-dir` at the main worktree
53-
and `--head-dir` at `$PWD`, and cut `--runs`/`--warmup` down to keep iteration
54-
fast.
50+
`just bench-history <ref|binary>...` builds each ref in release mode, runs every
51+
report command over the generated realistic fixture in interleaved rounds, and
52+
judges each ref against the previous one the same way the CI comment judges a
53+
PR against its merge base. Add `--real` to also measure the logs this
54+
environment points at. Judge by the CPU-time and peak-RSS ratios, not wall time.
55+
56+
When a regression slips past the comparison, fix the fixture generator
57+
(`apps/ccusage/scripts/generate-large-fixture.ts`) so its lines carry the shape
58+
that triggered it; do not add a special-case benchmark.

‎.github/workflows/ci.yaml‎

Lines changed: 77 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -489,97 +489,55 @@ jobs:
489489
}
490490
}
491491
492-
ccusage-perf-comment:
493-
name: ccusage ${{ matrix.label }}
492+
ccusage-perf:
493+
name: ccusage perf
494494
if: github.event_name == 'pull_request' && needs.changes.outputs.code-changed == 'true'
495-
needs:
496-
- changes
497-
- npm-publish-dry-run-and-upload-pkg-pr-now
495+
needs: [changes, tirith-scan, preflight]
496+
# Runs PR-built binaries and scripts, so it gets a read-only token; the
497+
# comment is posted by ccusage-perf-comment, which runs none of them.
498498
permissions:
499499
contents: read
500-
pull-requests: write
501-
issues: write
502500
runs-on: blacksmith-32vcpu-ubuntu-2404
503501
timeout-minutes: 30
504-
strategy:
505-
fail-fast: false
506-
matrix:
507-
include:
508-
# Same benchmark from two entry points, posted as two PR comments:
509-
# the JS launcher (default runtime) and the bundled Rust native binary.
510-
- runtime: js
511-
label: perf comment
512-
slug: perf-comment
513-
head-runtime-arg: ''
514-
- runtime: rust
515-
label: rust perf comment
516-
slug: rust-perf-comment
517-
head-runtime-arg: --head-runtime rust
502+
# The PR side is the checked-out merge commit (`github.sha`), built on top
503+
# of the base commit, so a branch behind its base is not credited with the
504+
# base's newer commits.
505+
env:
506+
BASE_SHA: ${{ github.event.pull_request.base.sha }}
507+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
518508

519509
steps:
520-
- name: Checkout PR
510+
- name: Checkout merge commit
521511
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
522512
with:
523-
ref: ${{ github.event.pull_request.head.sha }}
513+
fetch-depth: 0
524514
persist-credentials: false
525515

516+
# Base and PR are built here with the same derivation the Linux x64
517+
# native package ships, so that job's sticky disk already holds the
518+
# dependency artifacts both builds reuse.
526519
- name: Setup Nix
527520
uses: ./.github/actions/setup-nix-cache
528521
with:
529-
# This job only runs on pull requests, which never commit a sticky
530-
# disk, so both legs read test's instead: it installs `just` and the
531-
# same flake inputs on every push to main.
532-
disk-job: test
522+
disk-job: build-native-packages
523+
cache-scope: build-linux-x64
533524
save: ${{ env.PERSIST_CACHES }}
534-
# This job only benchmarks the prebuilt ccusage binary bundled in the
535-
# installed pkg.pr.new package (no Rust build happens here), so the full
536-
# `nix develop` dev shell is unnecessary. Install just the tools the perf
537-
# scripts invoke, pinned to the flake's nixpkgs via `--inputs-from .`.
525+
538526
- name: Install perf tooling
539527
run: nix profile install --inputs-from . nixpkgs#just
540528

541-
- name: Resolve PR preview package URL
542-
env:
543-
PKG_PR_NEW_URLS: ${{ needs.npm-publish-dry-run-and-upload-pkg-pr-now.outputs.urls }}
529+
- name: Check that the build sits on the base commit
544530
run: |
545-
preview_url="$(
546-
printf '%s\n' "$PKG_PR_NEW_URLS" \
547-
| tr '[:space:]' '\n' \
548-
| grep -E '(^|/)ccusage@' \
549-
| head -n 1
550-
)"
551-
if [ -z "$preview_url" ]; then
552-
echo "ccusage preview URL not found in: $PKG_PR_NEW_URLS" >&2
531+
if ! git merge-base --is-ancestor "$BASE_SHA" HEAD; then
532+
echo "::error::Base $BASE_SHA is not an ancestor of the checked-out commit $(git rev-parse HEAD)"
553533
exit 1
554534
fi
555-
echo "CCUSAGE_PREVIEW_URL=$preview_url" >> "$GITHUB_ENV"
556-
557-
# Pricing snapshot commits are pushed to main with GITHUB_TOKEN, which
558-
# triggers no CI, so they never get a preview package and the benchmark
559-
# would wait out its whole package timeout for one. They do not change
560-
# code, so walk back over them to the nearest commit that has a package.
561-
# Any other commit without one (e.g. its CI is still running) stops the
562-
# walk and keeps the real base, which is then waited for as before.
563-
- name: Resolve base preview package
564-
env:
565-
BASE_SHA: ${{ github.event.pull_request.base.sha }}
566-
REPOSITORY: ${{ github.repository }}
535+
536+
- name: Build base and PR binaries
567537
run: |
568-
git fetch --quiet --no-tags --depth=50 origin "$BASE_SHA"
569-
base="$BASE_SHA"
570-
for sha in $(git rev-list --first-parent -n 50 "$BASE_SHA"); do
571-
if curl --fail --head --location --silent --output /dev/null \
572-
"https://pkg.pr.new/$REPOSITORY/ccusage@$sha"; then
573-
base="$sha"
574-
break
575-
fi
576-
if git diff --name-only "$sha^" "$sha" \
577-
| grep -qvE '^(flake\.(lock|nix)|rust/crates/ccusage-pricing-snapshots/src/[^/]+\.json)$'; then
578-
break
579-
fi
580-
done
581-
echo "Comparing against base package ${base}"
582-
echo "BASE_PACKAGE_SHA=$base" >> "$GITHUB_ENV"
538+
NIX_CONFIG="access-tokens =" nix build .#ccusage-static --out-link "$RUNNER_TEMP/ccusage-head"
539+
NIX_CONFIG="access-tokens =" nix build "git+file://$PWD?rev=$BASE_SHA#ccusage-static" \
540+
--out-link "$RUNNER_TEMP/ccusage-base"
583541
584542
- name: Generate large ccusage fixture
585543
run: |
@@ -591,30 +549,60 @@ jobs:
591549
- name: Compare performance
592550
run: |
593551
apps/ccusage/scripts/compare-pr-performance.bb \
594-
--base-package-url "https://pkg.pr.new/${{ github.repository }}/ccusage@$BASE_PACKAGE_SHA" \
595-
--base-sha "$BASE_PACKAGE_SHA" \
596-
--head-dir . \
597-
--head-package-url "$CCUSAGE_PREVIEW_URL" \
598-
${{ matrix.head-runtime-arg }} \
599-
--fixture-dir apps/ccusage/test/fixtures/claude \
600-
--codex-fixture-dir apps/ccusage/test/fixtures/codex \
552+
--base-bin "$RUNNER_TEMP/ccusage-base/bin/ccusage" \
553+
--base-sha "$BASE_SHA" \
554+
--head-bin "$RUNNER_TEMP/ccusage-head/bin/ccusage" \
555+
--head-sha "$HEAD_SHA" \
556+
--fixture-dir "$RUNNER_TEMP/ccusage-large-fixture" \
557+
--codex-fixture-dir "$RUNNER_TEMP/ccusage-large-codex-fixture" \
601558
--warmup 2 \
602-
--runs 7 \
603-
--large-fixture-dir "$RUNNER_TEMP/ccusage-large-fixture" \
604-
--large-codex-fixture-dir "$RUNNER_TEMP/ccusage-large-codex-fixture" \
605-
--large-warmup 0 \
606-
--large-runs 1 \
607-
--package-runner-timeout-ms 300000 \
608-
--output "$RUNNER_TEMP/ccusage-${{ matrix.slug }}.md"
559+
--runs 11 \
560+
--output "$RUNNER_TEMP/ccusage-perf-comment.md"
609561
610562
- name: Write job summary
611-
run: cat "$RUNNER_TEMP/ccusage-${{ matrix.slug }}.md" >> "$GITHUB_STEP_SUMMARY"
563+
run: cat "$RUNNER_TEMP/ccusage-perf-comment.md" >> "$GITHUB_STEP_SUMMARY"
564+
565+
- name: Upload comment
566+
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
567+
with:
568+
name: ccusage-perf-comment
569+
path: ${{ runner.temp }}/ccusage-perf-comment.md
570+
if-no-files-found: error
571+
572+
ccusage-perf-comment:
573+
name: ccusage perf comment
574+
if: github.event_name == 'pull_request' && !github.event.pull_request.head.repo.fork
575+
needs: ccusage-perf
576+
permissions:
577+
contents: read
578+
pull-requests: write
579+
issues: write
580+
runs-on: blacksmith-32vcpu-ubuntu-2404
581+
timeout-minutes: 10
582+
steps:
583+
# The write token must not reach code the PR edited, so the action and
584+
# the upsert script come from the base commit; the downloaded report is
585+
# only passed to the script as text.
586+
- name: Checkout base commit
587+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
588+
with:
589+
ref: ${{ github.event.pull_request.base.sha }}
590+
persist-credentials: false
591+
592+
- uses: ./.github/actions/setup-nix-cache
593+
with:
594+
save: ${{ env.PERSIST_CACHES }}
595+
596+
- name: Download comment
597+
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
598+
with:
599+
name: ccusage-perf-comment
600+
path: ${{ runner.temp }}/ccusage-perf
612601

613602
- name: Upsert PR comment
614-
if: ${{ !github.event.pull_request.head.repo.fork }}
615603
env:
616-
COMMENT_FILE: ${{ runner.temp }}/ccusage-${{ matrix.slug }}.md
617-
COMMENT_MARKER: <!-- ccusage-${{ matrix.slug }}:${{ github.event.pull_request.head.sha }} -->
604+
COMMENT_FILE: ${{ runner.temp }}/ccusage-perf/ccusage-perf-comment.md
605+
COMMENT_MARKER: <!-- ccusage-perf-comment:${{ github.event.pull_request.head.sha }} -->
618606
GITHUB_TOKEN: ${{ github.token }}
619607
GITHUB_REPOSITORY: ${{ github.repository }}
620608
PR_NUMBER: ${{ github.event.pull_request.number }}
@@ -630,6 +618,7 @@ jobs:
630618
- build-native-packages
631619
- npm-publish-dry-run-and-upload-pkg-pr-now
632620
- ccusage-preview-e2e
621+
- ccusage-perf
633622
- ccusage-perf-comment
634623
if: ${{ always() }}
635624
runs-on: ubuntu-slim
@@ -651,6 +640,7 @@ jobs:
651640
- build-native-packages
652641
- npm-publish-dry-run-and-upload-pkg-pr-now
653642
- ccusage-preview-e2e
643+
- ccusage-perf
654644
- ccusage-perf-comment
655645
if: ${{ always() }}
656646
runs-on: ubuntu-slim
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env nix
2+
#! nix shell --inputs-from ../../.. nixpkgs#babashka-unwrapped nixpkgs#hyperfine nixpkgs#git --command bb
3+
4+
(require '[babashka.fs :as fs])
5+
6+
(def script-dir (fs/parent (fs/file *file*)))
7+
8+
(doseq [source ["data.clj" "system.clj" "benchmark.clj" "report.clj" "main.clj" "history.clj"]]
9+
(load-file (str (fs/path script-dir "compare_pr_performance" source))))
10+
11+
(when (= *file* (System/getProperty "babashka.file"))
12+
(try
13+
(apply history-main *command-line-args*)
14+
(catch Exception error
15+
(binding [*out* *err*]
16+
(println (ex-message error)))
17+
(System/exit 1))))
Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,5 @@
1-
declare namespace Bun {
2-
interface FileSink {
3-
write(data: string): Promise<number>;
4-
end(): Promise<void>;
5-
}
6-
7-
interface BunFile {
8-
writer(): FileSink;
9-
}
10-
}
11-
121
declare const Bun: {
13-
file(path: string): Bun.BunFile;
2+
write(destination: string, data: string | Uint8Array): Promise<number>;
3+
zstdCompressSync(data: string | Uint8Array): Uint8Array;
144
$(strings: TemplateStringsArray, ...values: readonly unknown[]): Promise<unknown>;
155
};

‎apps/ccusage/scripts/compare-pr-performance.bb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env nix
2-
#! nix shell --inputs-from ../../.. nixpkgs#babashka-unwrapped nixpkgs#bun nixpkgs#curl nixpkgs#git nixpkgs#hyperfine nixpkgs#nodejs ../../..#pnpm --command bb
2+
#! nix shell --inputs-from ../../.. nixpkgs#babashka-unwrapped nixpkgs#hyperfine --command bb
33

44
(require '[babashka.fs :as fs])
55

0 commit comments

Comments
 (0)