Repository navigation
perf(ci): replace coverage report with a plain Rust test run - #1273
Conversation
The nixified `cargo llvm-cov` coverage build recompiled the whole workspace with instrumentation (~230s) and produced a cobertura report nothing consumed (no badge, no docs reference). Drop coverage entirely and run the workspace tests instead via crane's `cargoTest`, which reuses the shared, non-instrumented `cargoArtifacts` and only compiles the test binaries. - nix/coverage.nix -> nix/tests.nix: `ccusage-tests` package built with `cargoTest` (doCheck enabled), source scoped to the Rust tree plus the few out-of-tree files the build/tests `include_str!`, so unrelated edits don't invalidate it. - CI test job: replace the coverage generate + upload steps with `nix build .#ccusage-tests`; drop the now-unused `code-quality` permission and the upload-code-coverage action. Verified locally: the derivation runs the suite (253 + 10 + 51 + 16 + 2 tests pass) rather than skipping it.
📝 WalkthroughWalkthroughThis PR transitions the Rust test and CI infrastructure from a coverage-focused build ( ChangesRust Test Build Transition
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No new issues found.
Big Pickle (free via Pullfrog for OSS) ｜ 𝕏
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runner startupExecution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one
Cached bunx execution performanceRuns the same large fixture through Fixtures: Claude
Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runner startupExecution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one
Cached bunx execution performanceRuns the same large fixture through Fixtures: Claude
Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
Summary
The nixified
cargo llvm-covcoverage build recompiled the whole workspace withinstrumentation (~230s on the arm runner) and produced a cobertura report that
nothing consumes (no badge, no docs reference). Drop coverage entirely and run
the workspace tests instead via crane's
cargoTest, which reuses the shared,non-instrumented
cargoArtifactsand only compiles the test binaries.What changed
ccusage-testspackage built withcargoTest(doCheckenabled). Source scoped to the Rust tree plus the fewout-of-tree files the build/tests
include_str!(flake.lock,ccusage.example.json, apps/ccusage/package.json) so unrelated edits don't bust
the cache.
nix build .#ccusage-tests; drop the now-unusedcode-qualitypermission andthe
upload-code-coverageaction.Notes
pass) rather than skipping it.
cargoArtifacts; afollow-up will add a GC root so the deps persist across runs (the same reason
the native build currently recompiles deps every run).
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.Summary by cubic
Replace the Rust coverage step with a plain test run in CI to speed up builds and remove an unused Cobertura report. Tests now run via
nix build .#ccusage-testsusingcrane’scargoTest, reusingcargoArtifacts.ccusage-coveragewithccusage-testsin nix/tests.nix, built withcargoTest(doCheckenabled); workspace tests must pass.code-qualitypermission; runnix build .#ccusage-tests.rustplusflake.lock,ccusage.example.json, andapps/ccusage/package.jsonto avoid cache busts from unrelated edits.Written for commit fccf571. Summary will update on new commits.
Summary by CodeRabbit