Skip to content

Commit b7948ab

Browse files
authored
perf(pricing): shrink embedded snapshots 58% and binary 2.9% (#1244)
* docs(skills): add Rust binary size guidance Add a repo-local rust-binary-size skill for release-profile, dependency feature, native package size, and executable bloat work. The skill captures stable min-sized-rust guidance, keeps aggressive nightly and packer techniques opt-in, and links back to the upstream reference so future agents can inspect the source material when size work recurs. Register the skill in AGENTS.md so agents discover it before changing Rust size-sensitive code or packaging. * perf(pricing): flatten embedded models.dev snapshot Regenerate the committed models.dev pricing snapshot as a flat map keyed by runtime model id instead of preserving upstream provider nesting. The runtime loader now accepts both the live provider-tree shape from models.dev and the flat embedded shape, so network fallback compatibility is unchanged while the checked-in snapshot shrinks from 97,159 bytes to 36,871 bytes. The release binary rebuilt successfully after the change at 2,845,920 bytes. Pricing tests cover the flat embedded shape and the existing offline fallback behaviour. * fix(pricing): validate models.dev snapshot shape Replace the untagged models.dev snapshot parser with explicit top-level shape validation so malformed provider payloads cannot be accepted as an empty flat model map. The regression test uses a fixture-backed malformed provider payload and confirms parsing fails without loading entries. This addresses CodeRabbit feedback on PR #1244. Validation: direnv exec . cargo test --manifest-path rust/Cargo.toml -p ccusage pricing::tests:: -- --nocapture; direnv exec . just fmt * perf(pricing): compact embedded LiteLLM snapshot Emit the build-time LiteLLM snapshot with short field keys and teach the runtime loader to read both the compact embedded shape and the normal LiteLLM response shape. The generated LiteLLM snapshot shrinks from 66,046 bytes to 31,949 bytes while keeping 401 model keys. Runtime-relevant fields compared equal after expanding the compact keys. Together with the models.dev flattening, the local release binary is now 2,812,864 bytes versus 2,895,424 bytes on the base commit. Validation: direnv exec . cargo test --manifest-path rust/Cargo.toml -p ccusage pricing::tests:: -- --nocapture; direnv exec . cargo build --manifest-path rust/Cargo.toml --release --bin ccusage; direnv exec . just fmt * fix(pricing): harden compact snapshot parsing Align the LiteLLM compact-shape detector with the build-time emitter by requiring both short price keys before compact decoding. If compact decoding fails, the parser now falls back to the full LiteLLM shape so unrelated metadata keys do not drop otherwise valid pricing rows. Also make the models.dev snapshot generator ignore empty catalog ids and fall back to the source model key. This prevents a malformed empty id from becoming the embedded pricing key while keeping the committed snapshot format unchanged. * fix(package): rewrite Darwin libiconv install name The macOS native package was copying a Nix-built binary that still referenced libiconv inside /nix/store. That works inside a Nix closure but fails when the binary is installed from the npm preview package on a clean GitHub macOS runner. Patch the staged Darwin binary with install_name_tool so the package uses /usr/lib/libiconv.2.dylib at runtime. This keeps the Nix build unchanged while making the published native package runnable outside Nix. * test(pricing): cover compact fallback branches Extend the LiteLLM fallback regression so it enters the compact-shape branch and then falls through to full-shape parsing when compact decoding fails. Also cover the undefined and non-empty catalog id branches for the models.dev compact key helper, matching the generator inputs called out during review. * ci(perf): skip unavailable base package benchmarks Perf comment jobs use pkg.pr.new URLs for both the base and PR package. The base URL can be unavailable for a commit that was never published by the preview package workflow, which previously made the script wait for the full job timeout and fail before writing a comment. Make package installation return undefined when the URL is not ready, emit a commit-specific skipped performance comment when no base-dir fallback exists, and lower the workflow wait to five minutes so the job has time to finish cleanly. * fix(pricing): warn on duplicate models.dev keys The compact models.dev generator keeps the first entry for a pricing key, but duplicate provider aliases were previously skipped without any signal. Cubic identified that this could hide future pricing drift when regenerating the snapshot. Emit a warning before skipping a duplicate pricing key and cover the warning text with a focused Vitest case so future duplicate handling remains visible. * ci(github): tolerate unauthenticated comment writes The rust perf comment job can produce a skipped benchmark summary when the base pkg.pr.new package is unavailable, but creating the new PR comment failed with a 401 from the workflow token. That made the job fail even though the benchmark script completed and wrote the job summary. Treat 401 the same as the existing 403 no-comment-permission path, keep other GitHub failures fatal, and add a small in-source Vitest project for this script so the auth classification is covered.
1 parent c408f64 commit b7948ab

13 files changed

Lines changed: 3017 additions & 5304 deletions

File tree

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
---
2+
name: rust-binary-size
3+
description: Guides Rust binary size reduction for ccusage. Use when changing release profiles, dependency features, native packaging size, or investigating Rust executable bloat.
4+
paths:
5+
- 'rust/Cargo.toml'
6+
- 'rust/**/*.rs'
7+
- 'rust/**/*.toml'
8+
- 'apps/ccusage/scripts/**'
9+
globs: 'rust/**/*.rs,rust/**/*.toml,apps/ccusage/scripts/**'
10+
---
11+
12+
# Rust Binary Size
13+
14+
Use this skill when applying binary-size guidance to the Rust-first `ccusage`
15+
CLI, release profiles, native package binaries, dependency features, or size
16+
regression investigations.
17+
18+
Primary external reference: <https://github.com/johnthagen/min-sized-rust>
19+
20+
## Baseline
21+
22+
Start by checking the existing release profile before adding new size settings:
23+
24+
```sh
25+
sed -n '1,120p' rust/Cargo.toml
26+
```
27+
28+
This workspace should keep stable release settings aligned with
29+
`min-sized-rust` unless a measured tradeoff argues otherwise:
30+
31+
- `opt-level = "z"` for workspace crates unless benchmarking shows `"s"` is smaller.
32+
- `lto = "fat"` for maximum cross-crate dead-code elimination in release builds.
33+
- `codegen-units = 1` for better release optimization.
34+
- `panic = "abort"` only for binaries where stack unwinding is not required.
35+
- `strip = "symbols"` or an equivalent stable strip setting for packaged binaries.
36+
37+
## Investigation
38+
39+
Prefer measurement before changing code or dependencies:
40+
41+
```sh
42+
direnv exec . cargo build --manifest-path rust/Cargo.toml --release --bin ccusage
43+
ls -lh rust/target/release/ccusage
44+
```
45+
46+
When a size regression is not explained by the release profile, inspect
47+
dependency features and large symbols before editing:
48+
49+
```sh
50+
direnv exec . cargo tree --manifest-path rust/Cargo.toml -e features -p ccusage
51+
direnv exec . cargo bloat --manifest-path rust/Cargo.toml --release --bin ccusage --crates
52+
```
53+
54+
If `cargo bloat` is unavailable, use the `missing-tools` skill or run it through
55+
the Nix/dev-shell path preferred by the repository. Do not add a new tool to the
56+
flake unless repeated project work needs it.
57+
58+
## Safe Changes
59+
60+
Prefer stable, low-risk changes first:
61+
62+
- Disable unnecessary dependency default features when tests prove the disabled
63+
features are not required.
64+
- Narrow optional dependency features instead of replacing a well-fitting crate.
65+
- Remove unused code paths, generated assets, or format-heavy diagnostics from
66+
release-only paths only when behavior remains correct.
67+
- Keep `ccusage` functionality, JSON output, table output, and packaging
68+
semantics unchanged unless the user explicitly asks for a behavior change.
69+
70+
## Risky Changes
71+
72+
Treat these as opt-in only unless the user explicitly asks for an aggressive
73+
minimum-size experiment:
74+
75+
- Nightly-only flags such as `-Zlocation-detail`, `-Zfmt-debug`,
76+
`panic=immediate-abort`, or `build-std`.
77+
- `#![no_std]`, `#![no_main]`, C entry points, or manual stdio.
78+
- UPX or other binary packers, especially for distributed CLI artifacts.
79+
- Dynamic Rust linking through `prefer-dynamic`, because deployment and ABI
80+
constraints make it unsuitable for ordinary CLI releases.
81+
82+
## Validation
83+
84+
After changes, run formatting and the relevant behavioral checks:
85+
86+
```sh
87+
direnv exec . just fmt
88+
direnv exec . just test
89+
```
90+
91+
For release-profile or packaging changes, also build the native CLI and compare
92+
the binary size with the previous measurement. Record the command and result in
93+
the PR body or review reply.

‎.github/scripts/upsert-pr-comment.ts‎

Lines changed: 50 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -52,23 +52,18 @@ async function githubRequest<T>(path: string, options: RequestInit = {}): Promis
5252
return response.json() as Promise<T>;
5353
}
5454

55-
const repository = requiredEnv('GITHUB_REPOSITORY');
56-
const prNumber = requiredEnv('PR_NUMBER');
57-
const marker = requiredEnv('COMMENT_MARKER');
58-
const body = await Bun.file(requiredEnv('COMMENT_FILE')).text();
59-
60-
async function createComment(): Promise<void> {
55+
async function createComment(repository: string, prNumber: string, body: string): Promise<void> {
6156
await githubRequest(`/repos/${repository}/issues/${prNumber}/comments`, {
6257
method: 'POST',
6358
body: JSON.stringify({ body }),
6459
});
6560
}
6661

67-
async function tryCreateComment(): Promise<void> {
62+
async function tryCreateComment(repository: string, prNumber: string, body: string): Promise<void> {
6863
try {
69-
await createComment();
64+
await createComment(repository, prNumber, body);
7065
} catch (error) {
71-
if (error instanceof GitHubRequestError && error.status === 403) {
66+
if (isCommentWriteAuthFailure(error)) {
7267
console.warn(
7368
`Skipping PR comment because GitHub token cannot write comments: ${error.message}`,
7469
);
@@ -78,24 +73,52 @@ async function tryCreateComment(): Promise<void> {
7873
}
7974
}
8075

81-
const comments = await githubRequest<GitHubComment[]>(
82-
`/repos/${repository}/issues/${prNumber}/comments?per_page=100`,
83-
);
84-
const existing = comments.find(
85-
(comment) => comment.user?.login === 'github-actions[bot]' && comment.body?.includes(marker),
86-
);
76+
function isCommentWriteAuthFailure(error: unknown): error is GitHubRequestError {
77+
return error instanceof GitHubRequestError && (error.status === 401 || error.status === 403);
78+
}
8779

88-
if (existing == null) {
89-
await tryCreateComment();
90-
} else {
91-
try {
92-
await githubRequest(`/repos/${repository}/issues/comments/${existing.id}`, {
93-
method: 'PATCH',
94-
body: JSON.stringify({ body }),
95-
});
96-
} catch (error) {
97-
console.warn(`Failed to update existing PR comment; creating a new comment instead.`);
98-
console.warn(error);
99-
await tryCreateComment();
80+
async function main(): Promise<void> {
81+
const repository = requiredEnv('GITHUB_REPOSITORY');
82+
const prNumber = requiredEnv('PR_NUMBER');
83+
const marker = requiredEnv('COMMENT_MARKER');
84+
const body = await Bun.file(requiredEnv('COMMENT_FILE')).text();
85+
const comments = await githubRequest<GitHubComment[]>(
86+
`/repos/${repository}/issues/${prNumber}/comments?per_page=100`,
87+
);
88+
const existing = comments.find(
89+
(comment) => comment.user?.login === 'github-actions[bot]' && comment.body?.includes(marker),
90+
);
91+
92+
if (existing == null) {
93+
await tryCreateComment(repository, prNumber, body);
94+
} else {
95+
try {
96+
await githubRequest(`/repos/${repository}/issues/comments/${existing.id}`, {
97+
method: 'PATCH',
98+
body: JSON.stringify({ body }),
99+
});
100+
} catch (error) {
101+
console.warn(`Failed to update existing PR comment; creating a new comment instead.`);
102+
console.warn(error);
103+
await tryCreateComment(repository, prNumber, body);
104+
}
100105
}
101106
}
107+
108+
if (import.meta.vitest != null) {
109+
describe(isCommentWriteAuthFailure, () => {
110+
it('treats unauthenticated comment writes as skippable', () => {
111+
expect(isCommentWriteAuthFailure(new GitHubRequestError('unauthenticated', 401))).toBe(true);
112+
});
113+
114+
it('treats forbidden comment writes as skippable', () => {
115+
expect(isCommentWriteAuthFailure(new GitHubRequestError('forbidden', 403))).toBe(true);
116+
});
117+
118+
it('keeps other GitHub errors fatal', () => {
119+
expect(isCommentWriteAuthFailure(new GitHubRequestError('server error', 500))).toBe(false);
120+
});
121+
});
122+
} else {
123+
await main();
124+
}

‎.github/workflows/ci.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -253,7 +253,7 @@ jobs:
253253
--large-codex-fixture-dir "$RUNNER_TEMP/ccusage-large-codex-fixture" \
254254
--large-warmup 0 \
255255
--large-runs 1 \
256-
--package-runner-timeout-ms 1800000 \
256+
--package-runner-timeout-ms 300000 \
257257
--output "$RUNNER_TEMP/ccusage-perf-comment.md"
258258
259259
- name: Write job summary
@@ -325,7 +325,7 @@ jobs:
325325
--large-codex-fixture-dir "$RUNNER_TEMP/ccusage-large-codex-fixture" \
326326
--large-warmup 0 \
327327
--large-runs 1 \
328-
--package-runner-timeout-ms 1800000 \
328+
--package-runner-timeout-ms 300000 \
329329
--output "$RUNNER_TEMP/ccusage-rust-perf-comment.md"
330330
331331
- name: Write job summary

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Use these skills before working in this repository:
1616
- `skill-creator` - repo-local skill creation, SKILL.md frontmatter, description trigger quality, and reference layout.
1717
- `ast-grep` - structural code searches in Rust or TypeScript and AST-based migration verification with the dev-shell `ast-grep` CLI.
1818
- `bun-api-reference` - local Bun runtime API docs and type references under `node_modules/bun-types`.
19+
- `rust-binary-size` - Rust release profile, dependency feature, native packaging size, and executable bloat reduction guidance based on min-sized-rust.
1920
- `tdd` - Red-Green-Refactor workflow for logic changes.
2021
- `reduce-similarities` - AST-based duplicate Rust code detection with similarity-rs; TypeScript duplication checks use `typescript` and `ast-grep`.
2122
- `cmux-debug` - terminal UI and responsive table verification in cmux.

‎apps/ccusage/scripts/compare-pr-performance.ts‎

Lines changed: 91 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,11 @@ type PackageRunnerComparison = {
5454
head?: PackageRunnerMeasurement;
5555
};
5656

57+
type PackageInstall = {
58+
acquisition: number;
59+
binEntry: string;
60+
};
61+
5762
type SampleOptions = {
5863
runs: number;
5964
warmup: number;
@@ -402,10 +407,11 @@ async function installPackageUrl({
402407
label: string;
403408
packageUrl: string;
404409
timeoutMs: number;
405-
}): Promise<{ acquisition: number; binEntry: string }> {
410+
}): Promise<PackageInstall | undefined> {
406411
await writeProgress(`${label} package install waiting for package URL`);
407412
if (!(await waitForPackageUrl(packageUrl, timeoutMs))) {
408-
throw new Error(`${label} package URL was not ready: ${packageUrl}`);
413+
await writeProgress(`${label} package install skipped because package URL was not ready`);
414+
return undefined;
409415
}
410416
await mkdir(installDir, { recursive: true });
411417
await Bun.write(
@@ -1695,6 +1701,38 @@ function renderMarkdown(
16951701
return `${lines.join('\n')}\n`;
16961702
}
16971703

1704+
function renderSkippedMarkdown(options: {
1705+
basePackageUrl?: string;
1706+
baseSha?: string;
1707+
headRuntime: HeadRuntime;
1708+
headSha?: string;
1709+
reason: string;
1710+
}): string {
1711+
const markerName =
1712+
options.headRuntime === 'rust' ? 'ccusage-rust-perf-comment' : 'ccusage-perf-comment';
1713+
const lines = [
1714+
`<!-- ${markerName} -->`,
1715+
...(options.headSha == null ? [] : [`<!-- ${markerName}:${options.headSha} -->`]),
1716+
'## ccusage performance comparison',
1717+
'',
1718+
...(options.headSha == null
1719+
? []
1720+
: [
1721+
`PR SHA: \`${formatSha(options.headSha)}\``,
1722+
...(options.baseSha == null ? [] : [`Base SHA: \`${formatSha(options.baseSha)}\``]),
1723+
'',
1724+
]),
1725+
'Performance comparison skipped.',
1726+
'',
1727+
options.reason,
1728+
...(options.basePackageUrl == null
1729+
? []
1730+
: ['', `Base package: \`${packageUrlSha(options.basePackageUrl)}\``]),
1731+
'',
1732+
];
1733+
return `${lines.join('\n')}\n`;
1734+
}
1735+
16981736
if (import.meta.vitest != null) {
16991737
describe(createCcusageCommandFromBin, () => {
17001738
it('builds hyperfine command text that benchmarks the published ccusage bin with both Claude and Codex fixture environment variables', () => {
@@ -1777,6 +1815,26 @@ if (import.meta.vitest != null) {
17771815
});
17781816

17791817
describe(createHeadCcusageCommand, () => {
1818+
describe(installPackageUrl, () => {
1819+
it('skips package installation when the package URL is not ready before the timeout', async () => {
1820+
await using fixture = await createFixture({});
1821+
vi.stubGlobal('Bun', {
1822+
sleep: vi.fn(),
1823+
stderr: {},
1824+
write: vi.fn(async () => undefined),
1825+
});
1826+
1827+
await expect(
1828+
installPackageUrl({
1829+
installDir: join(fixture.path, 'package'),
1830+
label: 'base',
1831+
packageUrl: 'https://pkg.pr.new/ryoppippi/ccusage/ccusage@missing',
1832+
timeoutMs: 0,
1833+
}),
1834+
).resolves.toBeUndefined();
1835+
});
1836+
});
1837+
17801838
it('resolves the installed package published bin instead of the package manager shim', async () => {
17811839
await using fixture = await createFixture({});
17821840
const packageDir = join(fixture.path, 'node_modules', 'ccusage');
@@ -2191,6 +2249,22 @@ if (import.meta.vitest != null) {
21912249
'| installed native package binary | 2.00 KiB | 1.00 KiB | -1.00 KiB | 2.00x |',
21922250
);
21932251
});
2252+
2253+
it('renders a commit-specific skip comment when the base package URL is unavailable', () => {
2254+
const markdown = renderSkippedMarkdown({
2255+
basePackageUrl: 'https://pkg.pr.new/ryoppippi/ccusage/ccusage@0123456789abcdef',
2256+
baseSha: '0123456789abcdef',
2257+
headRuntime: 'rust',
2258+
headSha: 'abcdef0123456789',
2259+
reason: 'Base package URL was not ready before the timeout.',
2260+
});
2261+
2262+
expect(markdown).toContain('<!-- ccusage-rust-perf-comment:abcdef0123456789 -->');
2263+
expect(markdown).toContain('PR SHA: `abcdef012345`');
2264+
expect(markdown).toContain('Base SHA: `0123456789ab`');
2265+
expect(markdown).toContain('Base package: `0123456789ab`');
2266+
expect(markdown).toContain('Base package URL was not ready before the timeout.');
2267+
});
21942268
});
21952269

21962270
describe(parsePeakRssBytes, () => {
@@ -2348,6 +2422,21 @@ const command = define({
23482422
packageUrl: headPackageUrl,
23492423
timeoutMs: ctx.values.packageRunnerTimeoutMs,
23502424
});
2425+
if (basePackageUrl != null && basePackageInstall == null && baseDir == null) {
2426+
const markdown = renderSkippedMarkdown({
2427+
basePackageUrl,
2428+
baseSha: ctx.values.baseSha,
2429+
headRuntime,
2430+
headSha: gitSha(resolve(ctx.values.headDir)),
2431+
reason: `Base package URL was not ready before ${formatDuration(ctx.values.packageRunnerTimeoutMs)}. Fixture performance comparison requires a base package when --base-dir is not provided.`,
2432+
});
2433+
if (ctx.values.output == null) {
2434+
await Bun.write(Bun.stdout, markdown);
2435+
} else {
2436+
await Bun.write(resolve(ctx.values.output), markdown);
2437+
}
2438+
return;
2439+
}
23512440
const headNativeBinEntry =
23522441
headPackageInstall == null
23532442
? undefined

‎apps/ccusage/scripts/stage-native-package.mjs‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { execFileSync } from 'node:child_process';
12
import { chmodSync, copyFileSync, mkdirSync } from 'node:fs';
23
import { dirname, resolve } from 'node:path';
34
import process from 'node:process';
@@ -24,6 +25,21 @@ function readOption(name, fallback) {
2425
return value;
2526
}
2627

28+
function rewriteDarwinSystemLibraries(binaryPath) {
29+
const linkedLibraries = execFileSync('otool', ['-L', binaryPath], { encoding: 'utf8' });
30+
for (const line of linkedLibraries.split('\n')) {
31+
const library = line.trim().split(/\s+/)[0];
32+
if (/^\/nix\/store\/[^/]+-libiconv-[^/]+\/lib\/libiconv\.2\.dylib$/.test(library)) {
33+
execFileSync('install_name_tool', [
34+
'-change',
35+
library,
36+
'/usr/lib/libiconv.2.dylib',
37+
binaryPath,
38+
]);
39+
}
40+
}
41+
}
42+
2743
const scriptDir = dirname(fileURLToPath(import.meta.url));
2844
const repoRoot = resolve(scriptDir, '../../..');
2945
const platform = readOption('platform', process.platform);
@@ -45,5 +61,8 @@ copyFileSync(source, target);
4561
if (platform !== 'win32') {
4662
chmodSync(target, 0o755);
4763
}
64+
if (platform === 'darwin') {
65+
rewriteDarwinSystemLibraries(target);
66+
}
4867

4968
process.stdout.write(`${target}\n`);

0 commit comments

Comments
 (0)