Repository navigation
Commit f5020ef
committed
perf(ccusage): parallelise JSONL loading by CPU count
Parallelise JSONL file loading across daily, session, and blocks reports using os.availableParallelism() by default. This keeps the worker count tied to the host CPU instead of a fixed cap, while --single-thread and CCUSAGE_JSONL_READ_CONCURRENCY provide deterministic overrides for benchmarking and constrained environments.
Small JSONL files are now read in one buffered pass, with the existing streaming path retained for files larger than 32 MiB. Cost calculation also keeps a synchronous fast path when --offline or cached costUSD values make async pricing unnecessary.
Dedupe is still applied after each file result is collected, preserving the original file-order replacement behaviour and keeping JS/Rust token totals identical for daily, session, monthly, and blocks JSON output.
Benchmark on Apple M3 Pro, Node 24.15.0, os.availableParallelism()=11, built JS CLI at apps/ccusage/dist/index.js, Rust release CLI at /Users/ryoppippi/ghq/github.com/ryoppippi/ccusage/target/release/ccusage, LOG_LEVEL=0, --offline --json, mitata min_samples=5:
JS: daily 3.51s, session 3.07s, monthly 3.17s, blocks 6.51s. Rust: daily 640.16ms, session 639.67ms, monthly 660.97ms, blocks 649.93ms. Previous JS PR baseline from bd7d2d4: daily 3.85s, session 3.99s, monthly 3.94s, blocks 8.36s.
Verified with pnpm run format, pnpm typecheck, pnpm --filter ccusage exec vitest run src/data-loader.ts, pnpm --filter ccusage run build, pnpm run test, and JS/Rust token-total parity checks for daily/session/monthly/blocks.1 parent bd7d2d4 commit f5020ef
5 files changed
Lines changed: 303 additions & 114 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
69 | 75 | | |
70 | 76 | | |
71 | 77 | | |
| |||
157 | 163 | | |
158 | 164 | | |
159 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
160 | 172 | | |
161 | 173 | | |
162 | 174 | | |
| |||
259 | 271 | | |
260 | 272 | | |
261 | 273 | | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
262 | 280 | | |
263 | 281 | | |
264 | 282 | | |
| |||
345 | 363 | | |
346 | 364 | | |
347 | 365 | | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
348 | 372 | | |
349 | 373 | | |
350 | 374 | | |
| |||
439 | 463 | | |
440 | 464 | | |
441 | 465 | | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
442 | 472 | | |
443 | 473 | | |
444 | 474 | | |
| |||
530 | 560 | | |
531 | 561 | | |
532 | 562 | | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
533 | 569 | | |
534 | 570 | | |
535 | 571 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
76 | 81 | | |
77 | 82 | | |
78 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
169 | 169 | | |
170 | 170 | | |
171 | 171 | | |
| 172 | + | |
172 | 173 | | |
173 | 174 | | |
174 | 175 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| 71 | + | |
71 | 72 | | |
72 | 73 | | |
73 | 74 | | |
| |||
0 commit comments