Repository navigation
test(pricing): stop the alias-precedence test depending on the LiteLLM snapshot - #2009
Conversation
…M snapshot `resolves_ultrafast_multiplier_through_model_aliases` loaded the embedded pricing map and expected `gpt-6.1-terra` to resolve to its own entry rather than through the `gpt-6.1-terra -> gpt-6.1-sol` alias. That only holds when the LiteLLM snapshot embedded at build time contains a top-level `gpt-6.1-terra` key. Snapshots pinned before that key existed, such as a dev shell whose `CCUSAGE_PRICING_JSON_PATH` still points at an older input, have no exact Terra entry, so the lookup follows the alias and reports Sol's 6.0 ultrafast multiplier instead of Terra's 1.0. CI always builds from the current pin, so it never saw the failure. The test now supplies both entries itself through `load_json`, which still applies the embedded speed-multiplier overrides, so the precedence rule it checks no longer depends on which snapshot the build embedded or on Terra's published rate.
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Local pricing and alias assertions rust/crates/ccusage-core/src/pricing.rs, rust/adapters/codex/src/lib.rs |
The tests load explicit Sol and Terra pricing. They check that private-sol resolves to Sol’s input rate and receives six times Sol’s standard cost in Auto mode. They also verify the exact Terra entry’s pricing relationship to Sol. The Codex fixture supplies rates for four models. |
Estimated code review effort: 2 (Simple) | ~10 minutes
Change: Other
Possibly related PRs
- ccusage/ccusage#1989: Adds Ultrafast pricing and alias resolution, including the test revised here.
Merge Risk: 
e0644
-
No actionable merge-blocking risk was identified in the pricing test changes.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly identifies the main change: making the pricing alias-precedence test independent of the embedded LiteLLM snapshot. It is concise and directly related to the changeset. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files.
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | e064417 | Commit Preview URL Branch Preview URL |
Nov 01 2026, 07:27 PM |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
ccusage
@ccusage/ccusage-darwin-arm64
@ccusage/ccusage-darwin-x64
@ccusage/ccusage-linux-arm64
@ccusage/ccusage-linux-x64
@ccusage/ccusage-win32-x64
commit: |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — reviewed the single-commit test isolation fix in this run.
- Hermetic alias-precedence fixture — builds a two-entry
PricingMapviaload_jsoninstead ofload_embedded, so the exact-match-wins check no longer depends on the build-time LiteLLM snapshot contents. - Preserved override path —
load_jsonstill applies the embedded speed-multiplier overrides, so the 6.0/1.0 multiplier assertions still exercise the intended production logic.
muse-spark-1.3-contributor (free via Pullfrog for OSS) ｜ 𝕏
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
…LLM snapshot Two codex adapter tests had the same dependency the previous commit removed from ccusage-core: `keeps_standard_pricing_for_ultrafast_without_a_published_rate` and `prices_ultrafast_from_the_same_entry_as_standard_rates` loaded the embedded pricing map and expected `gpt-6.1-terra` to have its own entry. A build whose `CCUSAGE_PRICING_JSON_PATH` points at a LiteLLM pin from before that key existed (such as dc0237cc) has no exact Terra entry, so the lookup follows the `gpt-6.1-terra -> gpt-6.1-sol` alias and bills Sol's 6.0 Ultrafast multiplier, failing both tests locally while CI stays green. Both tests now share a `load_json` map that supplies every model they look up, and the aliased Sol expectation is derived from those test rates times the published 6.0 multiplier instead of a hard-coded published dollar amount. The published-rate check stays with `prices_recorded_ultrafast_usage_at_published_rates`, which only reads models every pin contains.
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — reviewed the delta since the prior pullfrog review (36df9a1), which is the single new commit extending the snapshot isolation to the codex adapter tests.
- Hermetic codex Ultrafast fixture — added a shared
ultrafast_test_pricing()helper building a 4-entryPricingMapviaload_json, so both codex Ultrafast tests no longer depend on which LiteLLM snapshot the build embedded. - Derived alias assertion — replaced the hard-coded
0.01512expectation withprivate_sol == standard_sol * 6.0, keeping the published-multiplier check while removing the dependence on Terra's published dollar rate. - Terra exact-wins check preserved —
standard_terra == standard_sol / 2.0follows from the fixture rates (exactly half), and theAuto == Standardassertion still exercises the no-published-rate path.
load_json still applies the embedded speed-multiplier overrides, so the 6.0 multiplier comes from the committed overrides file rather than the snapshot. cargo fmt --check is clean on the adapter. Full test execution was not possible in this environment (no LiteLLM snapshot outside the Nix sandbox), but the change is test-only and the cost math was verified against report.rs by reading.
muse-spark-1.3-contributor (free via Pullfrog for OSS) ｜ 𝕏
Upstream tip: 30b653f (2026-11-06, chore(pricing): update models.dev snapshot) Previous fork base: c76874e (the upstream commit this fork was synced to) Synced 128 upstream commits on top of c76874e: 70 automated pricing snapshot refreshes and the following meaningful ones: b24e7a7 docs: remove LINE MAN sponsor logo (ccusage#1910) a93d2fe chore: release v20.0.38 (ccusage#1938) d91f296 docs: add ccx to community projects (ccusage#1943) aa7c675 chore: release v20.0.39 (ccusage#1944) 39fefa2 fix(cli): accept --start-of-week on top-level weekly (ccusage#1945) 600b3c9 chore: release v20.0.40 (ccusage#1946) baeb991 fix(antigravity): map model ID 1402 to claude-opus-5-5 (ccusage#1948) 06347d3 chore: release v20.0.41 (ccusage#1949) 5649632 test(hermes): derive DeepSeek provider pricing from embedded snapshot (ccusage#1951) d0847af chore(deps): update node.js to >=24.22.1 (ccusage#1950) 91dfe85 chore(deps): update pullfrog/pullfrog action to v0.1.100 (ccusage#1953) f335280 chore: release v20.0.42 (ccusage#1952) b86527d chore(deps): update rust crate rustls-webpki to 0.103.16 (ccusage#1954) 5c85ca2 chore(deps): update pullfrog/pullfrog action to v0.1.101 (ccusage#1958) f268798 fix(cli): apply source-specific config in unified reports (ccusage#1959) 37a51da chore: release v20.0.43 (ccusage#1955) e302366 chore(deps): update github/codeql-action action to v4.39.1 (ccusage#1960) 7af0db6 chore(deps): update pnpm to v12.15.0 (ccusage#1961) 730c336 chore(deps): update rust crate ureq to 3.5.0 (ccusage#1964) bd1a21a fix(release): publish platform packages before the wrapper (ccusage#1965) ca33842 chore: release v20.0.44 (ccusage#1962) b2c8f8b chore: remove unused npm registry gate test (ccusage#1966) 19e71d0 chore(deps): update songmu/tagpr action to v1.21.4 (ccusage#1969) 2a84385 fix(codex): count repeated first-response snapshots once (ccusage#1971) 60227f2 chore(deps): update rust crate smallvec to 1.16.4 (ccusage#1973) 3e31edd chore(deps): update hustcer/setup-nu action to v3.29 (ccusage#1974) d93c260 feat(pricing): add gpt-6.1-terra fast/flex multipliers (ccusage#1975) 154e13b chore: release v20.0.45 (ccusage#1967) f1b50e6 chore(deps): update pullfrog/pullfrog action to v0.1.102 (ccusage#1980) 4f1f470 perf: speed up CI and local builds (ccusage#1984) 0bd3d19 refactor(rust): move binary-only report helpers out of ccusage-core (ccusage#1985) 65c5837 fix(antigravity): skip unreadable conversation databases (ccusage#1992) 6372cac chore(deps): update dependency vitepress to v2.0.0-alpha.21 (ccusage#1994) 9a42066 chore(deps): update node.js to >=24.22.2 (ccusage#1995) 87fa788 chore(deps): update pnpm to v12.16.0 (ccusage#1996) 8cb2532 fix(codex): price recorded Ultrafast requests at Ultrafast rates (ccusage#1989) 128d6a0 chore(deps): update pnpm to v12.16.1 (ccusage#1997) 5aa9d3f chore: release v20.0.46 (ccusage#1982) 4712880 chore(deps): update rust crate unicode-width to 0.2.3 (ccusage#1998) 3d59863 chore(deps): update node.js to v26 (ccusage#1999) 96e98c6 chore(deps): update dependency @types/node to v26.6.6 (ccusage#2002) 60fac54 chore(deps): update pullfrog/pullfrog action to v0.1.103 (ccusage#2003) de83d19 fix(config): let space-form command sections override generic report sections (ccusage#2006) 72b8ad4 fix(all): count OpenCode Claude Agent turns once in unified reports (ccusage#2007) 7db7a30 test(pricing): stop the alias-precedence test depending on the LiteLLM snapshot (ccusage#2009) e7a7ee4 fix(all): keep sessions last active on the --until day (ccusage#2008) 4a3da73 chore: release v20.0.47 (ccusage#2001) 8e56c11 chore(deps): update pullfrog/pullfrog action to v0.1.104 (ccusage#2010) e2eddf7 chore(deps): update pullfrog/pullfrog action to v0.1.107 (ccusage#2012) e3d2dcb chore(deps): update pullfrog/pullfrog action to v0.1.108 (ccusage#2014) 8995f3b chore: release v20.0.48 (ccusage#2011) 3977fd0 chore(deps): update pullfrog/pullfrog action to v0.1.110 (ccusage#2017) 546ae51 fix(nix): make `nix flake show` work again on nixpkgs 26.11 (ccusage#2019) 1b5e8cc chore(deps): update kesin11/actions-timeline action to v3.2.1 (ccusage#1983) 23c5d62 chore(deps): update rust crate jiff to 0.2.39 (ccusage#2024) 7b7cec0 chore(deps): update rust crate ureq to 3.5.1 (ccusage#2025) a1e4a03 chore(deps): update pullfrog/pullfrog action to v0.1.111 (ccusage#2028) e84d142 chore(deps): update pnpm to v12.17.0 (ccusage#2029) Fork wins kept: the SQLite on-disk cache with its per-namespace ledger, --live-only and clear-cache, the pi adapter with replay suppression and per-path namespaces, the OpenCode V2 loader (now wrapped as LinkedEntry), the FxHash/Cow hot paths, parallel per-file loads with their single_thread seam, the table renderer, the fork CI and tag-only release workflow, and the pricing freshness, refresh and offline precompute paths. Hand-resolved conflicts: rust/adapters/claude/src/lib.rs: upstream's DedupedEntries/pruned dedupe lands beside the fork's collapse_replays and requestless replay rules; load_daily_summaries routes through load_daily_summaries_with_detection. rust/adapters/opencode/{parser,loader}.rs: upstream's V2 session_message parsing is wrapped as the fork's LinkedEntry, and the native Claude session link is read from the payload so it survives a cache hit. rust/adapters/antigravity/src/loader.rs: upstream's rewrite is taken, keeping the fork's per-database outcome reporting and file order. rust/crates/ccusage-core/src/pricing.rs: the fork's OnceLock embedded loader sits beside upstream's models.dev fallback and alias resolution. rust/crates/ccusage/src/main.rs: upstream dispatch plus the fork's ClearCache command. rust/crates/ccusage-core/src/cache.rs: the ledger is now generic over a LedgerPayload trait, so it covers plain entries and OpenCode's entry-plus-link pairs alike. rust/Cargo.toml: the fork's exact `=` pins stay; jiff 0.2.39, smallvec 1.16.4, ureq 3.5.1, unicode-width 0.2.3, rustls-webpki 0.103.16. .github/workflows/{ci,release}.yaml: the fork's design is kept. packages/ccusage-*/package.json (6 files) and rust/adapters/claude/src/daily.rs: stay deleted. rust/crates/ccusage-core/src/{home,last_window}.rs: upstream's crate split moved them to ccusage-cli and ccusage-report, with their snapshots. apps/ccusage/config-schema.json: regenerated from the merged Rust source (32 numeric minimums `0.0` -> `0`, new pi description), plus three snapshots regenerated in the fork layout for the codex and qwen tests. Fixes made while landing this sync, folded in so the merge stands green: fix(rust): decode cache blobs written before `data_session_id`. postcard encodes struct fields positionally, so a field added mid-struct made every blob the released 20.0.36 binary wrote undecodable. A failed decode is not merely slow: it drops the ledger rows of deleted source files, which no later run can rebuild. In an isolated fixture the released binary still reported 101,570,002 pi tokens after the source files were deleted; a warm build without the decoders reported 0. LegacyCachedEntry and LegacyLedgerEntry decode the old layout and reconstruct `data_session_id` as None; decode_exact requires the whole buffer, so a newer blob can no longer "decode" into an older struct as garbage. Four regression tests cover both layouts and the tail rule. fix(rust): warn once per table when a cached blob decodes as no known layout. The files table is rebuildable; the ledger table is not, so the warning says to save a copy before clear-cache. test(opencode): isolate the Claude-agent pairing test from the developer's real cache; it had been writing 'agent'/'plain' ledger rows into the opencode-db namespace. Version: take upstream's 20.0.48 in package.json, apps/ccusage/package.json and docs/package.json; the Rust workspace keeps version 0.0.0. DO NOT push a v20.0.48 tag from this fork: release.yaml would publish fork bytes under upstream identity.
What was wrong
resolves_ultrafast_multiplier_through_model_aliasespassed in CI but failed on a local machine: thegpt-6.1-terralookup returned Sol's ultrafast multiplier (6.0) instead of Terra's (1.0).Root cause
The test loaded the embedded pricing map and relied on the build-time LiteLLM snapshot containing a top-level
gpt-6.1-terrakey. The snapshot comes fromCCUSAGE_PRICING_JSON_PATH, which the Nix dev shell sets from the pinnedlitellminput. A dev shell whose environment still pointed at an older pin (one without that key) had no exact Terra entry, so the lookup followed thegpt-6.1-terra -> gpt-6.1-solalias. CI always builds from the current pin, so it never saw this.Fix
The test now supplies both entries itself via
load_json, which still applies the embedded speed-multiplier overrides. The alias-precedence rule it checks no longer depends on which snapshot the build embedded or on Terra's published rate.How tested
CCUSAGE_PRICING_JSON_PATHpointing at the older snapshot; the rewritten test passes against both the old and current snapshots.cargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings,TZ=UTC cargo test --workspacenix build .#checks.<system>.ccusage-hawk,nix build .#ccusage-tests,nix fmtSummary by cubic
Makes the Ultrafast pricing tests deterministic by supplying the Sol and Terra model entries directly instead of loading the embedded pricing map. The fix now covers both the
ccusage-corealias-precedence test and the two codex adapter tests, which share aload_jsonmap with every model they look up.The tests previously relied on the build-time LiteLLM snapshot containing a top-level
gpt-6.1-terrakey. Dev shells pinned to older snapshots lacked that key, so the lookup followed thegpt-6.1-terra -> gpt-6.1-solalias and reported Sol's multiplier instead of Terra's. CI always builds from the current pin, so it never caught the failure. The aliased Sol expectation is now derived from the test rates times the published 6.0 multiplier instead of a hard-coded dollar amount, and the published-rate check stays inprices_recorded_ultrafast_usage_at_published_rates.Written for commit e064417. Summary will update on new commits.
Summary by CodeRabbit
private-solalias uses Sol’s input rate and configured multiplier, while exact Terra entries retain their expected rates and multiplier behavior, including in Auto mode. Also confirmed Terra’s standard rate is half Sol’s.