Repository navigation
test(hermes): derive DeepSeek provider pricing from embedded snapshot - #1951
Conversation
The precedence regression pinned deepseek/deepseek-v4-flash at the old $0.14/M models.dev rate, so the scheduled models.dev refresh failed once upstream repriced the retired alias to V4.1 Flash ($0.15/M). Compare against the embedded provider entry instead and assert it differs from the raw model's timestamp-aware schedule so the precedence check stays meaningful.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Description Check |  |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | |
The title clearly and concisely describes the main change: updating the Hermes test to derive DeepSeek provider pricing from the embedded snapshot. |
| Docstring Coverage | |
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
✨ Finishing Touches
-
📝 Generate docstrings
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | 5ecdff1 | Commit Preview URL Branch Preview URL |
Oct 23 2026, 08:02 AM |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
No issues found across 1 file
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack
View guided diff | Turn on auto-fix | Re-trigger cubic
ccusage
@ccusage/ccusage-darwin-arm64
@ccusage/ccusage-darwin-x64
@ccusage/ccusage-linux-arm64
@ccusage/ccusage-linux-x64
@ccusage/ccusage-win32-x64
commit: |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
ℹ️ No critical issues — one test-robustness nit inline.
Reviewed changes
Reviewed the Hermes DeepSeek pricing precedence test fix in this run. Changes reviewed:
- Derive expected cost from snapshot — replace the hard-coded
0.14with the embeddeddeepseek/deepseek-v4-flashinput rate times 1M tokens, so upstream reprices no longer break the refresh workflow. - Guard precedence meaningfully — assert the provider-qualified cost differs from the raw timestamp-aware model cost before asserting Hermes prefers the provider entry.
muse-spark-1.3-contributor (free via Pullfrog for OSS) ｜ 𝕏
| assert!((calculate_hermes_cost(&entry, &pricing) - 0.14).abs() < 1e-12); | ||
| let provider_cost = pricing | ||
| .find_exact_with_fallback("deepseek/deepseek-v4-flash") | ||
| .unwrap() |
There was a problem hiding this comment.
Consider .expect("embedded snapshot must contain deepseek/deepseek-v4-flash") so a future snapshot regeneration that drops or renames this key fails with context instead of a bare unwrap panic.
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
Upstream tip: 30b653f (2026-11-06, chore(pricing): update models.dev snapshot) Previous fork base: c76874e (the upstream commit this fork was synced to) Synced 128 upstream commits on top of c76874e: 70 automated pricing snapshot refreshes and the following meaningful ones: b24e7a7 docs: remove LINE MAN sponsor logo (ccusage#1910) a93d2fe chore: release v20.0.38 (ccusage#1938) d91f296 docs: add ccx to community projects (ccusage#1943) aa7c675 chore: release v20.0.39 (ccusage#1944) 39fefa2 fix(cli): accept --start-of-week on top-level weekly (ccusage#1945) 600b3c9 chore: release v20.0.40 (ccusage#1946) baeb991 fix(antigravity): map model ID 1402 to claude-opus-5-5 (ccusage#1948) 06347d3 chore: release v20.0.41 (ccusage#1949) 5649632 test(hermes): derive DeepSeek provider pricing from embedded snapshot (ccusage#1951) d0847af chore(deps): update node.js to >=24.22.1 (ccusage#1950) 91dfe85 chore(deps): update pullfrog/pullfrog action to v0.1.100 (ccusage#1953) f335280 chore: release v20.0.42 (ccusage#1952) b86527d chore(deps): update rust crate rustls-webpki to 0.103.16 (ccusage#1954) 5c85ca2 chore(deps): update pullfrog/pullfrog action to v0.1.101 (ccusage#1958) f268798 fix(cli): apply source-specific config in unified reports (ccusage#1959) 37a51da chore: release v20.0.43 (ccusage#1955) e302366 chore(deps): update github/codeql-action action to v4.39.1 (ccusage#1960) 7af0db6 chore(deps): update pnpm to v12.15.0 (ccusage#1961) 730c336 chore(deps): update rust crate ureq to 3.5.0 (ccusage#1964) bd1a21a fix(release): publish platform packages before the wrapper (ccusage#1965) ca33842 chore: release v20.0.44 (ccusage#1962) b2c8f8b chore: remove unused npm registry gate test (ccusage#1966) 19e71d0 chore(deps): update songmu/tagpr action to v1.21.4 (ccusage#1969) 2a84385 fix(codex): count repeated first-response snapshots once (ccusage#1971) 60227f2 chore(deps): update rust crate smallvec to 1.16.4 (ccusage#1973) 3e31edd chore(deps): update hustcer/setup-nu action to v3.29 (ccusage#1974) d93c260 feat(pricing): add gpt-6.1-terra fast/flex multipliers (ccusage#1975) 154e13b chore: release v20.0.45 (ccusage#1967) f1b50e6 chore(deps): update pullfrog/pullfrog action to v0.1.102 (ccusage#1980) 4f1f470 perf: speed up CI and local builds (ccusage#1984) 0bd3d19 refactor(rust): move binary-only report helpers out of ccusage-core (ccusage#1985) 65c5837 fix(antigravity): skip unreadable conversation databases (ccusage#1992) 6372cac chore(deps): update dependency vitepress to v2.0.0-alpha.21 (ccusage#1994) 9a42066 chore(deps): update node.js to >=24.22.2 (ccusage#1995) 87fa788 chore(deps): update pnpm to v12.16.0 (ccusage#1996) 8cb2532 fix(codex): price recorded Ultrafast requests at Ultrafast rates (ccusage#1989) 128d6a0 chore(deps): update pnpm to v12.16.1 (ccusage#1997) 5aa9d3f chore: release v20.0.46 (ccusage#1982) 4712880 chore(deps): update rust crate unicode-width to 0.2.3 (ccusage#1998) 3d59863 chore(deps): update node.js to v26 (ccusage#1999) 96e98c6 chore(deps): update dependency @types/node to v26.6.6 (ccusage#2002) 60fac54 chore(deps): update pullfrog/pullfrog action to v0.1.103 (ccusage#2003) de83d19 fix(config): let space-form command sections override generic report sections (ccusage#2006) 72b8ad4 fix(all): count OpenCode Claude Agent turns once in unified reports (ccusage#2007) 7db7a30 test(pricing): stop the alias-precedence test depending on the LiteLLM snapshot (ccusage#2009) e7a7ee4 fix(all): keep sessions last active on the --until day (ccusage#2008) 4a3da73 chore: release v20.0.47 (ccusage#2001) 8e56c11 chore(deps): update pullfrog/pullfrog action to v0.1.104 (ccusage#2010) e2eddf7 chore(deps): update pullfrog/pullfrog action to v0.1.107 (ccusage#2012) e3d2dcb chore(deps): update pullfrog/pullfrog action to v0.1.108 (ccusage#2014) 8995f3b chore: release v20.0.48 (ccusage#2011) 3977fd0 chore(deps): update pullfrog/pullfrog action to v0.1.110 (ccusage#2017) 546ae51 fix(nix): make `nix flake show` work again on nixpkgs 26.11 (ccusage#2019) 1b5e8cc chore(deps): update kesin11/actions-timeline action to v3.2.1 (ccusage#1983) 23c5d62 chore(deps): update rust crate jiff to 0.2.39 (ccusage#2024) 7b7cec0 chore(deps): update rust crate ureq to 3.5.1 (ccusage#2025) a1e4a03 chore(deps): update pullfrog/pullfrog action to v0.1.111 (ccusage#2028) e84d142 chore(deps): update pnpm to v12.17.0 (ccusage#2029) Fork wins kept: the SQLite on-disk cache with its per-namespace ledger, --live-only and clear-cache, the pi adapter with replay suppression and per-path namespaces, the OpenCode V2 loader (now wrapped as LinkedEntry), the FxHash/Cow hot paths, parallel per-file loads with their single_thread seam, the table renderer, the fork CI and tag-only release workflow, and the pricing freshness, refresh and offline precompute paths. Hand-resolved conflicts: rust/adapters/claude/src/lib.rs: upstream's DedupedEntries/pruned dedupe lands beside the fork's collapse_replays and requestless replay rules; load_daily_summaries routes through load_daily_summaries_with_detection. rust/adapters/opencode/{parser,loader}.rs: upstream's V2 session_message parsing is wrapped as the fork's LinkedEntry, and the native Claude session link is read from the payload so it survives a cache hit. rust/adapters/antigravity/src/loader.rs: upstream's rewrite is taken, keeping the fork's per-database outcome reporting and file order. rust/crates/ccusage-core/src/pricing.rs: the fork's OnceLock embedded loader sits beside upstream's models.dev fallback and alias resolution. rust/crates/ccusage/src/main.rs: upstream dispatch plus the fork's ClearCache command. rust/crates/ccusage-core/src/cache.rs: the ledger is now generic over a LedgerPayload trait, so it covers plain entries and OpenCode's entry-plus-link pairs alike. rust/Cargo.toml: the fork's exact `=` pins stay; jiff 0.2.39, smallvec 1.16.4, ureq 3.5.1, unicode-width 0.2.3, rustls-webpki 0.103.16. .github/workflows/{ci,release}.yaml: the fork's design is kept. packages/ccusage-*/package.json (6 files) and rust/adapters/claude/src/daily.rs: stay deleted. rust/crates/ccusage-core/src/{home,last_window}.rs: upstream's crate split moved them to ccusage-cli and ccusage-report, with their snapshots. apps/ccusage/config-schema.json: regenerated from the merged Rust source (32 numeric minimums `0.0` -> `0`, new pi description), plus three snapshots regenerated in the fork layout for the codex and qwen tests. Fixes made while landing this sync, folded in so the merge stands green: fix(rust): decode cache blobs written before `data_session_id`. postcard encodes struct fields positionally, so a field added mid-struct made every blob the released 20.0.36 binary wrote undecodable. A failed decode is not merely slow: it drops the ledger rows of deleted source files, which no later run can rebuild. In an isolated fixture the released binary still reported 101,570,002 pi tokens after the source files were deleted; a warm build without the decoders reported 0. LegacyCachedEntry and LegacyLedgerEntry decode the old layout and reconstruct `data_session_id` as None; decode_exact requires the whole buffer, so a newer blob can no longer "decode" into an older struct as garbage. Four regression tests cover both layouts and the tail rule. fix(rust): warn once per table when a cached blob decodes as no known layout. The files table is rebuildable; the ledger table is not, so the warning says to save a copy before clear-cache. test(opencode): isolate the Claude-agent pairing test from the developer's real cache; it had been writing 'agent'/'plain' ledger rows into the opencode-db namespace. Version: take upstream's 20.0.48 in package.json, apps/ccusage/package.json and docs/package.json; the Rust workspace keeps version 0.0.0. DO NOT push a v20.0.48 tag from this fork: release.yaml would publish fork bytes under upstream identity.
Summary
The scheduled
update pricingworkflow failed on themodels-devleg (run 39895553989) inccusage-adapter-hermes:Root cause
Upstream models.dev (anomalyco/models.dev@d24949e2fe) repriced the retired
deepseek/deepseek-v4-flashalias to the DeepSeek V4.1 Flash rate. Input went from $0.14/M to $0.15/M and output from $0.28/M to $0.60/M. This is a legitimate price update. The test checks that Hermes prefers the exact provider-qualified models.dev entry over the raw model's built-in timestamp-aware DeepSeek schedule ($0.44/M at the Monday peak timestamp used). But it hard-coded the upstream $0.14 value, so any upstream reprice breaks the refresh.Change
deepseek/deepseek-v4-flashentry instead of a literal.Validation
nix build .#ccusage-testspasses with the snapshot currently onmain.nix build .#ccusage-testsalso passes with a freshly regenerated snapshot (nix flake update models-dev && just gen-models-dev-pricing, which bumps models.dev13c21bdtoa051b3c).cargo clippy -p ccusage-adapter-hermes --all-targets -D warningsand treefmt are clean.I also spot-checked the regenerated snapshot for schema-2 parsing gaps. Of the 112 keys it removes, only 7 still exist in the live models.dev API: six are case-renames that come back under lowercase keys, and one has no price. That matches upstream, so the generator needs no fix.
Summary by cubic
Fixes the Hermes pricing precedence regression test so the scheduled models.dev pricing refresh no longer fails when upstream reprices retired aliases.
The test pinned
deepseek/deepseek-v4-flashat the old $0.14/M models.dev rate, which broke once upstream repriced the retired alias to the V4.1 Flash rate. It now derives the expected cost from the embedded snapshot entry instead of a literal, and asserts that this cost differs from the raw model's timestamp-aware schedule so the precedence check stays meaningful. The snapshot itself is unchanged; the pricing workflow will commit the refreshed snapshot on its next run.Written for commit 5ecdff1. Summary will update on new commits.
Summary by CodeRabbit