Skip to content

Repository navigation

Commit b184bdc

Browse files
ryoppippiryoppippi
authored
ci(pricing): land pricing updates on main instead of opening PRs (#1509)
* ci(pricing): land pricing updates on main instead of opening PRs The hourly pricing refresh opened a PR with github.token, which cannot trigger ci.yaml, so the PR arrived with no checks and nothing merged it. Pricing data was therefore only as fresh as the last time someone merged that PR by hand, even though the workflow already accepts a workflow_dispatch. Both jobs now run the checks that a pull request would have run - flake check, Rust tests and the Node tests, in a parallel step - and push the result straight to main. `just gen-models-dev-pricing` already runs before validation, so the committed snapshots never lag the locked revision. The commit and rebase-onto-main dance is shared by both jobs, so it lives in a composite action. A concurrency group keeps a manual dispatch from racing the hourly run's push. * refactor(pricing): move the lock update logic into nushell scripts The two update steps were shell pipelines around jq, curl and cmp writing JSON to /tmp. Nushell handles the same work as structured data: the lock file is read with `open`, the pricing document is fetched with `http get` and compared as a record, so no temporary files or byte-level formatting assumptions are involved. The scripts carry the repository's Nix shebang, so they need no extra nushell setup step in the workflow and stay runnable by hand. models.dev now decides through a `match` over {snapshots, lock}, which names the three states the step can end in - snapshots moved, only the lock moved, nothing moved - instead of inferring them from consecutive `git diff` calls. * ci(pricing): run the full check set for both pricing sources The validation legs only covered flake check, the Rust tests and the Node tests, so the Babashka harness test from ci.yaml's test job was missing. Now that the push lands on main without any pull request checks behind it, the workflow has to run what a PR would have run: `nix flake check` already covers the preflight job's treefmt and gitleaks checks, and the remaining legs match the test job leg for leg. The two sources were separate jobs with a duplicated leg list, which is how the Babashka leg went missing in the first place. They are a matrix now, so the checks are written once and both sources cannot drift apart. `max-parallel: 1` preserves the sequential push the previous `needs:` ordering gave.
1 parent 61f80bc commit b184bdc

4 files changed

Lines changed: 161 additions & 120 deletions

File tree

‎.github/actions/commit-and-push-main/action.yaml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: Commit and push to main
2+
description: >
3+
Commit the given paths as github-actions[bot] and push them to main, rebasing
4+
onto whatever main has become while the caller was validating. Callers must
5+
check out with fetch-depth: 0, because a shallow clone has no merge base to
6+
rebase onto, and must only run this when the paths actually changed.
7+
inputs:
8+
message:
9+
description: Commit message
10+
required: true
11+
paths:
12+
description: Whitespace-separated list of paths to commit
13+
required: true
14+
runs:
15+
using: composite
16+
steps:
17+
- name: Commit and push to main
18+
shell: bash
19+
env:
20+
GH_TOKEN: ${{ github.token }}
21+
MESSAGE: ${{ inputs.message }}
22+
PATHS: ${{ inputs.paths }}
23+
run: |
24+
git config user.name "github-actions[bot]"
25+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
26+
gh auth setup-git
27+
28+
# PATHS is word-split on purpose: it carries a list of paths.
29+
# shellcheck disable=SC2086
30+
git commit --message "$MESSAGE" -- $PATHS
31+
32+
git fetch origin main
33+
if ! git rebase origin/main; then
34+
git rebase --abort
35+
echo 'main moved with a conflicting change; the next run will retry.'
36+
exit 1
37+
fi
38+
git push origin HEAD:main

‎.github/scripts/update-litellm-lock.nu‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#!/usr/bin/env nix
2+
#! nix shell --inputs-from ../.. nixpkgs#nushell nixpkgs#git --command nu
3+
4+
# Bump the pinned LiteLLM input and report whether it moved the pricing JSON that
5+
# the build embeds, so the workflow can skip validating lock-only churn.
6+
def main [] {
7+
let before = (locked-pricing)
8+
^nix flake update litellm
9+
10+
let changed = match ((locked-pricing) == $before) {
11+
false => true
12+
true => {
13+
print 'LiteLLM pricing JSON is unchanged; dropping the lock-only bump.'
14+
^git checkout -- flake.lock
15+
false
16+
}
17+
}
18+
19+
$"changed=($changed)(char nl)" | save --append $env.GITHUB_OUTPUT
20+
}
21+
22+
# Fetch model_prices_and_context_window.json at the revision flake.lock pins.
23+
def locked-pricing []: nothing -> record {
24+
let rev = open --raw flake.lock | from json | get nodes.litellm.locked.rev
25+
http get $"https://raw.githubusercontent.com/BerriAI/litellm/($rev)/model_prices_and_context_window.json"
26+
}

‎.github/scripts/update-models-dev-lock.nu‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/usr/bin/env nix
2+
#! nix shell --inputs-from ../.. nixpkgs#nushell nixpkgs#git --command nu
3+
4+
const SNAPSHOTS = [
5+
'rust/crates/ccusage-core/src/models-dev-pricing.json'
6+
'rust/crates/ccusage-adapter-codex/src/codex-auto-review-fallbacks.json'
7+
]
8+
9+
# Bump the pinned models.dev input, regenerate the committed snapshots, and report
10+
# whether anything the build embeds actually moved.
11+
def main [] {
12+
^nix flake update models-dev
13+
^nix develop --command just gen-models-dev-pricing
14+
15+
let state = {
16+
snapshots: (dirty ...$SNAPSHOTS)
17+
lock: (dirty flake.lock)
18+
}
19+
let changed = match $state {
20+
{snapshots: true} => true
21+
{snapshots: false, lock: true} => {
22+
print 'models.dev pricing snapshots are unchanged; dropping the lock-only bump.'
23+
^git checkout -- flake.lock ...$SNAPSHOTS
24+
false
25+
}
26+
_ => false
27+
}
28+
29+
$"changed=($changed)(char nl)" | save --append $env.GITHUB_OUTPUT
30+
}
31+
32+
# Whether git reports tracked changes for the given paths.
33+
def dirty [...paths: string]: nothing -> bool {
34+
(^git diff --quiet -- ...$paths | complete).exit_code != 0
35+
}

‎.github/workflows/update-pricing.yaml‎

Lines changed: 62 additions & 120 deletions
Original file line numberDiff line numberDiff line change
@@ -7,135 +7,77 @@ on:
77

88
permissions: {}
99

10+
# Both legs push to main, so a manual dispatch landing on top of the hourly
11+
# schedule would otherwise race the other run's push.
12+
concurrency:
13+
group: update-pricing
14+
cancel-in-progress: false
15+
1016
jobs:
1117
update-pricing:
18+
name: ${{ matrix.name }}
1219
runs-on: blacksmith-32vcpu-ubuntu-2404
20+
timeout-minutes: 60
1321
permissions:
1422
contents: write
15-
pull-requests: write
23+
strategy:
24+
fail-fast: false
25+
# The legs push to the same branch, so they must not race each other's
26+
# rebase onto main.
27+
max-parallel: 1
28+
matrix:
29+
include:
30+
- name: litellm
31+
script: .github/scripts/update-litellm-lock.nu
32+
message: 'chore(pricing): update LiteLLM snapshot'
33+
paths: flake.lock
34+
- name: models-dev
35+
script: .github/scripts/update-models-dev-lock.nu
36+
message: 'chore(pricing): update models.dev snapshot'
37+
paths: >-
38+
flake.lock
39+
rust/crates/ccusage-core/src/models-dev-pricing.json
40+
rust/crates/ccusage-adapter-codex/src/codex-auto-review-fallbacks.json
1641
steps:
1742
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
1843
with:
44+
fetch-depth: 0
1945
persist-credentials: false
2046
- uses: ./.github/actions/setup-nix-cache
21-
- name: Capture current LiteLLM pricing JSON
22-
run: |
23-
rev="$(jq -r '.nodes.litellm.locked.rev' flake.lock)"
24-
curl --fail --location --silent --show-error \
25-
"https://raw.githubusercontent.com/BerriAI/litellm/${rev}/model_prices_and_context_window.json" \
26-
--output /tmp/litellm-pricing-before.json
27-
- name: Update LiteLLM pricing input
28-
run: |
29-
nix flake update litellm
30-
- name: Skip lock-only LiteLLM updates
31-
run: |
32-
rev="$(jq -r '.nodes.litellm.locked.rev' flake.lock)"
33-
curl --fail --location --silent --show-error \
34-
"https://raw.githubusercontent.com/BerriAI/litellm/${rev}/model_prices_and_context_window.json" \
35-
--output /tmp/litellm-pricing-after.json
36-
37-
if cmp --silent /tmp/litellm-pricing-before.json /tmp/litellm-pricing-after.json; then
38-
echo "LiteLLM pricing JSON is unchanged; skipping PR."
39-
git checkout -- flake.lock
40-
fi
41-
- name: Validate updated pricing snapshot
42-
run: |
43-
if git diff --quiet; then
44-
echo "Pricing snapshot is already up to date."
45-
exit 0
46-
fi
47-
nix develop --command just check
48-
- name: Create pull request
49-
env:
50-
GH_TOKEN: ${{ github.token }}
51-
run: |
52-
if git diff --quiet; then
53-
echo "Pricing snapshot is already up to date."
54-
exit 0
55-
fi
56-
57-
git config user.name "github-actions[bot]"
58-
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
59-
gh auth setup-git
60-
61-
branch="automation/litellm-pricing"
62-
git checkout -B "$branch"
63-
git add flake.lock
64-
git commit -m "chore(pricing): update LiteLLM snapshot"
65-
git push --force origin "$branch"
66-
67-
body="$(printf '%s\n' \
68-
'Updates the locked LiteLLM pricing input used by Nix builds.' \
69-
'' \
70-
'Validation:' \
71-
'- just check')"
72-
73-
if gh pr view "$branch" >/dev/null 2>&1; then
74-
printf '%s\n' "$body" | gh pr edit "$branch" --title "chore(pricing): update LiteLLM snapshot" --body-file -
75-
else
76-
printf '%s\n' "$body" | gh pr create --base main --head "$branch" --title "chore(pricing): update LiteLLM snapshot" --body-file -
77-
fi
78-
79-
update-models-dev-pricing:
80-
needs: update-pricing
81-
if: ${{ always() }}
82-
runs-on: blacksmith-32vcpu-ubuntu-2404
83-
permissions:
84-
contents: write
85-
pull-requests: write
86-
steps:
87-
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
8847
with:
89-
persist-credentials: false
90-
- uses: ./.github/actions/setup-nix-cache
91-
- name: Update models.dev input
92-
run: |
93-
nix flake update models-dev
94-
- name: Regenerate committed models.dev pricing snapshot
95-
run: |
96-
nix develop --command just gen-models-dev-pricing
97-
- name: Skip lock-only models.dev updates
98-
run: |
99-
# Only the compacted snapshot affects the build; if a models.dev bump
100-
# leaves embedded pricing unchanged, drop the lock-only churn.
101-
if git diff --quiet -- rust/crates/ccusage-core/src/models-dev-pricing.json rust/crates/ccusage-adapter-codex/src/codex-auto-review-fallbacks.json; then
102-
echo "models.dev pricing snapshots are unchanged; skipping PR."
103-
git checkout -- flake.lock rust/crates/ccusage-core/src/models-dev-pricing.json rust/crates/ccusage-adapter-codex/src/codex-auto-review-fallbacks.json
104-
fi
105-
- name: Validate updated pricing snapshot
106-
run: |
107-
if git diff --quiet; then
108-
echo "Pricing snapshot is already up to date."
109-
exit 0
110-
fi
111-
nix develop --command just check
112-
- name: Create pull request
48+
cache-scope: ${{ matrix.name }}
49+
- name: Update pricing input
50+
id: update
11351
env:
114-
GH_TOKEN: ${{ github.token }}
115-
run: |
116-
if git diff --quiet; then
117-
echo "Pricing snapshot is already up to date."
118-
exit 0
119-
fi
120-
121-
git config user.name "github-actions[bot]"
122-
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
123-
gh auth setup-git
124-
125-
branch="automation/models-dev-pricing"
126-
git checkout -B "$branch"
127-
git add flake.lock rust/crates/ccusage-core/src/models-dev-pricing.json rust/crates/ccusage-adapter-codex/src/codex-auto-review-fallbacks.json
128-
git commit -m "chore(pricing): update models.dev snapshot"
129-
git push --force origin "$branch"
130-
131-
body="$(printf '%s\n' \
132-
'Updates the pinned models.dev input and its committed pricing snapshot.' \
133-
'' \
134-
'Validation:' \
135-
'- just check')"
136-
137-
if gh pr view "$branch" >/dev/null 2>&1; then
138-
printf '%s\n' "$body" | gh pr edit "$branch" --title "chore(pricing): update models.dev snapshot" --body-file -
139-
else
140-
printf '%s\n' "$body" | gh pr create --base main --head "$branch" --title "chore(pricing): update models.dev snapshot" --body-file -
141-
fi
52+
SCRIPT: ${{ matrix.script }}
53+
run: $SCRIPT
54+
# The push below cannot trigger ci.yaml, so the checks a pull request would
55+
# have run happen here instead: `nix flake check` covers the preflight job's
56+
# treefmt and gitleaks checks, and the remaining legs mirror the test job.
57+
# Each leg no-ops when the update turned out to be lock-only churn.
58+
- parallel:
59+
- name: Run nix flake check
60+
run: |
61+
git diff --quiet && exit 0
62+
nix flake check --print-build-logs
63+
- name: Run Rust tests
64+
run: |
65+
git diff --quiet && exit 0
66+
nix build .#ccusage-tests --no-link --print-build-logs
67+
- name: JS test
68+
run: |
69+
git diff --quiet && exit 0
70+
nix profile install --inputs-from . nixpkgs#pnpm nixpkgs#nodejs nixpkgs#just
71+
pnpm install --frozen-lockfile
72+
mkdir -p "$HOME/.claude/projects"
73+
mkdir -p "$HOME/.config/claude/projects"
74+
just test-node
75+
- name: Babashka performance harness test
76+
run: |
77+
git diff --quiet && exit 0
78+
apps/ccusage/scripts/compare-pr-performance_test.clj
79+
- uses: ./.github/actions/commit-and-push-main
80+
if: steps.update.outputs.changed == 'true'
81+
with:
82+
message: ${{ matrix.message }}
83+
paths: ${{ matrix.paths }}

0 commit comments

Comments
 (0)