Skip to content

Commit de01939

Browse files
committed
Prep a dispatch-only CodSpeed job alongside pytest-benchmark
Adds pytest-codspeed to the benchmark extra; it recognizes pytest-benchmark's benchmark fixture, so tests/test_benchmarks.py needs no changes to run under either. Verified locally: a --codspeed run disables pytest-benchmark's own instrumentation for that run rather than double-measuring. The new codspeed job runs the same benchmark through CodSpeed's action, in simulation mode. It's workflow_dispatch-only: CodSpeed's action authenticates against a project for this repo that doesn't exist until an org admin installs CodSpeed's GitHub App and connects it, so push or pull_request would just fail today. Once connected, push and pull_request can join the trigger -- unlike the pytest-benchmark job, CodSpeed doesn't need gh-pages write access, so it can safely run on every PR, including from forks. Issue xqlsystems#298.
1 parent 82fa0c5 commit de01939

3 files changed

Lines changed: 138 additions & 5 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 58 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -124,9 +124,7 @@ jobs:
124124

125125
benchmarks:
126126
# tests/test_benchmarks.py, tracked over time with pytest-benchmark
127-
# (issue #298). pytest-codspeed, which counts instructions instead
128-
# of wall time, could join as a second job later without changing
129-
# the benchmark itself (both use a fixture named `benchmark`).
127+
# (issue #298).
130128
name: "benchmarks"
131129
runs-on: ubuntu-latest
132130
# Only on main: a PR from a fork has no permission to push to
@@ -181,3 +179,60 @@ jobs:
181179
alert-threshold: "200%"
182180
comment-on-alert: true
183181
fail-on-alert: false
182+
183+
codspeed:
184+
# The same benchmark as the job above, run by CodSpeed
185+
# (https://codspeed.io/), which counts instructions rather than
186+
# wall time, so it doesn't need repeats to average out a hosted
187+
# runner's noise. It can also report on pull requests, including
188+
# from forks, since it needs no write access to this repository.
189+
#
190+
# Dispatch-only for now: CodSpeedHQ/action authenticates against a
191+
# CodSpeed project for this repo, which doesn't exist until an org
192+
# admin installs CodSpeed's GitHub App on xqlsystems/xarray-sql and
193+
# connects it (https://codspeed.io/docs). Until then, `push` or
194+
# `pull_request` would just fail. Once it's connected, add `push`
195+
# and `pull_request` to the `if:` below (matching the `benchmarks`
196+
# job isn't required: CodSpeed's own token doesn't need gh-pages's
197+
# write access, so it can run on every PR, not just main).
198+
name: "codspeed"
199+
runs-on: ubuntu-latest
200+
if: github.event_name == 'workflow_dispatch'
201+
permissions:
202+
contents: read
203+
id-token: write
204+
steps:
205+
- uses: actions/checkout@v4
206+
207+
- uses: dtolnay/rust-toolchain@stable
208+
209+
- name: Setup sccache
210+
uses: mozilla-actions/[email protected]
211+
212+
- name: Configure sccache
213+
run: |
214+
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
215+
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
216+
217+
- name: Cache Cargo
218+
uses: actions/cache@v4
219+
with:
220+
path: ~/.cargo
221+
key: cargo-cache-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }}
222+
223+
- uses: astral-sh/setup-uv@v5
224+
with:
225+
python-version: "3.12"
226+
enable-cache: true
227+
228+
- name: Install xarray_sql
229+
run: uv sync --locked --dev --extra benchmark --no-install-package xarray-sql
230+
- name: build rust
231+
run: uv run --no-project maturin develop --uv
232+
- name: Run benchmarks
233+
uses: CodSpeedHQ/action@v5
234+
env:
235+
VIRTUAL_ENV: ${{ github.workspace }}/.venv
236+
with:
237+
mode: simulation
238+
run: uv run --no-project pytest tests/test_benchmarks.py -m benchmark --codspeed

‎pyproject.toml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,15 @@ ml = [
6565
"ddxdb>=0.4.6,<0.5; sys_platform != 'emscripten'",
6666
]
6767
# Benchmarks tracked over time in CI (see tests/test_benchmarks.py).
68-
# pytest-codspeed uses the same `benchmark` fixture name, so a benchmark
69-
# written against pytest-benchmark runs unmodified under either.
68+
# pytest-codspeed recognizes pytest-benchmark's `benchmark` fixture, so
69+
# a benchmark written against pytest-benchmark runs unmodified under
70+
# either. Installed together, a --codspeed run disables pytest-
71+
# benchmark's own instrumentation for that run rather than double-
72+
# measuring; CodSpeed's CI job (.github/workflows/ci.yml) is
73+
# dispatch-only until its GitHub App is installed on the repo.
7074
benchmark = [
7175
"pytest-benchmark>=4.0",
76+
"pytest-codspeed>=3.0",
7277
]
7378
test = [
7479
"adbc-driver-flightsql>=1.12",

‎uv.lock‎

Lines changed: 73 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)