Skip to content

Commit de2bb28

Browse files
alxmrsclaude
andcommitted
Run the ML demos in CI, and say how to contribute one
tests/test_ml_demos.py runs every demo at its smallest size, each checked against JAX, and a new ml demos CI job installs JAX to run it. benchmarks/ml/CONTRIBUTING.md says what a demo is made of, how to write its SQL, and what to do when ddx or an engine can't run it. The performance work is tracked in #285. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h
1 parent 27e332d commit de2bb28

4 files changed

Lines changed: 224 additions & 6 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,3 +82,45 @@ jobs:
8282
env:
8383
VIRTUAL_ENV: ${{ github.workspace }}/.venv
8484
run: uv run --no-project pytest -v . -m "not integration"
85+
86+
ml-demos:
87+
# Each ML demo in benchmarks/ml, at its smallest size, checked against
88+
# JAX (tests/test_ml_demos.py).
89+
name: "ml demos"
90+
runs-on: ubuntu-latest
91+
steps:
92+
- uses: actions/checkout@v4
93+
94+
- uses: dtolnay/rust-toolchain@stable
95+
96+
- name: Setup sccache
97+
uses: mozilla-actions/[email protected]
98+
99+
- name: Configure sccache
100+
run: |
101+
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
102+
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
103+
104+
- name: Cache Cargo
105+
uses: actions/cache@v4
106+
with:
107+
path: ~/.cargo
108+
key: cargo-cache-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }}
109+
110+
- uses: astral-sh/setup-uv@v5
111+
with:
112+
python-version: "3.12"
113+
enable-cache: true
114+
115+
- name: Install xarray_sql
116+
run: uv sync --dev --no-install-package xarray-sql
117+
- name: build rust
118+
run: uv run --no-project maturin develop --uv
119+
- name: Install JAX
120+
env:
121+
VIRTUAL_ENV: ${{ github.workspace }}/.venv
122+
run: uv pip install jax
123+
- name: Run the demos
124+
env:
125+
VIRTUAL_ENV: ${{ github.workspace }}/.venv
126+
run: uv run --no-project pytest -v tests/test_ml_demos.py

‎benchmarks/ml/CONTRIBUTING.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Contributing an ML demo
2+
3+
A demo here is a machine-learning model written as SQL over tables that
4+
xarray-sql registers from xarray Datasets. It is trained or differentiated
5+
with [ddx](https://github.com/xqlsystems/ddx), which differentiates SQL
6+
queries, and checked against the same model in
7+
[JAX](https://github.com/jax-ml/jax), Google's library for differentiable
8+
array programs. Each demo is also a benchmark: the same SQL runs on every
9+
engine in [`_hosts.py`](_hosts.py), and its times are what ddx, xarray-sql
10+
and [einfold](https://github.com/xqlsystems/einfold) (a query optimizer
11+
that rewrites SQL plans into array algorithms) work to bring down. The
12+
performance work is tracked in
13+
[#285](https://github.com/xqlsystems/xarray-sql/issues/285).
14+
15+
Good demos are models whose SQL teaches something: a model people know
16+
from another library, written so a reader can follow it line by line.
17+
Some that would fit well:
18+
19+
- linear or logistic regression on a gridded dataset, such as ERA5
20+
temperatures;
21+
- a small convolutional network over a grid, where a convolution is a
22+
join on shifted coordinates;
23+
- an MLP on MNIST;
24+
- matrix factorization, or word2vec;
25+
- fitting a physical model's parameters to observations, such as a
26+
diffusion coefficient in the heat equation.
27+
28+
## What a demo is made of
29+
30+
Read [`01_attention.py`](01_attention.py) (short) and
31+
[`02_nanogpt.py`](02_nanogpt.py) (a whole training loop) first. A demo
32+
is one script, `NN_name.py`, numbered after the last one, with:
33+
34+
1. **Inline script metadata**, copied from the other demos, so that
35+
`uv run benchmarks/ml/NN_name.py` runs it against this checkout. Add
36+
any dependency the demo needs beyond `xarray-sql[ml,duckdb]` and `jax`.
37+
2. **A module docstring** that says:
38+
- what the model is, and links its reference implementation or paper;
39+
- how each piece of the model maps to SQL (`02_nanogpt.py` has a
40+
table for this);
41+
- every way the demo differs from the reference, and why.
42+
3. **The model as SQL**, built by small Python functions that each return
43+
one piece of SQL, named after the pieces of the reference. In
44+
`02_nanogpt.py`, `block()` is `layer_norm()`, `attention()`, `mlp()`
45+
and `residual()`, as nanoGPT's `Block` is. Write the model as a view
46+
(`CREATE OR REPLACE VIEW … AS`), and the loss as a separate query
47+
that reads it, as a Python loss calls its model.
48+
4. **The same model in JAX**, from the same inputs and parameters, as the
49+
reference. Turn on 64-bit floats
50+
(`jax.config.update("jax_enable_x64", True)`) so the two agree to
51+
about 1e-15.
52+
5. **A `main()`** that:
53+
- parses the shared command line from `_bench.cli` (`--hosts`,
54+
`--sizes`, `--reps`, `--csv`), plus the demo's own options;
55+
- loads the tables on each host;
56+
- times each step with `Report.time`, or `Report.attempt` for a
57+
gradient that a host may refuse;
58+
- checks each result with `assert_close` and `Report.check`;
59+
- ends with `Report.finish`.
60+
61+
A mismatch with JAX must fail the run, which `assert_close` does.
62+
6. **A line in `DEMOS`** in [`tests/test_ml_demos.py`](../../tests/test_ml_demos.py),
63+
with arguments that make the demo run in a few seconds. CI's `ml demos`
64+
job runs every demo there, and fails if one is missing.
65+
7. **A row in the table in [`README.md`](README.md)**, and anything it
66+
found under "Findings".
67+
68+
## Writing the SQL
69+
70+
- **Legibility over speed.** Write the model as its definition reads,
71+
even where a faster spelling exists. The slow parts are what the
72+
engines and optimizers need to make fast, and a demo that hides them
73+
measures nothing.
74+
- **Introduce every reference before relying on it**, in docstrings,
75+
comments and the README: say in a few words what a project, paper,
76+
algorithm or term is, and link it. This rule, and the others in
77+
[`AGENTS.md`](../../AGENTS.md), apply to demos too.
78+
- **Name issues only in TODOs.** Docstrings and comments name no issue
79+
numbers, except a workaround's `TODO(#NN): <action item>`, or
80+
`TODO(owner/repo#NN)` for another repository's issue.
81+
- **Make results deterministic** with `ORDER BY`, so they can be checked
82+
against JAX.
83+
- **Use no NaN as a value.** xarray-sql reads a NaN as SQL `NULL`, which
84+
aggregates skip. To check a mask, use an infinity instead, as
85+
`01_attention.py` does.
86+
- **Run the same SQL on every host.** If an engine needs different
87+
SQL, that difference is a finding: write it up rather than fork the
88+
query.
89+
90+
## Running
91+
92+
```shell
93+
uv run benchmarks/ml/NN_name.py --sizes 16 --hosts datafusion
94+
uv run pytest tests/test_ml_demos.py # every demo, small
95+
```
96+
97+
Against a ddx checkout rather than the release:
98+
99+
1. Build ddxdb into this checkout's `.venv`, with `maturin develop
100+
--release` in ddx's `python/ddxdb`.
101+
2. Run the demos with `uv run --no-sync`, or the venv's `python`. A plain
102+
`uv run` or `uv sync` reinstalls the release.
103+
104+
## When ddx or an engine can't run your model
105+
106+
That's a useful result. Each of these is a finding, not a failure:
107+
108+
- ddx raises a typed error (a `ddxdb.DdxError`) for anything it can't
109+
differentiate;
110+
- an engine may refuse a plan;
111+
- an engine may compute the wrong number.
112+
113+
`Report.attempt` records a refusal in the results table as `refused:`
114+
with the error. Then:
115+
116+
1. Cut the query down to a minimal repro.
117+
2. File it on [ddx](https://github.com/xqlsystems/ddx/issues), or on
118+
xarray-sql if it's about how a table is read.
119+
3. List the limit in the README's findings.
120+
121+
A demo may land with a refused step.
122+
123+
## Measuring
124+
125+
The output ends with a table: one row per step, host and size, with the
126+
median time, the error against JAX, and the number of operators in the
127+
host's plan. To compare runs, pass `--reps 3 --csv results.csv`, and post
128+
the table on [#285](https://github.com/xqlsystems/xarray-sql/issues/285),
129+
with your CPU and the versions of DataFusion, DuckDB and ddxdb.

‎benchmarks/ml/README.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ The SQL favors legibility over speed. Each model reads like its definition,
2424
and the slow parts are left slow on purpose: they are what ddx, xarray-sql
2525
and einfold need to make fast.
2626

27+
To add a demo, see [CONTRIBUTING.md](CONTRIBUTING.md). The work to make
28+
these models fast is tracked in
29+
[#285](https://github.com/xqlsystems/xarray-sql/issues/285), which lists
30+
places to start.
31+
2732
Every result is checked before it is timed into the table. Each script
2833
computes the same model in JAX from the same inputs and parameters, and a
2934
mismatch fails the run.
@@ -210,12 +215,11 @@ SQL gives a number.
210215

211216
## Next
212217

213-
- Part B of the NanoGPT demo: the four tricks behind a record of the
214-
[NanoGPT speedrun](https://github.com/KellerJordan/modded-nanogpt), a
215-
public race to train GPT-2 fastest, as a small diff to `02_nanogpt.py`.
216-
Its sparse optimizer updates build on `--optimizer adamw`.
217-
- Peak memory per host, from a fresh process per measurement.
218-
- An einfold host, once einfold produces plans.
218+
The open work, and how to measure it, is in
219+
[#285](https://github.com/xqlsystems/xarray-sql/issues/285). Part B of
220+
the NanoGPT demo is also still to come: the four tricks behind a record
221+
of the [NanoGPT speedrun](https://github.com/KellerJordan/modded-nanogpt),
222+
a public race to train GPT-2 fastest, as a small diff to `02_nanogpt.py`.
219223

220224
[`_bench.py`](_bench.py) holds the shared command line, timing, checks and
221225
results table.

‎tests/test_ml_demos.py‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
"""The ML demos in ``benchmarks/ml`` run, at their smallest sizes.
2+
3+
Each demo checks its results against JAX and exits with an error on a
4+
mismatch, so this runs each one and checks that it succeeds. It needs the
5+
``ml`` extra, DuckDB and JAX; CI's ``ml demos`` job installs them.
6+
7+
A new demo adds a line to ``DEMOS``: its script, and the arguments that
8+
make it small enough to run in a few seconds.
9+
"""
10+
11+
import pathlib
12+
import subprocess
13+
import sys
14+
15+
import pytest
16+
17+
pytest.importorskip("ddxdb")
18+
pytest.importorskip("duckdb")
19+
pytest.importorskip("jax")
20+
21+
DEMOS_DIR = pathlib.Path(__file__).parent.parent / "benchmarks" / "ml"
22+
23+
DEMOS = {
24+
"01_attention.py": ["--sizes", "16"],
25+
"02_nanogpt.py": ["--steps", "1"],
26+
}
27+
28+
29+
def test_every_demo_is_listed():
30+
scripts = {p.name for p in DEMOS_DIR.glob("[0-9][0-9]_*.py")}
31+
assert scripts == set(DEMOS)
32+
33+
34+
@pytest.mark.parametrize("script", sorted(DEMOS))
35+
def test_the_demo_runs_and_matches_jax(script):
36+
result = subprocess.run(
37+
[sys.executable, str(DEMOS_DIR / script), *DEMOS[script]],
38+
check=False,
39+
capture_output=True,
40+
text=True,
41+
timeout=600,
42+
)
43+
assert result.returncode == 0, result.stdout + result.stderr

0 commit comments

Comments
 (0)