Skip to content

Commit 1361a59

Browse files
alxmrsclaude
andcommitted
Require ddxdb 0.4.3, which trains the GPT on DuckDB
ddxdb 0.4.3 differentiates a view whose CTE computes a window on DuckDB (xqlsystems/ddx#204), so the demos no longer expect DuckDB to refuse attention's window spelling or the GPT's training step. Record the GPT on DuckDB: its weights match JAX's, its loss takes 0.062 s against DataFusion's 2.7 s, and an SGD step takes 0.87 s against 0.41 s, most of it DuckDB planning ddx's 77 steps afresh and the adapter's reads. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h
1 parent 883108b commit 1361a59

5 files changed

Lines changed: 36 additions & 45 deletions

File tree

‎benchmarks/ml/01_attention.py‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -202,14 +202,7 @@ def main():
202202
if not host.differentiates:
203203
report.skip(step, host, n, "no ddx backend")
204204
continue
205-
known = {}
206-
if spelling == "window":
207-
# TODO(xqlsystems/ddx#204): drop once ddx differentiates
208-
# a view whose CTE computes a window on DuckDB.
209-
known = {"duckdb": "Not implemented Error: CTE"}
210-
rows = report.attempt(
211-
step, host, n, host.query, GRADIENT, known=known
212-
)
205+
rows = report.attempt(step, host, n, host.query, GRADIENT)
213206
if rows is None:
214207
continue
215208
errors = [

‎benchmarks/ml/02_nanogpt.py‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -603,12 +603,9 @@ def main():
603603
if not host.differentiates:
604604
report.skip(train, host, size, "no ddx backend")
605605
continue
606-
# TODO(xqlsystems/ddx#204): drop once ddx differentiates a
607-
# view whose CTE computes a window on DuckDB.
608-
known = {"duckdb": "Not implemented Error: CTE"}
609606
if adamw:
610607
updated = report.attempt(
611-
train, host, size, adamw_step, host, step, known=known
608+
train, host, size, adamw_step, host, step
612609
)
613610
else:
614611
updated = report.attempt(
@@ -617,7 +614,6 @@ def main():
617614
size,
618615
host.query_all,
619616
step_statements(LEARNING_RATE),
620-
known=known,
621617
)
622618
if updated is None:
623619
# Every later step starts from this one's weights.

‎benchmarks/ml/README.md‎

Lines changed: 23 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,8 @@ The GPT and attention demos were timed again on 2026-10-21 with
121121
DataFusion 55.0.0 and ddxdb 0.4.0, after xarray-sql began registering
122122
small in-memory Datasets as DataFusion in-memory tables. Both demos were
123123
timed again on 2026-10-25 with DataFusion 55.0.0, DuckDB 1.5.5 and
124-
ddxdb 0.4.2; where those runs give new numbers, they are given here.
124+
ddxdb 0.4.2, and on DuckDB on 2026-10-27 with DuckDB 1.5.6 and ddxdb
125+
0.4.3; where those runs give new numbers, they are given here.
125126

126127
**ddx's gradients agree with JAX.** The gradients of attention with
127128
respect to Q, K and V, in both spellings, match `jax.grad` to 3e-15 of the
@@ -200,21 +201,27 @@ against 0.20 s for one step), but copying Q, K and V into native tables
200201
only brings the gradient to 1.7 s, and slows the forward pass (measured
201202
on 2026-10-24, with ddx's development branch and DuckDB 1.5.6).
202203

203-
**What ddx cannot differentiate yet.** Each of these raised a typed
204-
error naming the construct, never a wrong number:
205-
206-
- A `GROUP BY` key that depends on a differentiated column. einfold's
207-
window spelling groups by the normalizer `z`; the demos divide inside
208-
the sum instead, `SUM(w / z * v)`, which einfold also lists.
209-
- On DuckDB, a view whose CTE computes a window and is read more than
210-
once, as the softmax's weights are in attention's window spelling and
211-
in the GPT, so neither trains on DuckDB yet. ddx reads DuckDB's plans
212-
as [Substrait](https://substrait.io/), a portable format for query
213-
plans, and DuckDB's Substrait producer has no plan for a CTE it
214-
computes once for several reads. ddxdb 0.4.2 writes such a view's SQL
215-
into the query instead, except when the CTE computes a window
216-
([ddx#204](https://github.com/xqlsystems/ddx/issues/204)). These rows
217-
say `refused: Not implemented Error: CTE`.
204+
**The GPT trains on DuckDB.** Measured on 2026-10-27 with ddxdb 0.4.3,
205+
the first to differentiate a view whose CTE computes a window on DuckDB,
206+
as the softmax's weights are in the GPT and in attention's window
207+
spelling. After each of four SGD steps and three AdamW steps, the GPT's
208+
weights match JAX's to 2e-15 on DuckDB, and attention's window spelling
209+
matches `jax.grad` to 2e-15, taking 0.68 s and 1.9 s at 256 and 512
210+
tokens. DuckDB computes the GPT's loss in 0.062 s, against 2.7 s on
211+
DataFusion in the same run, but an SGD step takes 0.87 s, against 0.41
212+
s (1.15 to 1.3 s with AdamW). ddx runs its program's 77 steps on DuckDB
213+
as `CREATE TEMP TABLE … AS` statements, which take 0.56 s of the 0.87 s:
214+
DuckDB plans each one afresh, which takes about 0.3 s (an `EXPLAIN` of
215+
each takes 0.31 s in all), and xarray-sql's adapter reads the parameter
216+
Datasets in Python for about 0.19 s of the running. Most of the rest is ddx
217+
planning on DuckDB, against an empty copy of the connection's catalog
218+
that it keeps current.
219+
220+
**What ddx cannot differentiate yet.** A `GROUP BY` key that depends on
221+
a differentiated column raises a typed error naming the construct, never
222+
a wrong number. einfold's window spelling groups by the normalizer `z`;
223+
the demos divide inside the sum instead, `SUM(w / z * v)`, which einfold
224+
also lists.
218225

219226
**xarray-sql reads NaN as missing.** A NaN in a Dataset becomes SQL
220227
`NULL`, which `SUM` skips, so Demo 1 checks masks with an infinity: a

‎pyproject.toml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ geo = [
6262
# and breaks its API between minor versions. It reads and writes the
6363
# Substrait of DataFusion 55, so bump it together with that pin.
6464
ml = [
65-
"ddxdb>=0.4.2,<0.5; sys_platform != 'emscripten'",
65+
"ddxdb>=0.4.3,<0.5; sys_platform != 'emscripten'",
6666
]
6767
test = [
6868
"adbc-driver-flightsql>=1.12",
@@ -138,7 +138,7 @@ dev = [
138138
"maturin>=1.9.1",
139139
# The ML demos: ddx's DuckDB backend, at the DuckDB it declares, and
140140
# JAX, which the demos check their results against.
141-
"ddxdb[duckdb]>=0.4.2,<0.5; sys_platform != 'emscripten'",
141+
"ddxdb[duckdb]>=0.4.3,<0.5; sys_platform != 'emscripten'",
142142
"jax",
143143
]
144144

‎uv.lock‎

Lines changed: 9 additions & 14 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)