Repository navigation
Commit 86d5c00
Split a query across partitions only where enough rows reach it (#289)
Part of workstream 1, per-query overhead, in #285.
## What
`Context()`, given no `config`, now sets DataFusion's
`datafusion.execution.use_row_number_estimates_to_optimize_partitioning`
to `true`. Everything else stays as `SessionContext()` sets it,
including the information schema. A `config` given is used as it is.
## Why
DataFusion decides whether to split an input across partitions from its
row count. When the count is exact, as it is for a table, DataFusion
splits only inputs larger than a batch. When the count is an estimate,
as it is after a join, a filter or an aggregate, DataFusion splits the
input however few rows the estimate says it has, unless this setting is
on. apache/datafusion#26087 describes this, and apache/datafusion#18595
asks to stop repartitioning small datasets.
The ML demos run many queries of a few thousand rows each. On 12 cores,
every one of them was split into 12 partitions and merged back:
| Measured (DataFusion 55.0.0, i7-8700, quiet machine) | before | after
| one partition |
|---|---:|---:|---:|
| ddx's 67 step plans of one GPT SGD step, executing only | 151 ms | 69
ms | 61 ms |
| GPT SGD step (`02_nanogpt.py`, later steps) | 0.655 s | 0.415 s | |
| GPT forward loss: operators, seconds | 13,756, 3.6 s | 9,882, 3.1 s |
|
| Attention, 512 tokens, forward and gradient, both spellings | 0.28 /
0.86 / 0.41 / 0.97 s | 0.27 / 0.87 / 0.42 / 0.98 s | |
Every check against JAX passes with the setting on. Lazy Datasets of 4
and 16 million rows plan the same `RepartitionExec`s, and take the same
time, with and without it: a group-by, a filtered group-by and a
self-join.
## Risk
A low estimate on an input that's really large would leave it in one
partition. A filter's estimate is a fraction of its input's rows, so a
large table still gives a large estimate. Anyone who wants DataFusion's
default can pass `SessionConfig()`.
## Tests
- `Context()` runs a small join and aggregate with no `RepartitionExec`.
- A given config is used as it is: with 4 target partitions, the same
query repartitions.
- `Context()` still has the information schema. A bare `SessionConfig()`
turns it off, and ddx reads view definitions from it.
The ML demos' README numbers will be re-recorded in #269, or after it,
once this merges.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01NyN7idACeDMh94qrnFMu6h
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* SQL contexts now expose the information schema by default, including
table listings.
* By default, small-input joins use row-number estimates when
determining partitioning and omit repartitioning when it is unnecessary.
Configuring a target partition count can still result in repartitioning.
* You can provide a custom session configuration when creating a
context; supplied settings are used unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent c70b1b2 commit 86d5c00
2 files changed
Lines changed: 57 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
16 | 51 | | |
17 | 52 | | |
18 | 53 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
36 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
37 | 56 | | |
38 | 57 | | |
39 | 58 | | |
| |||
0 commit comments