Skip to content

Commit 5fecb86

Browse files
committed
docs: audit refined against the real sort-pushdown epic - phases 1-3 ship in 54
Epic apache/datafusion#23036 (not 17348): Phase 1 (PR 19064, DF 52 - PushdownSort rule + hooks + reverse iteration), Phase 2 (PR 21182, DF 53 - statistics-based file reordering + runtime TopK DynamicFilters), Phase 3 (PR 21956, DF 54 - multi-partition) are all in our pin; Phase 4 (PR 22450, runtime row-group early stopping) is in review for DF 55 and becomes row 38's concrete watch target, with arrow-rs 9937/10158 gating the DESC exact path. Verified live end to end in vendored sources: FileScanConfig::try_pushdown_sort passes eq_properties - the provider's declared ordering is a direct input - and ParquetSource implements the hook (source.rs:830). Plan 37's H5 now names the full call chain (declared ordering -> eq_properties -> try_pushdown_sort -> file/row-group reordering + dynamic filters, invisible in EXPLAIN) with Phase-2 reordering cross-linked to H1; the enable_sort_pushdown=false lever covers all phases in one same-binary toggle
1 parent 1b79a74 commit 5fecb86

3 files changed

Lines changed: 47 additions & 30 deletions

File tree

‎docs/notes/2026-07-13-datafusion-upgrade-audit.md‎

Lines changed: 45 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -8,40 +8,58 @@ thing the plan convention exists to prevent. Row 38 therefore sits **blocked
88
upstream**, and this note is the standing recipe for whoever writes the plan
99
when the trigger fires.
1010

11-
## Already in 54 — sort-pushdown Phase 1 (correction, 2026-07-13)
11+
## Already in 54 — sort-pushdown Phases 1–3 (corrections, 2026-07-13)
1212

13-
[PR #19064] ("Establish the high level API for sort pushdown and the optimizer
14-
rule…", merged 2025-12-17, released in **DataFusion 52**) is *not* a watch
15-
item — it is in our pin. Verified in the vendored 54 sources:
16-
`enable_sort_pushdown: bool, default = true`
17-
(`datafusion-common-54.0.0/src/config.rs:1289`) — the `PushdownSort` rule runs
18-
by default in every ukiel session today — and `try_pushdown_sort` trait hooks
19-
on `FileSource`/`DataSource` (`datafusion-datasource-54.0.0/src/file.rs:260`,
20-
`src/source.rs:215,499`), with `reverse_scan_inexact` for reversed
21-
file/row-group order (Inexact: the `SortExec` stays for correctness, TopK
22-
terminates early). Two consequences, neither belonging to row 38:
13+
The sort-pushdown epic is [apache/datafusion#23036] ("skip sorts and skip
14+
IO / decode for ORDER BY / TopK queries") — a phased initiative whose first
15+
three phases are **in our pin**, not upstream-future:
2316

24-
- **Row 37**: the rule is a *named candidate mechanism* for the unexplained
25-
declared-ordering concurrency effect (identical plans, different achieved
26-
parallelism — sort-aware machinery keying off declared orderings would look
27-
exactly like that), and the config flag is a clean same-binary A/B lever.
28-
Recorded in plan 37's hypothesis table.
29-
- **Opportunity on 54, no upgrade needed**: the provider could implement
30-
`try_pushdown_sort` (ORDER-BY-ts TopK over ts-sliced/dedicated files is the
31-
product-shaped beneficiary). If row 37's attribution or the product suite
32-
motivates it, that is a new row — not a reason to hold row 38 open.
17+
| phase | released in | what it does |
18+
|---|---|---|
19+
| 1 — [PR #19064] | DF **52** | `PushdownSort` optimizer rule (`enable_sort_pushdown` default **true**, `datafusion-common-54.0.0/src/config.rs:1289`), `try_pushdown_sort` hooks on `DataSource`/`FileSource`, reverse file/row-group iteration (Inexact: `SortExec` stays, TopK terminates early) |
20+
| 2 — PR #21182 | DF **53** | **statistics-based file reordering** for ORDER BY, runtime `DynamicFilter` pruning from the TopK heap (upstream: 27× on `ORDER BY LIMIT`) |
21+
| 3 — PR #21956 | DF **54** | multi-partition support (`BufferExec`) |
22+
| 4 — PR #22450 | DF **55**, in review | runtime row-group early stopping |
23+
24+
Verified live end-to-end in the vendored 54 sources: the machinery lives in
25+
`datafusion-datasource-54.0.0/src/file_scan_config/sort_pushdown.rs`;
26+
`FileScanConfig::try_pushdown_sort` (`file_scan_config/mod.rs:956`) delegates
27+
to the file source **passing `self.eq_properties()`** — i.e. the provider's
28+
declared output ordering is a *direct input* to the decision — and
29+
`ParquetSource` implements the hook
30+
(`datafusion-datasource-parquet-54.0.0/src/source.rs:830`), first checking
31+
whether the natural ordering already satisfies the requested sort. Outcomes:
32+
Exact (drop the `SortExec`), Inexact (keep it; reorder files/row groups by
33+
statistics; TopK dynamic filters prune at runtime), Unsupported. **None of
34+
this shows in EXPLAIN output.** Open arrow-rs dependencies for the DESC exact
35+
path: arrow-rs#9937 (page-level reverse decode) and arrow-rs#10158
36+
(`peek_next_row_group`). Consequences, neither belonging to row 38:
37+
38+
- **Row 37**: this is the leading *named candidate mechanism* for the
39+
declared-ordering effects we measured and could not see in plans — declared
40+
ordering → eq_properties → `ParquetSource::try_pushdown_sort` →
41+
file/row-group reordering + runtime dynamic filters, on by default,
42+
invisible in EXPLAIN. Plan 37's H5 carries it plus the one-line lever
43+
(`datafusion.optimizer.enable_sort_pushdown = false`, same binary).
44+
- **Opportunity on 54, no upgrade needed**: ukiel's provider already benefits
45+
passively wherever it declares an ordering (predicated scans, post
46+
predicate-gate); the *active* opportunity — catalog-aware ordering claims so
47+
ORDER-BY-ts TopK over ts-sliced/dedicated files hits the Exact/Inexact
48+
paths — is implementable against 54's hooks. If row 37's attribution or the
49+
product suite motivates it, that is a new row — not a reason to hold row 38
50+
open.
3351

3452
## Watch triggers (any one re-opens the row)
3553

3654
1. **A DataFusion release newer than 54.0.0 on crates.io** — check
3755
`https://crates.io/api/v1/crates/datafusion` (`max_stable_version`).
38-
2. **Sort-pushdown later phases landing in a release beyond 54**: the epic
39-
[apache/datafusion#17348] tracks what Phase 1 left out (exact row-level
40-
reversal; [PR #17337]'s `preferred_ordering` on `TableScan`). Relevant to
41-
the problems we measured on 54: predicate pushdown rebuilding the scan and
56+
2. **Sort-pushdown Phase 4 landing**: PR #22450 (runtime row-group early
57+
stopping) is in review targeting **DF 55** — the first concrete release
58+
target worth auditing; the DESC exact path additionally waits on
59+
arrow-rs#9937/#10158 (epic [apache/datafusion#23036]). Also still relevant
60+
on any newer release: whether predicate pushdown rebuilding the scan stops
4261
clearing the declared output ordering (plan-11 sort-elimination block; the
43-
plan-32-era predicate-gate workaround in `provider.rs`). When a release
44-
note mentions these, audit that release first.
62+
plan-32-era predicate-gate workaround in `provider.rs`).
4563
3. **Row 37's H5 disposition** naming a DF code path fixed in a newer release.
4664

4765
## What the eventual plan must contain (pre-written checklist)
@@ -102,6 +120,5 @@ If the release notes show only additive change on our surface: one plan,
102120
→ docs). If `AsyncFileReader`/`ObjectStore`/writer traits break: consider
103121
splitting read-side and write-side into two rows rather than one mega-plan.
104122

105-
[apache/datafusion#17348]: https://github.com/apache/datafusion/issues/17348
106-
[PR #17337]: https://github.com/apache/datafusion/pull/17337
123+
[apache/datafusion#23036]: https://github.com/apache/datafusion/issues/23036
107124
[PR #19064]: https://github.com/apache/datafusion/pull/19064

‎docs/superpowers/plans/2026-07-05-ukiel-v1-roadmap.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Spec: `docs/superpowers/specs/2026-07-05-ukiel-design.md`. V1 is split into sequ
4242
| 35 | ukiel-catalog-aggregate-pushdown | Issue 0010: metadata-answerable aggregates (`COUNT(*)`, `MIN`/`MAX` over stats-carrying Int64 columns) answered from `parts.row_count`/`parts.column_stats` instead of scanning — the official suite's worst ratios even with transport and layout controlled away (q07 12.9×, q01 9.7× over raw *on identical local files*; `docs/notes/2026-07-12-official-suite-gap.md`). Session optimizer rule or provider-level plan-shape check (issue 0010 sketches both); sound only when every candidate part carries the stat — conservative like the pruning path; namespaced `COUNT(*)` exact only over dedicated parts | Not written — sketch: `docs/issues/0010-aggregates-rescan-what-the-catalog-already-knows.md`; slots with the read-side tail (after 16; independent of 15), after 34 anchors the ceiling |
4343
| 36 | ukiel-narrow-column-types | Narrow int/date types in `ukiel-core` (today everything is int64/utf8/float64): the gap note priced the widening+ZSTD+re-sort bundle at 1.12× of the official-suite total (~10–25% extra decode on full scans vs the narrow-typed source files). Touches the type system end to end — schema JSON, arrow mapping, expr, stats, writers — so it needs a design sketch first; cheapest-first alternatives (dictionary encoding for low-cardinality utf8, per-table codec knob trading ZSTD for LZ4 on read-hot tables) belong in the same sketch | Not written — needs a sketch note; unscheduled (benchmark-ceiling pressure, not product-path; row 34's ladder will sharpen the price first) |
4444
| 37 | ukiel-stack-residual | Close the read-stack residual — post-15 the largest remaining official-suite lever: ukiel-local runs the 43 queries in 38.6 s where raw DataFusion over **ukiel's own files** runs 27.7 s (1.39×, ~10.9 s), spread 15–40% across scan-heavy queries. Known candidate mechanisms, all generic: the unpinned scan-concurrency effect (declared-ordering presence at scan-build time changes achieved parallelism ~1130%→~1450% CPU with byte-identical plans/metrics/total-CPU — gap note §declared-ordering; needs a DataFusion-level explanation), per-query catalog planning round-trips, 1,102 one-file scan partitions vs balanced groups, session construction per request. Attribution first (profile/flamegraph the same-bytes pair), fixes second | **Ready to execute** — plan `2026-07-13-ukiel-stack-residual.md` (instruments → hypothesis gate → pre-registered conditional fixes → exit criteria: ≤1.15× same-bytes or every component named-and-dispositioned). **Independent of 16/35**: 16 is scoped-slice-only, invisible to the unscoped suite; 35's q01/q07 are excluded from scope — executes in any order relative to them |
45-
| 38 | ukiel-datafusion-upgrade | DataFusion 54 → current: moves the whole pin set in lockstep (arrow/parquet 58.3, object_store 0.13 are "the versions DF 54 links against"), so it is a plan, not a chore. Motivations, each measured on 54: pushdown clears declared ordering (blocked plan-11 sort elimination; forced the plan-32-era predicate-gate workaround in `provider.rs` — on a fixed DF, restore the unconditional declaration and delete the gate), parquet reader improvements upstream, and whatever row 37 pins on the engine side. Exit criteria: all suites green, baselines re-run (11/13/30/32 recorded numbers re-anchored), predicate-gate revisited with a same-binary A/B | **Blocked upstream — audit done, nothing to upgrade to** (`docs/notes/2026-07-13-datafusion-upgrade-audit.md`): 54.0.0 (2026-06-08) is crates.io's newest; a plan against unreleased 55 would guess interfaces. Watch triggers: a newer release, sort-pushdown later phases landing beyond 54 (epic apache/datafusion#17348; Phase 1 = PR #19064 is **already in 54** — `PushdownSort` runs by default, named in plan 37's H5, and `try_pushdown_sort` is implementable on 54 as its own row), or row 37's H5 naming a fixed path. The audit note carries the full plan checklist (pin lockstep, per-crate API surface, behavior re-validations, baseline re-anchoring); still after 8, coordinated with 37 |
45+
| 38 | ukiel-datafusion-upgrade | DataFusion 54 → current: moves the whole pin set in lockstep (arrow/parquet 58.3, object_store 0.13 are "the versions DF 54 links against"), so it is a plan, not a chore. Motivations, each measured on 54: pushdown clears declared ordering (blocked plan-11 sort elimination; forced the plan-32-era predicate-gate workaround in `provider.rs` — on a fixed DF, restore the unconditional declaration and delete the gate), parquet reader improvements upstream, and whatever row 37 pins on the engine side. Exit criteria: all suites green, baselines re-run (11/13/30/32 recorded numbers re-anchored), predicate-gate revisited with a same-binary A/B | **Blocked upstream — audit done, nothing to upgrade to** (`docs/notes/2026-07-13-datafusion-upgrade-audit.md`): 54.0.0 (2026-06-08) is crates.io's newest; a plan against unreleased 55 would guess interfaces. Watch triggers: a newer release; **sort-pushdown Phase 4** (PR #22450, runtime row-group early stopping) landing in DF 55 — phases 1–3 of epic apache/datafusion#23036 are **already in 54** (`PushdownSort` default-on, statistics-based file reordering + TopK dynamic filters, multi-partition; `ParquetSource::try_pushdown_sort` live and fed by our declared orderings — named in plan 37's H5, and actively implementable on 54 as its own row); or row 37's H5 naming a fixed path. The audit note carries the full plan checklist (pin lockstep, per-crate API surface, behavior re-validations, baseline re-anchoring); still after 8, coordinated with 37 |
4646
| 30 | `2026-07-08-ukiel-macro-perf.md` | Macro perf harness: volume-scale benchmarks on real datasets, bridging the plan-11 micro tier (~100k synthetic rows) to the README's PB+ intent. `bench` binary in `ukiel-e2e` reusing the e2e `Stack`: **read benchmark** on ClickBench `hits` (~100M rows / 14.8 GB Parquet; cast to a 14-column ukiel schema, per-file L1 commits, per-tenant ClickBench-derived scenario suite with warm medians — heavy/median/light tenants), **write-path benchmark** on Bluesky/JSONBench ndjson (1M events/file; 100M tier ≈ 12.5 GB gz, 1B = 125 GB stretch) through Kafka → ingest → ladder → finalization with exactly-once *asserted* at volume, drain-waves bounding Kafka disk, deferral/capped-merge counters captured; runbook + baselines note | **Executed** — `bench` binary in `ukiel-e2e` (manual-only, `--release`): ClickBench `hits` loader + per-tenant read suite (warm medians, stats-pruning probes), Bluesky ndjson mapper + rate-aware Kafka producer + full write-path run (ingest→ladder→finalization) that *asserts* exactly-once at volume; JSON reports + `make bench-fetch-*` targets. Smoke-verified end to end (1-file each: bsky 1M events → stored==mapped, finalized to one L2 run; hits 7-tenant read suite). Runbook + full command reference + baselines: `bench/README.md` (design rationale in `docs/notes/2026-07-08-macro-perf.md`; full 10 GB-tier numbers pending a dedicated run). Compacted-hits layout deferred to row 29 (a 15 GB merge trips the plan-28 cap by design) |
4747

4848
Cross-plan constraints (repeat in every plan's Global Constraints):

‎docs/superpowers/plans/2026-07-13-ukiel-stack-residual.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
| H2 | Per-query planning cost (catalog round-trip, provider construction, physical planning) is material | Task-1 plan/execute split across the suite | F2 (or a new row if the fix is a catalog snapshot cache) |
2121
| H3 | 1,102 one-file partitions cost per-partition setup (opener, stream, readahead) vs 24 balanced groups | `clickbench analyze` on loaded (1,102 parts) vs the same data compacted; flamegraph diff | F1 |
2222
| H4 | Per-file reader-factory/schema-adaptation overhead (`CachingParquetFileReaderFactory` wrapping, physical-schema mapping) | flamegraph diff ukiel-vs-raw on the uniform-band queries | F3 |
23-
| H5 | The concurrency effect is DataFusion-54-internal (not reachable from provider code). **Named candidate: the sort-pushdown Phase-1 machinery** — PR #19064 shipped in DF 52, so the `PushdownSort` rule runs by default in 54 (`enable_sort_pushdown = true`, `datafusion-common-54.0.0/src/config.rs:1289`; `try_pushdown_sort` hooks at `datafusion-datasource-54.0.0/src/file.rs:260`) and sort-aware machinery keying off declared orderings would produce exactly "identical displayed plans, different execution" | the config flag is a clean same-binary lever: repeat the ordering-toggle pairs with `datafusion.optimizer.enable_sort_pushdown = false` — if the effect vanishes, H5 is pinned | defer to row 38 with the path named; if pinned to `PushdownSort`, also evaluate implementing `try_pushdown_sort` in the provider as its own row (audit note §already-in-54) |
23+
| H5 | The concurrency effect is DataFusion-54-internal (not reachable from provider code). **Named candidate: the sort-pushdown machinery, phases 1–3 of which ship in 54** (epic apache/datafusion#23036; `enable_sort_pushdown = true` by default, `datafusion-common-54.0.0/src/config.rs:1289`). The call chain is live and fed by our declaration: declared ordering → `eq_properties()` → `FileScanConfig::try_pushdown_sort` (`file_scan_config/mod.rs:956`) → `ParquetSource::try_pushdown_sort` (`datafusion-datasource-parquet-54.0.0/src/source.rs:830`) → statistics-based file/row-group reordering + runtime TopK `DynamicFilter`s — **none of it visible in EXPLAIN**, which is exactly the observed signature. Phase-2 file *reordering* is also H1-relevant (composition differences) | the config flag is a clean same-binary lever: repeat the ordering-toggle pairs with `datafusion.optimizer.enable_sort_pushdown = false` on both configs — if the effect vanishes, H5 is pinned; `clickbench plan`'s group dump then shows the reordering directly | defer to row 38 with the path named (Phase 4 / PR #22450 targets DF 55); if pinned, also evaluate actively implementing catalog-aware ordering for the Exact/Inexact TopK paths as its own row (audit note §already-in-54) |
2424

2525
**Tech Stack:** verified against vendored sources — `DataFrame::create_physical_plan(&self) -> Result<Arc<dyn ExecutionPlan>>` (`datafusion-54.0.0/src/dataframe/mod.rs:301`); plan walking via `ExecutionPlan::children()` with `DataSourceExec::data_source()` + `Arc::downcast`/`downcast_ref::<FileScanConfig>()` (`datafusion-datasource-54.0.0/src/source.rs:559,298`) exposing `file_groups`, and node `properties()` exposing partitioning + equivalence orderings; per-partition attribution via `ExecutionPlan::metrics() -> Option<MetricsSet>` whose `Metric.partition: Option<usize>` labels values per partition (`datafusion-physical-expr-common-54.0.0/src/metrics/mod.rs:76–197`); `/usr/bin/perf` is present on the bench box (fallback: any sampling profiler; record which). The suite harness is plan-34's `Session` enum (`suite.rs`) — the split and instruments apply to the `Session::DataFusion` arm; the ClickHouse arm records `None`.
2626

0 commit comments

Comments
 (0)