You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5fecb86
Browse filesBrowse the repository at this point in the historyBrowse files
docs: audit refined against the real sort-pushdown epic - phases 1-3 ship in 54
Epic apache/datafusion#23036 (not 17348): Phase 1 (PR 19064, DF 52 -
PushdownSort rule + hooks + reverse iteration), Phase 2 (PR 21182, DF 53 -
statistics-based file reordering + runtime TopK DynamicFilters), Phase 3
(PR 21956, DF 54 - multi-partition) are all in our pin; Phase 4 (PR 22450,
runtime row-group early stopping) is in review for DF 55 and becomes row 38's
concrete watch target, with arrow-rs 9937/10158 gating the DESC exact path.
Verified live end to end in vendored sources: FileScanConfig::try_pushdown_sort
passes eq_properties - the provider's declared ordering is a direct input -
and ParquetSource implements the hook (source.rs:830). Plan 37's H5 now names
the full call chain (declared ordering -> eq_properties -> try_pushdown_sort
-> file/row-group reordering + dynamic filters, invisible in EXPLAIN) with
Phase-2 reordering cross-linked to H1; the enable_sort_pushdown=false lever
covers all phases in one same-binary toggle
| 2 — PR #21182| DF **53**|**statistics-based file reordering** for ORDER BY, runtime `DynamicFilter` pruning from the TopK heap (upstream: 27× on `ORDER BY LIMIT`) |
Copy file name to clipboardExpand all lines: docs/superpowers/plans/2026-07-05-ukiel-v1-roadmap.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -42,7 +42,7 @@ Spec: `docs/superpowers/specs/2026-07-05-ukiel-design.md`. V1 is split into sequ
42
42
| 35 | ukiel-catalog-aggregate-pushdown | Issue 0010: metadata-answerable aggregates (`COUNT(*)`, `MIN`/`MAX` over stats-carrying Int64 columns) answered from `parts.row_count`/`parts.column_stats` instead of scanning — the official suite's worst ratios even with transport and layout controlled away (q07 12.9×, q01 9.7× over raw *on identical local files*; `docs/notes/2026-07-12-official-suite-gap.md`). Session optimizer rule or provider-level plan-shape check (issue 0010 sketches both); sound only when every candidate part carries the stat — conservative like the pruning path; namespaced `COUNT(*)` exact only over dedicated parts | Not written — sketch: `docs/issues/0010-aggregates-rescan-what-the-catalog-already-knows.md`; slots with the read-side tail (after 16; independent of 15), after 34 anchors the ceiling |
43
43
| 36 | ukiel-narrow-column-types | Narrow int/date types in `ukiel-core` (today everything is int64/utf8/float64): the gap note priced the widening+ZSTD+re-sort bundle at 1.12× of the official-suite total (~10–25% extra decode on full scans vs the narrow-typed source files). Touches the type system end to end — schema JSON, arrow mapping, expr, stats, writers — so it needs a design sketch first; cheapest-first alternatives (dictionary encoding for low-cardinality utf8, per-table codec knob trading ZSTD for LZ4 on read-hot tables) belong in the same sketch | Not written — needs a sketch note; unscheduled (benchmark-ceiling pressure, not product-path; row 34's ladder will sharpen the price first) |
44
44
| 37 | ukiel-stack-residual | Close the read-stack residual — post-15 the largest remaining official-suite lever: ukiel-local runs the 43 queries in 38.6 s where raw DataFusion over **ukiel's own files** runs 27.7 s (1.39×, ~10.9 s), spread 15–40% across scan-heavy queries. Known candidate mechanisms, all generic: the unpinned scan-concurrency effect (declared-ordering presence at scan-build time changes achieved parallelism ~1130%→~1450% CPU with byte-identical plans/metrics/total-CPU — gap note §declared-ordering; needs a DataFusion-level explanation), per-query catalog planning round-trips, 1,102 one-file scan partitions vs balanced groups, session construction per request. Attribution first (profile/flamegraph the same-bytes pair), fixes second | **Ready to execute** — plan `2026-07-13-ukiel-stack-residual.md` (instruments → hypothesis gate → pre-registered conditional fixes → exit criteria: ≤1.15× same-bytes or every component named-and-dispositioned). **Independent of 16/35**: 16 is scoped-slice-only, invisible to the unscoped suite; 35's q01/q07 are excluded from scope — executes in any order relative to them |
45
-
| 38 | ukiel-datafusion-upgrade | DataFusion 54 → current: moves the whole pin set in lockstep (arrow/parquet 58.3, object_store 0.13 are "the versions DF 54 links against"), so it is a plan, not a chore. Motivations, each measured on 54: pushdown clears declared ordering (blocked plan-11 sort elimination; forced the plan-32-era predicate-gate workaround in `provider.rs` — on a fixed DF, restore the unconditional declaration and delete the gate), parquet reader improvements upstream, and whatever row 37 pins on the engine side. Exit criteria: all suites green, baselines re-run (11/13/30/32 recorded numbers re-anchored), predicate-gate revisited with a same-binary A/B | **Blocked upstream — audit done, nothing to upgrade to** (`docs/notes/2026-07-13-datafusion-upgrade-audit.md`): 54.0.0 (2026-06-08) is crates.io's newest; a plan against unreleased 55 would guess interfaces. Watch triggers: a newer release, sort-pushdown later phases landing beyond 54 (epic apache/datafusion#17348; Phase 1 = PR #19064 is **already in 54** — `PushdownSort` runs by default, named in plan 37's H5, and `try_pushdown_sort` is implementable on 54 as its own row), or row 37's H5 naming a fixed path. The audit note carries the full plan checklist (pin lockstep, per-crate API surface, behavior re-validations, baseline re-anchoring); still after 8, coordinated with 37 |
45
+
| 38 | ukiel-datafusion-upgrade | DataFusion 54 → current: moves the whole pin set in lockstep (arrow/parquet 58.3, object_store 0.13 are "the versions DF 54 links against"), so it is a plan, not a chore. Motivations, each measured on 54: pushdown clears declared ordering (blocked plan-11 sort elimination; forced the plan-32-era predicate-gate workaround in `provider.rs` — on a fixed DF, restore the unconditional declaration and delete the gate), parquet reader improvements upstream, and whatever row 37 pins on the engine side. Exit criteria: all suites green, baselines re-run (11/13/30/32 recorded numbers re-anchored), predicate-gate revisited with a same-binary A/B | **Blocked upstream — audit done, nothing to upgrade to** (`docs/notes/2026-07-13-datafusion-upgrade-audit.md`): 54.0.0 (2026-06-08) is crates.io's newest; a plan against unreleased 55 would guess interfaces. Watch triggers: a newer release; **sort-pushdown Phase 4** (PR #22450, runtime row-group early stopping) landing in DF 55 — phases 1–3 of epic apache/datafusion#23036 are **already in 54** (`PushdownSort` default-on, statistics-based file reordering + TopK dynamic filters, multi-partition; `ParquetSource::try_pushdown_sort` live and fed by our declared orderings — named in plan 37's H5, and actively implementable on 54 as its own row); or row 37's H5 naming a fixed path. The audit note carries the full plan checklist (pin lockstep, per-crate API surface, behavior re-validations, baseline re-anchoring); still after 8, coordinated with 37 |
46
46
| 30 | `2026-07-08-ukiel-macro-perf.md` | Macro perf harness: volume-scale benchmarks on real datasets, bridging the plan-11 micro tier (~100k synthetic rows) to the README's PB+ intent. `bench` binary in `ukiel-e2e` reusing the e2e `Stack`: **read benchmark** on ClickBench `hits` (~100M rows / 14.8 GB Parquet; cast to a 14-column ukiel schema, per-file L1 commits, per-tenant ClickBench-derived scenario suite with warm medians — heavy/median/light tenants), **write-path benchmark** on Bluesky/JSONBench ndjson (1M events/file; 100M tier ≈ 12.5 GB gz, 1B = 125 GB stretch) through Kafka → ingest → ladder → finalization with exactly-once *asserted* at volume, drain-waves bounding Kafka disk, deferral/capped-merge counters captured; runbook + baselines note | **Executed** — `bench` binary in `ukiel-e2e` (manual-only, `--release`): ClickBench `hits` loader + per-tenant read suite (warm medians, stats-pruning probes), Bluesky ndjson mapper + rate-aware Kafka producer + full write-path run (ingest→ladder→finalization) that *asserts* exactly-once at volume; JSON reports + `make bench-fetch-*` targets. Smoke-verified end to end (1-file each: bsky 1M events → stored==mapped, finalized to one L2 run; hits 7-tenant read suite). Runbook + full command reference + baselines: `bench/README.md` (design rationale in `docs/notes/2026-07-08-macro-perf.md`; full 10 GB-tier numbers pending a dedicated run). Compacted-hits layout deferred to row 29 (a 15 GB merge trips the plan-28 cap by design) |
47
47
48
48
Cross-plan constraints (repeat in every plan's Global Constraints):
Copy file name to clipboardExpand all lines: docs/superpowers/plans/2026-07-13-ukiel-stack-residual.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,7 +20,7 @@
20
20
| H2 | Per-query planning cost (catalog round-trip, provider construction, physical planning) is material | Task-1 plan/execute split across the suite | F2 (or a new row if the fix is a catalog snapshot cache) |
21
21
| H3 | 1,102 one-file partitions cost per-partition setup (opener, stream, readahead) vs 24 balanced groups |`clickbench analyze` on loaded (1,102 parts) vs the same data compacted; flamegraph diff | F1 |
| H5 | The concurrency effect is DataFusion-54-internal (not reachable from provider code). **Named candidate: the sort-pushdown Phase-1 machinery** — PR #19064 shipped in DF 52, so the `PushdownSort` rule runs by default in 54 (`enable_sort_pushdown = true`, `datafusion-common-54.0.0/src/config.rs:1289`; `try_pushdown_sort` hooks at `datafusion-datasource-54.0.0/src/file.rs:260`) and sort-aware machinery keying off declared orderings would produce exactly "identical displayed plans, different execution" | the config flag is a clean same-binary lever: repeat the ordering-toggle pairs with `datafusion.optimizer.enable_sort_pushdown = false` — if the effect vanishes, H5 is pinned | defer to row 38 with the path named; if pinned to `PushdownSort`, also evaluate implementing `try_pushdown_sort` in the provider as its own row (audit note §already-in-54) |
23
+
| H5 | The concurrency effect is DataFusion-54-internal (not reachable from provider code). **Named candidate: the sort-pushdown machinery, phases 1–3 of which ship in 54** (epic apache/datafusion#23036; `enable_sort_pushdown = true` by default, `datafusion-common-54.0.0/src/config.rs:1289`). The call chain is live and fed by our declaration: declared ordering → `eq_properties()` → `FileScanConfig::try_pushdown_sort` (`file_scan_config/mod.rs:956`) → `ParquetSource::try_pushdown_sort` (`datafusion-datasource-parquet-54.0.0/src/source.rs:830`) → statistics-based file/row-group reordering + runtime TopK `DynamicFilter`s — **none of it visible in EXPLAIN**, which is exactly the observed signature. Phase-2 file *reordering* is also H1-relevant (composition differences) | the config flag is a clean same-binary lever: repeat the ordering-toggle pairs with `datafusion.optimizer.enable_sort_pushdown = false` on both configs — if the effect vanishes, H5 is pinned; `clickbench plan`'s group dump then shows the reordering directly | defer to row 38 with the path named (Phase 4 / PR #22450 targets DF 55); if pinned, also evaluate actively implementing catalog-aware ordering for the Exact/Inexact TopK paths as its own row (audit note §already-in-54) |
24
24
25
25
**Tech Stack:** verified against vendored sources — `DataFrame::create_physical_plan(&self) -> Result<Arc<dyn ExecutionPlan>>` (`datafusion-54.0.0/src/dataframe/mod.rs:301`); plan walking via `ExecutionPlan::children()` with `DataSourceExec::data_source()` + `Arc::downcast`/`downcast_ref::<FileScanConfig>()` (`datafusion-datasource-54.0.0/src/source.rs:559,298`) exposing `file_groups`, and node `properties()` exposing partitioning + equivalence orderings; per-partition attribution via `ExecutionPlan::metrics() -> Option<MetricsSet>` whose `Metric.partition: Option<usize>` labels values per partition (`datafusion-physical-expr-common-54.0.0/src/metrics/mod.rs:76–197`); `/usr/bin/perf` is present on the bench box (fallback: any sampling profiler; record which). The suite harness is plan-34's `Session` enum (`suite.rs`) — the split and instruments apply to the `Session::DataFusion` arm; the ClickHouse arm records `None`.
0 commit comments