feat(layout): zoned pruning, shares, and dictionary value reuse in physical plans by joseph-isaacs · Pull Request #10462 · vortex-data/vortex

joseph-isaacs · 2026-10-13T12:01:40Z

Summary

First of three stacked PRs carved out of #10339, rebuilt on develop's push-based pipelines (#10391–#10393):

This PR: plan. The physical-plan operators the planner-driven scan needs.
IO: a coalescing IoService for planning-protocol scans (stacked on this).
Planning: the planner-driven V2 scan (stacked on IO).

This PR is independent of the other two, and nothing outside plan::physical uses it yet.

Changes

Zoned plan. Data plus zone statistics in the aggregate schema. An Eval of an expression over statistics rewrites it into a pruning plan, which proves the expression per zone once (re-proving when a dynamic comparison changes) and expands the result to the selected rows. Rebuilding a pruning plan over new zones discards the old proof. Zoned layouts lower to it, flat layouts lower to a filtered segment scan, and zoned or legacy stats layouts with no zones or no statistics lower to their data.
LegacyStats plan. Legacy vortex.stats layouts lower to it. The optimizer rewrites it:
- into its data alone when a Zoned plan already summarises the same rows (above or beneath it), so legacy stats are only used when no aggregate zone map exists;
- into its data alone for a zero zone length, or a table with no stat an aggregate can stand for;
- otherwise into Zoned(data, Eval(convert, zones)), where convert packs each legacy stat column under its aggregate's name. Truncation flags are dropped, as the legacy reader does.
Zoned therefore only ever sees the aggregate schema, and ZoneMap::try_new_legacy stays test-only.
Share plan. Canonicalizes a whole child once and serves its rows.
Take.
- The expression rule pushes the strict, infallible, negative-cost part of an expression beneath a share of dictionary values, e.g. byte_length inside cast(byte_length($)), rebinding it to the values' dtype when null codes make the take more nullable than its values.
- An expression holding a dynamic comparison is never pushed onto dictionary values: it stays above the take, reads the latest bound per batch, and the dictionary batches still evaluate it over their values. Nothing dynamic is ever beneath a take's values, so they are always kept between scans.
ValueCache. Keeps whole values such as dictionary values between the scans that share it, keyed by plan identity (PlanByPtr, which holds the plan so its address can never name another plan). Its owner bounds its lifetime: nothing is kept on plans.
WholeSource. Collects an inlet whole before producing one batch; used by shares and zone tables.
Shared-segment counting. A pipeline Scan only counts shared segments over the rows its splits read, not the whole plan.
lower is documented as the layout lowering rather than a test-only fixture builder.

Eval still applies its bound expression to each batch: evaluating predicates on encoded values (dictionary, run-end, PCO) is array-level optimization and belongs in vortex-array and the encoding crates, not in a plan operator. No existing trait definition changes.

Tests:

New coverage in plan::physical::tests for zoned pruning, the negative-cost take rule and shares.
Legacy stats: lowering, the rewrite to Zoned over converted zones, a converted legacy table pruning the expected zones, legacy stats dropped above or beneath Zoned.
Zoned and legacy layouts without zones or statistics lower to their data; a rebuilt pruning plan discards its proof.
Dynamic predicates, alone or with static conjuncts, stay above dictionary values; an expression is rebound to values less nullable than their take.
A pipeline test checks that a scan sharing a ValueCache reads only the codes, while a fresh cache reads the values again.

Checks run locally on the branch merged with develop:

cargo clippy --locked -p vortex-layout --all-targets --all-features -- -D warnings
cargo +nightly-2026-09-10 fmt -p vortex-layout --check
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --document-private-items -p vortex-layout
cargo test --locked -p vortex-layout --lib (492 passed)

Not run locally: workspace-wide clippy and tests, and the other CI suites (SQL logic, DuckDB, Python, CUDA).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

codspeed · 2026-10-13T12:10:38Z

Merging this PR will regress 1 benchmark

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

2 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

density_sweep_dense_runs[0.9]

bench_compare_sliced_dict_primitive[(3333, 10000)]

⚠️

24 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

random_i128[0.01]

test_repeated_scan_scalar_at[int_str]

test_repeated_scan_scalar_at[int_str_float]

test_scan_scalar_at[int_str_float]

test_repeated_scan_scalar_at[int_float]

test_repeated_scan_filter[int]

test_scan_filter[int_str_float]

test_scan_filter[int_float]

test_scan_read_all_to_arrow_array[int_float]

test_scan_filter[int_str]

…and 14 more

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 15 improved benchmarks
❌ 1 regressed benchmark
✅ 2243 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`test_scan_filter[int]`	3.1 ms	3.4 ms	-10.72%
⚡	Simulation	`optimize_lookup_predicate[ids=1, shape=balanced_or]`	50.8 µs	20 µs	×2.5
⚡	Simulation	`optimize_lookup_predicate[ids=1, shape=in_list]`	90.1 µs	39.6 µs	×2.3
⚡	Simulation	`optimize_lookup_predicate[ids=256, shape=in_list]`	105 µs	52.3 µs	×2
⚡	Simulation	`optimize_lookup_predicate[ids=16, shape=in_list]`	69.4 µs	42 µs	+65.15%
⚡	Simulation	`optimize_lookup_predicate[ids=64, shape=in_list]`	70.7 µs	42.9 µs	+64.8%
⚡	Simulation	`optimize_lookup_predicate[ids=128, shape=in_list]`	73.3 µs	45.9 µs	+59.84%
⚡	Simulation	`optimize_lookup_predicate[ids=192, shape=in_list]`	75.8 µs	47.9 µs	+58.31%
⚡	Simulation	`optimize_lookup_predicate[ids=16, shape=balanced_or]`	87 µs	55.9 µs	+55.68%
⚡	WallTime	`mul_u64_nonnull_neon`	39.5 µs	28.6 µs	+37.92%
⚡	Simulation	`optimize_lookup_predicate[ids=64, shape=balanced_or]`	194.2 µs	162.6 µs	+19.46%
⚡	WallTime	`multiply_shapes_neon[(32768, PerRowPerRow)]`	38.2 µs	32.5 µs	+17.8%
⚡	WallTime	`mul_i64_nonnull_neon`	38.2 µs	32.4 µs	+17.74%
⚡ ⚠️	Simulation	`random_i128[0.01]`	6.2 µs	5.4 µs	+15.27%
⚡	WallTime	`bitpack_blocked_compress_avx2`	7.6 µs	6.8 µs	+12.12%
⚡	Simulation	`optimize_lookup_predicate[ids=128, shape=balanced_or]`	335.4 µs	301.8 µs	+11.13%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/scan-v2-plan (68c37d4) with develop (845511f)¹}

No successful run was found on develop (5f31505) during the generation of this report, so 845511f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…ysical plans Adds the plan operators the planner-driven scan lowers to, compiled to the push-based pipelines: - `Zoned` plans: data plus zone statistics, which an expression over statistics rewrites into a pruning plan that proves per-zone results once and expands them to the selected rows. - `Share` plans, which canonicalize a whole child once and serve its rows. - `Take` keeps whether its values may be reused, and the expression rule pushes the negative-cost part of an expression beneath a share of dictionary values. - `Eval` applies compound infallible predicates to encoded values (dictionary, run-end, PCO) before expanding them. - `ValueCache` keeps whole values, such as dictionary values, between the scans that share it; its owner bounds its lifetime. - `WholeSource` collects an inlet whole before producing one batch. - Scans only count shared segments over the rows their splits read. - Flat layouts lower to a filtered segment scan, and zoned layouts lower. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko Signed-off-by: Claude <[email protected]>

EvalPlan::apply rewrote decoded batches (dictionary, run-end, PCO) before evaluating compound predicates. That is array-level optimization: apply_bound already runs the dictionary push-down and codes pull-up rules, and the remaining gaps belong in vortex-array and the encoding crates, not in a plan operator. EvalStage goes back to applying the bound expression, and vortex-layout no longer depends on vortex-pco. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

…nto Zoned Legacy `vortex.stats` layouts now lower to a `LegacyStats` plan instead of a `Zoned` plan carrying the legacy stat list. The optimizer rewrites it: - into its data alone when a `Zoned` plan already summarises the same rows, whether above or beneath it, so legacy stats are used only when no aggregate zone map exists; - into its data alone for a zero zone length or a table with no stat an aggregate can stand for; - otherwise into `Zoned(data, Eval(convert, zones))`, where `convert` packs each legacy stat column under its aggregate's name. `Zoned` therefore only ever sees the aggregate schema: it drops its legacy field and accessor, prunes through `ZoneMap::try_new` alone, and `ZoneMap::try_new_legacy` is test-only again. Unoptimized, a `LegacyStats` plan reads its data and prunes nothing. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

ValueCache keyed its map by `PlanRef::addr()`, a raw usize, and relied on storing a clone of the plan in each value so that address could not be freed and reused by another plan. `PlanByPtr` holds the plan and hashes and compares by pointer, so the key itself keeps the plan alive and an address can never name a different plan. `PlanRef::addr` is removed. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

`lower` is how a layout tree becomes a physical plan, not only a test fixture builder. Also applies rustfmt to the legacy stats changes. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

develop moved `SharedArray` from `vortex_array::arrays` to `crate::arrays::shared`, constructed through `Shared::new`, which broke the `Share` plan once merged. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

robert3005 · 2026-10-15T14:15:24Z

+}
+
+impl TakeData {
+    fn for_values(values: &PlanRef) -> VortexResult<Self> {


I think because of the rewrite this can mark dynamic values as cacheable. The example is Eval(dynamic, Concat(...)) into Concat(Eval(dynamic, ...), ...) but not sure whether this actually fits into this flow

robert3005 · 2026-10-15T14:16:38Z

            return Ok(None);
+        };
+        let values = child.values()?;
+        let values = match values.as_opt::<Share>() {


Values might have different nullability than the overall plan

robert3005 · 2026-10-15T14:19:45Z

+        None
+    }
+
+    fn with_children(


do you want to invalidate the cache here?

robert3005 · 2026-10-15T14:22:01Z

        let Some(codes) = compiler.compile(&plan.codes()?, rows, mask)? else {
            return Ok(None);
        };
+        let cache = plan.cacheable().then(|| compiler.values().clone());


I think there's double caching here? If you add Share you will cache there and here

robert3005 · 2026-10-15T14:28:59Z

    })
 }
+
+fn lower_zoned(layout: &LayoutRef) -> VortexResult<PlanRef> {


we should fix the base case here zone_len == 0 and the stats dtype is {}

robert3005

I think just lower and dict rewriting is what we need to fix here. The caching mostly needed explaining

- A take's values are uncacheable when a dynamic comparison sits anywhere beneath them, not only in an `Eval` directly over them: rewrites push expressions down, as into the chunks of a concatenation. - An expression pushed onto a take's values is rebound to their dtype. Codes that may be null make the take nullable while its values are not, and the expression bound to the take failed to bind to the values, failing the whole optimization. Push-down is declined when the rebound take would change the dtype. - A pruning `Zoned` rebuilt over new zones discards the proof over the old ones. - Zoned and legacy stats layouts with no zones, or a zone table holding no statistics, lower to their data. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

An expression holding a dynamic comparison now stays above a take. Kept there it reads the latest bound on every batch, and the dictionary batches the take emits still evaluate it over their values; pushed onto the values it was fixed per split and made the values uncacheable. With nothing dynamic ever beneath a take's values, `TakeData` and its cacheability check go: a take's values are always kept between scans. Signed-off-by: Claude <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PV5czm45tDpWBSSdgPPuCm

This was referenced Oct 13, 2026

feat(file): a coalescing IO service for planning-protocol scans #10463

Closed

feat(scan): planner-driven V2 scans on the physical plan pipelines #10464

Closed

joseph-isaacs mentioned this pull request Oct 13, 2026

do not merge(scan): planning, physical plans, and plan execution #10339

Closed

joseph-isaacs force-pushed the ji/scan-v2-plan branch from ad53eaf to d4d0acd Compare October 13, 2026 12:52

joseph-isaacs mentioned this pull request Oct 13, 2026

do not merge(scan): V2 scans on physical plan pipelines — full change #10466

Closed

joseph-isaacs added the changelog/feature A new feature label Oct 13, 2026 — with Claude

claude added 5 commits October 14, 2026 14:28

Merge remote-tracking branch 'origin/develop' into ji/brave-knuth-qpv33h

7a924a0

joseph-isaacs marked this pull request as ready for review October 15, 2026 10:06

joseph-isaacs requested a review from robert3005 October 15, 2026 10:07

robert3005 reviewed Oct 15, 2026

View reviewed changes

robert3005 approved these changes Oct 15, 2026

View reviewed changes

claude added 2 commits October 15, 2026 16:11

joseph-isaacs deployed to duckdb-build October 15, 2026 16:48 — with GitHub Actions Active

joseph-isaacs merged commit 18aebea into develop Oct 15, 2026
93 of 94 checks passed

joseph-isaacs deleted the ji/scan-v2-plan branch October 15, 2026 17:43

joseph-isaacs mentioned this pull request Oct 16, 2026

feat(layout): planner-driven V2 scan over physical plans #10550

Closed

This was referenced Oct 17, 2026

Epic: Refactor Statistics #10175

Open

Tracking Issue: Retire the legacy statistics APIs #10177

Open

joseph-isaacs mentioned this pull request Nov 9, 2026

Epic: Plan (replacing LayoutReaders) #9062

Open

16 of 25 tasks

Navigation Menu

Repository navigation

feat(layout): zoned pruning, shares, and dictionary value reuse in physical plans - #10462

feat(layout): zoned pruning, shares, and dictionary value reuse in physical plans#10462
joseph-isaacs merged 9 commits into
developfrom
ji/scan-v2-plan

joseph-isaacs commented Oct 13, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 13, 2026 •

edited

Loading

Uh oh!

robert3005 Oct 15, 2026

Uh oh!

robert3005 Oct 15, 2026

Uh oh!

robert3005 Oct 15, 2026

Uh oh!

robert3005 Oct 15, 2026

Uh oh!

robert3005 Oct 15, 2026

Uh oh!

robert3005 left a comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

joseph-isaacs commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

codspeed Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 1 benchmark

Performance Changes

Footnotes

Uh oh!

robert3005 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joseph-isaacs commented Oct 13, 2026 •

edited

Loading

codspeed Bot commented Oct 13, 2026 •

edited

Loading