Skip to content

feat(scan): planner-driven V2 scans on the physical plan pipelines - #10464

Closed
joseph-isaacs wants to merge 4 commits into
ji/scan-v2-iofrom
ji/scan-v2-planning
Closed

feat(scan): planner-driven V2 scans on the physical plan pipelines#10464
joseph-isaacs wants to merge 4 commits into
ji/scan-v2-iofrom
ji/scan-v2-planning

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Third of three stacked PRs carved out of #10339: plan (#10462), IO (#10463), then planning (this PR).

Adds an opt-in scan (VORTEX_SCAN_V2=1). It lowers a file's layout once into a physical plan and runs each split through planners on the planning protocol, executing every plan on develop's pipeline Scan. #10339's own ExecGraph and PipelineGraph executors are gone.

On a 4-core VM, DataFusion over Vortex at SF=1, V2 vs the default scan:

Suite Geomean of per-query medians (V2/V1)
TPC-H 0.93x
TPC-DS 0.94x

I reran every apparent loss with alternating V1/V2 runs; none were reproducible slowdowns.

Changes

  • vortex-scan. A planning driver that runs planners and morsels, publishes their IO through an IoService, and traces their work.
  • vortex-file.
  • vortex-layout.
    • Announce, pruning, filter and projection planners. The filter planner orders conjuncts adaptively.
    • Each stage of a split runs as one pipeline Scan.
    • A split's scans share a ValueCache that is dropped with the split, so dictionary values are never kept for the rest of a scan.
    • The V2 ScanBuilder and RepeatedScanV2, plan-derived splits, early IO announcements and file pruning.
  • Engines.
    • DataFusion and DuckDB select V2 when it is enabled and supported.
    • Files with list columns, and scans combining a filter with a limit, use the default executor: a list's element range is only known after its offsets are read, and pipelines are compiled before any read.
    • DuckDB can prepare later files while earlier ones scan (VORTEX_DUCKDB_FILE_PREFETCH, off by default).
    • DataFusion reports predicate and IO diagnostics.

Checks run:

  • cargo clippy -p vortex-scan -p vortex-file -p vortex-layout -p vortex-datafusion --all-targets --all-features -- -D warnings
  • cargo +nightly-2026-09-10 fmt
  • cargo test --lib for those four crates, with and without VORTEX_SCAN_V2=1

Not run: vortex-duckdb was not built, because its build downloads the DuckDB source and that download is blocked in this environment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko

Generated by Claude Code

@codspeed

codspeed Bot commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.1%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

⚠️ 12 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

❌ 1 regressed benchmark
✅ 2212 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime bitpack_blocked_compress_avx2 6.8 µs 7.6 µs -11.1%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

Comparing ji/scan-v2-planning (42b29cc) with ji/scan-v2-io (3c37da1)

Open in CodSpeed

@joseph-isaacs joseph-isaacs mentioned this pull request Oct 13, 2026
Closed
@claude
feat(layout): zoned pruning, shares, and dictionary value reuse in ph…
d4d0acd 
…ysical plans

Adds the plan operators the planner-driven scan lowers to, compiled to the
push-based pipelines:

- `Zoned` plans: data plus zone statistics, which an expression over
  statistics rewrites into a pruning plan that proves per-zone results once
  and expands them to the selected rows.
- `Share` plans, which canonicalize a whole child once and serve its rows.
- `Take` keeps whether its values may be reused, and the expression rule
  pushes the negative-cost part of an expression beneath a share of
  dictionary values.
- `Eval` applies compound infallible predicates to encoded values
  (dictionary, run-end, PCO) before expanding them.
- `ValueCache` keeps whole values, such as dictionary values, between the
  scans that share it; its owner bounds its lifetime.
- `WholeSource` collects an inlet whole before producing one batch.
- Scans only count shared segments over the rows their splits read.
- Flat layouts lower to a filtered segment scan, and zoned layouts lower.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko
Signed-off-by: Claude <[email protected]>
@joseph-isaacs joseph-isaacs mentioned this pull request Oct 13, 2026
Closed
claude added 2 commits October 13, 2026 13:07
@claude
feat(file): a coalescing IO service for planning-protocol scans
3c37da1 
Serves the planning protocol's `IoService` reads of a file:

- `FileIoService` coalesces the ranges a batch of requests or announcements
  asks for, reads each range once for every session interested in it, and
  withdraws a dropped session's interest. `FileScanIo` serves the same
  protocol through the file's segment source and read driver.
- Local files reuse their handles and admit native reads up to a bound;
  object-store reads stream completions as they land.
- The read driver keeps its coalescing scratch between passes and can
  leave optional requests out of coalescing.
- Segment caches can answer synchronously from `get_if_ready`.
- Request tracing ids for diagnostics.

A file opened over a reader carries the service as `VortexFile::scan_io`.
Nothing drives it yet; the planning scan that does follows separately.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko
Signed-off-by: Claude <[email protected]>
@claude
feat(scan): planner-driven V2 scans on the physical plan pipelines
42b29cc 
An opt-in scan (`VORTEX_SCAN_V2=1`) that lowers a file's layout once into a
physical plan and runs each split through planners on the planning
protocol:

- vortex-scan: a planning driver that runs planners and morsels, publishes
  their IO through an `IoService`, and traces their work.
- vortex-file: footer open and footer prune stages, and `scan_file` to
  describe a file to the scan.
- vortex-layout: announce, pruning, filter (adaptive conjunct order) and
  projection planners whose plans run on the pipeline `Scan`, one scan per
  stage of a split, with a `ValueCache` shared by a split's scans and
  dropped with it. The V2 `ScanBuilder`/`RepeatedScanV2`, plan-derived
  splits, early IO announcements and file pruning.
- DataFusion and DuckDB select V2 when it is enabled and supported. Files
  with list columns, and scans combining a filter with a limit, use the
  default executor.
- DuckDB can prepare later files while earlier ones scan
  (`VORTEX_DUCKDB_FILE_PREFETCH`, off by default); DataFusion reports
  predicate and IO diagnostics.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko
Signed-off-by: Claude <[email protected]>
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Oct 13, 2026 — with Claude
This was referenced Oct 15, 2026
Closed
Closed
Merged
Closed
Closed
Closed
Closed
@joseph-isaacs Claude

joseph-isaacs commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded: the planning layer was redone and merged as #10706–#10708. It is now the PrunePlanner → ChunkPlanner → FilterPlanner → ProjectionPlanner chain in vortex-layout/src/scan/execution/.

Generated by Claude Code

@robert3005
robert3005 deleted the ji/scan-v2-planning branch November 10, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@joseph-isaacs @claude