Skip to content

Add an engine-independent Scan API driver - #10926

Closed
gatesn wants to merge 1 commit into
developfrom
ngates/scan-api-driver
Closed

Add an engine-independent Scan API driver#10926
gatesn wants to merge 1 commit into
developfrom
ngates/scan-api-driver

Conversation

@gatesn

@gatesn gatesn commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Scan API consumers currently have to implement their own partition driver. Add vortex_scan::driver::ScanDriver to produce lazy array streams with bounded partition read-ahead, independent output consumers, ordered concatenation, exact limits, and cancellation. Partition implementations retain ownership of runtime scheduling; the driver spawns no tasks.

Reuse the driver in the DataFusion v2 source, leaving Arrow conversion and engine-specific planning in the adapter.

Validation: three focused driver tests, the existing DataFusion v2 tests, the scan doctest, and scoped all-targets/all-features Clippy pass.

Generated with Codex.

@gatesn @codex
Add an engine-independent Scan API driver
33c2702 
Share bounded partition execution, ordering, limits and cancellation between direct array consumers and the DataFusion v2 source.

Co-authored-by: Codex <[email protected]>
Signed-off-by: Nicholas Gates <[email protected]>
@gatesn gatesn added the changelog/feature A new feature label Oct 28, 2026 — with ChatGPT Codex Connector
@gatesn gatesn closed this Oct 28, 2026
@codspeed

codspeed Bot commented Oct 28, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 29.47%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 5 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 4 improved benchmarks
✅ 2200 untouched benchmarks
⏩ 370 skipped benchmarks1
🗄️ 30 archived benchmarks run2

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime threshold_neon[64] 4.1 µs 2.6 µs +60.7%
⚡ Walltime threshold_neon[80] 4.7 µs 3.5 µs +34.28%
⚡ Walltime filter_neon[80] 2.8 µs 2.4 µs +17.06%
⚡ Walltime filter_neon[64] 2 µs 1.8 µs +11.22%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

Comparing ngates/scan-api-driver (33c2702) with develop (9a844c1)

Open in CodSpeed

Footnotes

  1. 370 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 30 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

@TanishGudise TanishGudise mentioned this pull request Oct 28, 2026
Merged
@robert3005
robert3005 deleted the ngates/scan-api-driver branch November 10, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gatesn