Repository navigation
Add ScalarFnVTable::execute_step for iterative scalar fn execution - #11100
Conversation
Scalar functions can now return ScalarFnExecutionResult::execute_input::<M>(idx) to ask the scheduler to execute an input until it matches M, instead of executing it recursively. The default execute_step delegates to execute, so existing functions are unchanged. ScalarFnArray maps the request to an ExecuteSlot step, and Not is migrated as the first user. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UeniBCqoK7v3TPh3sV6qER Signed-off-by: Joe Isaacs <[email protected]>
Merging this PR will degrade performance by 7.58%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
Signed-off-by: Joe Isaacs <[email protected]>
Signed-off-by: Joe Isaacs <[email protected]>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.019x ➖, 0↑ 1↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.023x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.067x ➖, 1↑ 4↓)
datafusion / parquet / ns (1.056x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.101x ➖, 0↑ 5↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Joe Isaacs <[email protected]>
## Summary Stacked on #11310. Until that merges, this PR shows only the last commit. `Lower` and `Strpos` decode their inputs inside `execute`, in a nested executor. When the input is a slice or a mask over a run-end or chunked array, the function gets a canonical array and runs once per row, even when the encoding could have pushed it into its values. #11310 lets the executor retry parent-reduce rules once a child is rewritten, but only for functions that hand their inputs back to the executor through `execute_step`. This continues the `execute_step` migration from #11100 and #11154 for `Lower` and `Strpos`. ## Changes - `Lower::execute_step` and `Strpos::execute_step` request their inputs with `require_input!(args, i => AnyColumnar)`, as `Like` and `Between` do, then call `execute`. - Tests: `lower`'s `preserves_runs_and_dictionary` gets a sliced case, and `strpos` gets `preserves_runs_of_a_slice`. Both check that a function over a sliced run-end input stays run-end encoded. Both fail without this change, and both fail without #11310. Checks: - `cargo test -p vortex-array` passes. - `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings` passes. - `cargo +nightly-2026-09-10 fmt -p vortex-array --check` passes. Benchmark numbers will follow. No existing benchmark covers scalar functions over sliced or masked encoded inputs. This change was written with help from an AI coding agent and I haven't reviewed it yet. --------- Signed-off-by: Robert Kruszewski <[email protected]> Signed-off-by: Alfonso Subiotto <[email protected]> Co-authored-by: Robert Kruszewski <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Amp <[email protected]> Co-authored-by: Robert Kruszewski <[email protected]>
Summary
ScalarFnArrayexecution is currently recursive:ScalarFn::executepasses unexecuted children toScalarFnVTable::execute, and each function executes its inputs itself viachild.execute::<X>(ctx). Deep trees of scalar functions therefore recurse instead of using the iterative executor's explicit stack.This PR adds the hook needed to move scalar functions onto the iterative executor, one function at a time. It is the first step of a longer migration;
notis moved over here as the first user.Changes
ScalarFnVTable::execute_step(new, with a default): performs one step of execution and returns aScalarFnExecutionResult. The default delegates toexecuteand wraps the result as done, so existing functions are unchanged.ScalarFnExecutionResult:done(array)finishes,execute_input::<M>(idx)asks the scheduler to execute inputidxuntil it matchesM, mirroringExecutionResult::execute_slot::<M>on the array side.ScalarFn::execute(array vtable) now callsexecute_stepand mapsExecuteInput(idx, pred)to anExecuteSloton the same array. It errors if the requested input already matches, which would otherwise loop until the iteration limit.DynScalarFnandScalarFnRefgainexecute_step. The length and dtype result checks are shared betweenexecuteandexecute_step, and an out-of-range input index is rejected.executor.rs: addspub(crate) ExecutionResult::execute_slot_until, which takes a runtimeDonePredicate.Notimplementsexecute_step: it inverts a constant orBoolinput, otherwise requestsexecute_input::<AnyColumnar>(0).executeshares the same helper.executeis still required and still used by callers outside the array executor (for example the bloom filter scalar fn invortex-layout), so a migrated function currently implements both methods.Tested with a new
nottest that executesnot(not(not(chunked)))to canonical. Note that a rewrite rule may push the chunked array below thenotbeforeexecute_stepruns, so the test checks the result but does not prove theExecuteInputpath was taken.Checks run:
cargo check -p vortex-arrayandcargo test -p vortex-array --lib scalar_fn::fns::not(6 passed). Not run: clippy, andfmtwith the pinned nightly (not installed in the environment).API Changes
Additive only: new
ScalarFnVTable::execute_stepmethod with a default implementation, new publicScalarFnExecutionResultandScalarFnExecutionSteptypes, and newScalarFnRef::execute_step. No existing signatures change.Generated by Claude Code