Skip to content

Move scalar functions out of vortex-array into vortex-functions - #10762

Open
robert3005 wants to merge 1 commit into
developfrom
rk/split-array-06-scalar-fns
Open

robert3005 wants to merge 1 commit into
developfrom
rk/split-array-06-scalar-fns

Conversation

@robert3005

@robert3005 robert3005 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

vortex-array keeps the scalar function machinery (the ScalarFnVTable trait, binding, reduction, execution, the session registry and kernel registry) and the few functions its own arrays, validity and expressions are built on: literal, get_item, pack and select, is_null, is_not_null, not and the binary operators, and cast, mask and fill_null, which canonical arrays apply to their own children. That module is renamed from scalar_fn::fns to scalar_fn::builtin, matching ArrayBuiltins. Every other scalar function moves to the new vortex-functions crate:

  • byte_length, case_when, ext_storage, list_length, list_sum, list_transform, merge, replace_time_zone, dynamic, dynamic_expression, stat, between, like, list_contains, variant_get, is_nan and zip, with their expression constructors in vortex_functions::expr (and expr::bound)
  • their kernels and parent-reduce rules for vortex-array's encodings, including Widen's, which vortex_functions::initialize registers with the session's kernel registry
  • the ArrayBuiltins/ExprBuiltins methods for the moved functions, which become the ArrayFunctions/ExprFunctions extension traits

ArrayHaystack, LazyBitBufferBuilder, DeduplicatedBuffers::{extend_from_iter, finish}, ListScalar::element_values, is_not_null_node, build_row_comparator, merge_typed_scalar_as_variant and the Widen compute helpers become public for the moved kernels; ForeignExtDType is public under _test-harness for the moved tests.

@robert3005 robert3005 added the changelog/break A breaking API change label Oct 23, 2026
@robert3005
robert3005 marked this pull request as draft October 23, 2026 22:47
@robert3005
robert3005 force-pushed the rk/split-array-06-scalar-fns branch from 714ed9d to fd2bbf1 Compare October 23, 2026 22:57
@codspeed

codspeed Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 6 benchmarks

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 3 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 5 improved benchmarks
❌ 6 regressed benchmarks
✅ 1908 untouched benchmarks
🆕 64 new benchmarks
⏩ 67 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation filter_random_by_random[10000] 7.1 µs 10.2 µs -30.54%
❌ CPU Simulation isolated_runs[u8, (161, 0)] 65.5 µs 82.8 µs -20.86%
❌ CPU Simulation compare[1] 47.6 µs 57.4 µs -17.15%
❌ CPU Simulation bench_compare_varbin[(50000, 2)] 134.7 µs 162.4 µs -17.05%
❌ CPU Simulation bench_compare_sliced_dict_varbinview[(1000, 10000)] 81.4 µs 92.1 µs -11.56%
❌ CPU Simulation bench_compare_sliced_dict_varbinview[(9999, 10000)] 206.8 µs 233.3 µs -11.36%
⚡ CPU Simulation random_i128[0.01] 36 µs 7.4 µs ×4.8
⚡ CPU Simulation random_i128[0.5] 124.9 µs 97.4 µs +28.3%
⚡ CPU Simulation random_i128[0.8] 150.7 µs 123.1 µs +22.39%
⚡ CPU Simulation compare[1] 71.4 µs 61.4 µs +16.2%
⚡ CPU Simulation encode_primitives[f32, (2000, 2)] 208.6 µs 180.9 µs +15.29%
🆕 CPU Simulation nonnull N/A 105.8 µs N/A
🆕 CPU Simulation nullable N/A 184 µs N/A
🆕 CPU Simulation case_when_all_false[1000] N/A 112.7 µs N/A
🆕 CPU Simulation case_when_all_false[10000] N/A 185.3 µs N/A
🆕 CPU Simulation case_when_all_false[40000] N/A 427.5 µs N/A
🆕 CPU Simulation case_when_all_true[1000] N/A 114.4 µs N/A
🆕 CPU Simulation case_when_all_true[10000] N/A 184.4 µs N/A
🆕 CPU Simulation case_when_all_true[40000] N/A 426.9 µs N/A
🆕 CPU Simulation case_when_decimal_product[1] N/A 232 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/split-array-06-scalar-fns (29fe7c3) with develop (26a12a6)

Open in CodSpeed

Footnotes

  1. 67 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005
robert3005 added this pull request to stack #10770 October 24, 2026 12:37
@robert3005
robert3005 force-pushed the rk/split-array-06-scalar-fns branch 4 times, most recently from 0713434 to e33c436 Compare October 26, 2026 11:25
@robert3005
robert3005 force-pushed the rk/split-array-06-scalar-fns branch 4 times, most recently from 595165d to 7644dd2 Compare November 9, 2026 03:32
@robert3005
robert3005 marked this pull request as ready for review November 9, 2026 10:30
@robert3005

Copy link
Copy Markdown
Contributor Author

The follow up to this is #10768 which flips the dependency so it's vortex-array -> vortex-expr -> vortex-functions and then #10769

@robert3005
robert3005 force-pushed the rk/split-array-06-scalar-fns branch 4 times, most recently from dcaa482 to 9383bb2 Compare November 10, 2026 12:55
@connortsui20 connortsui20 mentioned this pull request Nov 12, 2026
12 of 49 tasks
vortex-array keeps the scalar function machinery (the `ScalarFnVTable` trait,
binding, reduction, execution, the session registry and kernel registry) and
the few functions its own arrays, validity and expressions are built on:
literal, get_item, pack and select, is_null, is_not_null, not and the binary
operators, and cast, mask and fill_null, which canonical arrays apply to their
own children. That module is renamed from `scalar_fn::fns` to
`scalar_fn::builtin`, matching `ArrayBuiltins`. Select stays because get_item
reads through a selection. Every other scalar function moves to the new
`vortex-functions` crate:

- byte_length, case_when, ext_storage, list_length, list_sum, list_transform,
  merge, replace_time_zone, dynamic, dynamic_expression, stat, between, like,
  list_contains, variant_get, is_nan and zip, with their expression
  constructors in `vortex_functions::expr` (and `expr::bound`)
- the kernels and parent-reduce rules for vortex-array's encodings, including
  `Widen`'s between and zip kernels, which `vortex_functions::initialize`
  registers with a session's kernel registry
- the `ArrayBuiltins`/`ExprBuiltins` methods for the moved functions, which
  become the `ArrayFunctions`/`ExprFunctions` extension traits

`ArrayHaystack`, `LazyBitBufferBuilder`,
`DeduplicatedBuffers::{extend_from_iter, finish}`, `ListScalar::element_values`,
`is_not_null_node`, `build_row_comparator`, `merge_typed_scalar_as_variant` and
the `Widen` compute helpers become public for the moved kernels, and
`ForeignExtDType` is public under `_test-harness` for the moved tests.

The umbrella crate re-exports the moved constructors under `vortex::expr` and
the crate as `vortex::functions`. vortex-array's integration tests and benches
use vortex-functions as a dev-dependency.

The CodSpeed shard plan picks up the benchmarks that move to `vortex-functions`;
their recorded counts and durations move with them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ps4mp7KF5c4p66aoSoXx5j
Signed-off-by: Claude <[email protected]>
@robert3005
robert3005 force-pushed the rk/split-array-06-scalar-fns branch from 9383bb2 to 29fe7c3 Compare November 13, 2026 18:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants