Skip to content

feat(layout): compile evals, row indices, and takes to pipelines - #10393

Merged
joseph-isaacs merged 1 commit into
developfrom
ji/exec-4-eval-row-idx-take-nodes
Oct 12, 2026
Merged

feat(layout): compile evals, row indices, and takes to pipelines#10393
joseph-isaacs merged 1 commit into
developfrom
ji/exec-4-eval-row-idx-take-nodes

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Compiles expressions, row indices, and dictionary takes to pipelines. Tracking: #10349. Stacked on #10392.

  • Eval applies its expression to each input batch and returns Step::Final; Input::End returns Step::End.
  • RowIdx emits selected global row indices through the shared one-batch source.
  • Take implements Source: Operator. Its ready() only checks that dictionary values are complete and codes are available or finished. Its non-blocking compute() assembles and retains shared dictionary values, then wraps each codes batch as a dictionary. The values inlet remains unbounded because the source needs the complete dictionary.

All operators use the inherited Input::{None, Batch, End} and Step::{End, More, Final} protocol. Readiness handles waiting; computation handles batch construction.

Existing tests cover waiting for complete dictionary values, predicates pushed onto values, and expressions evaluated only on selected rows. Runtime continuation and flushing tests are inherited from #10391.

Physical plans and their optimizer live under vortex_layout::plan::physical. The sibling operator module defines Operator, Source, and the execution context; operators contains concrete implementations; pipeline owns compilation, scheduling, and ports.

Validation: cargo +nightly-2026-09-10 fmt -p vortex-layout; scoped cargo clippy -p vortex-layout --all-targets --all-features -- -D warnings; targeted physical-plan tests (97 passed).

@codspeed

codspeed Bot commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.15%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

⚠️ 10 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 2211 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime bitpack_blocked_compress_avx512 4.8 µs 5.6 µs -14.07%
⚡ WallTime threshold_avx512[8] 2.1 µs 1.9 µs +11.42%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

Comparing ji/exec-4-eval-row-idx-take-nodes (d5fc0af) with develop (62775a6)1

Open in CodSpeed

Footnotes

  1. No successful run was found on develop (2651b89) during the generation of this report, so 62775a6 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@joseph-isaacs
joseph-isaacs added this pull request to stack #10395 October 8, 2026 16:01
@joseph-isaacs
joseph-isaacs force-pushed the ji/exec-4-eval-row-idx-take-nodes branch from 7a91045 to b21ac66 Compare October 8, 2026 20:16
@joseph-isaacs joseph-isaacs mentioned this pull request Oct 9, 2026
Closed
@joseph-isaacs joseph-isaacs changed the title feat(layout): add Eval, RowIdx, and Take exec nodes feat(layout): compile evals, row indices, and takes to pipelines Oct 9, 2026
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Oct 12, 2026 — with Claude
@joseph-isaacs joseph-isaacs mentioned this pull request Oct 12, 2026
Merged
@joseph-isaacs Claude

joseph-isaacs commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

The failing CodSpeed Performance Analysis check is not from this PR.

The one regression is the WallTime compare_int_neon in vortex-array/benches/compare.rs, measured on hosted runners that CodSpeed flags as inconsistent and in different runtime environments. This PR changes only vortex-layout (the eval, row-index and take pipeline operators), which that benchmark doesn't reach. The same report shows bitpack_blocked_compress_avx2 +12.65%, the mirror of its -12.5% on #10391, which is runner noise.

There is no fix to port.

Generated by Claude Code

@joseph-isaacs joseph-isaacs mentioned this pull request Oct 12, 2026
Merged
Base automatically changed from ji/exec-3-scan-concat-pack-nodes to develop October 12, 2026 15:27
@joseph-isaacs @claude
feat(layout): add Eval, RowIdx, and Take exec nodes
d5fc0af 
- `EvalNode` applies its bound expression to each array its child
  produces as it streams through. The child runs over the same
  selection, so the expression never sees an unselected row.
- `RowIdxNode` emits the global row index of every selected row at
  start, offset by the graph's row offset.
- `TakeNode` runs the codes over the selection and the values over their
  whole domain. It is `Ready::Closed(values)`: it does not run until the
  values have closed, joins them once, and then wraps each codes array
  as a dictionary over them as it arrives.

Signed-off-by: Joe Isaacs <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NWYmfHTFtaEdhdyd3u5hv3
Signed-off-by: Joe Isaacs <[email protected]>
@joseph-isaacs
joseph-isaacs force-pushed the ji/exec-4-eval-row-idx-take-nodes branch from 70332ea to d5fc0af Compare October 12, 2026 15:28
@joseph-isaacs
joseph-isaacs merged commit 84e0d73 into develop Oct 12, 2026
86 of 87 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/exec-4-eval-row-idx-take-nodes branch October 12, 2026 15:36
This was referenced Oct 13, 2026
Merged
Closed
Closed
@joseph-isaacs joseph-isaacs mentioned this pull request Nov 9, 2026
Open
16 of 25 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robert3005 robert3005 robert3005 approved these changes

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@joseph-isaacs @robert3005