Repository navigation
Plan and execute common RowFn batches in one dispatch - #10675
Conversation
Merging this PR will degrade performance by 10.42%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
55d9af7 to
178dcd8
Compare
178dcd8 to
5cbf00a
Compare
Polar Signals Profiling ResultsLatest Run
Previous Runs (7)
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.977x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.022x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 2↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.083x ➖, 1↑ 5↓)
datafusion / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.069x ➖, 0↑ 1↓)
|
5cbf00a to
7c47d4b
Compare
7c47d4b to
69f5fc0
Compare
## Summary - Tracking Issue: #9128 - Independent validity candidate: #10742 Routes float32 comparisons through the existing word-packing collectors. This recovers the measured float32 gains from the RowFn comparison investigation with two private dispatch changes. It is independent of the comparison migration and fusion work in #10676 and #10675. ## Changes Adds `F32` to the existing collector selections for array pairs and array/constant comparisons. Integer collector policies remain unchanged. The floating-point documentation now accounts for NaN signs and payloads. **Draft pending fresh isolated benchmarks on this head.** Saved measurements show a consistent float32 gain, but also some unrelated ARM regressions. The later combined packing/validity candidate had an AVX2 regression that has not been attributed to either change individually. <details> <summary>Saved benchmark evidence and validation</summary> The isolated packing snapshot `4c98df49e379853f8e990d9764791b5330511771` was compared with frozen `develop` at `a1693389301e446eec41373cb274d98a303d3ca0`. Measurements used Rust 1.99.1 / LLVM 23.1.1, 16 CGUs, no LTO, CPU 2, and 21 balanced paired blocks on Graviton3 C7g and Sapphire Rapids C7i. This was native wall time, not CodSpeed simulation. Across 48 float32 cases at 8,192 and 65,536 rows, covering array pairs, both constant orientations, equality/ordering, and four validity shapes, the median block ratios of equally weighted geometric means were: | Target | Candidate / baseline time | 95% paired bootstrap interval | | --- | ---: | ---: | | NEON | 0.611 | 0.610–0.611 | | AVX2 | 0.632 | 0.630–0.633 | | AVX-512 | 0.679 | 0.678–0.681 | Among all 375 probe cases, the isolated snapshot had no flagged regressions on AVX2 or AVX-512. ARM had four, including roughly 4.7% slower u64 multiply and 4.6% slower large f64 equality. Those paths do not use the changed float32 collector, so their movement is not evidence of extra float32 work, but it still counts against a no-regression claim. A separate combined packing/validity candidate at `8d8ef8151454bbc73e1cd0b236567542a1401cbe`, based on `d834a11faa79370b4d2e0bbcc73195f451a8a212`, retained the float32 gain but regressed large nullable u64 add by about 18% on AVX2. Profiling found a byte-identical retry hot loop with essentially unchanged instructions per operation and more cycles. The precise placement-sensitive mechanism remains unresolved. That combined result does not isolate this PR. The combined candidate passed 225 RowFn tests and 232 binary-function tests on that newer base. Separate native probes checked scalar-reference results and float/encoding/tail edge cases. The changed file passed the pinned formatter. These are saved checks, not fresh validation of this individual PR. This PR is rebased onto `c750fbd80882f78efd6a2703bc0f3e17f1aca9f9`. The patch is unchanged by the rebase. Tests and benchmarks were not rerun after it. </details> Signed-off-by: Connor Tsui <[email protected]>
e8675b5 to
5cf73d7
Compare
322b0ae to
d624249
Compare
d624249 to
712ed09
Compare
712ed09 to
d3e04bb
Compare
Tracking Issue: #9128 ## Summary Prepares the RowFn batch types for #10675 without changing how a batch executes. Execution still plans with one dispatch and executes with a second one, as on `develop`. This is a purely mechanical change that should help with function call and access overhead. ## Changes - Adds `InputElement::decode_ref`, which decodes a borrowed input. The default clones the array for `decode`. Native primitives share their canonical buffer without executing the array. - Inspects the inputs once in `BatchProbe`: row counts, batch-constant classification, null literals, and the conjoined validity. Execution reads the probe instead of scanning the inputs again. A null literal skips the validity conjunction, because every row is null either way. - Keeps the columns in `BatchInputs`, which borrows them when the caller holds them as one slice. `RowFnExecutionArgs` borrows the batch and the dtypes instead of owning copies. `execute_row_inputs` runs over any `RowInputs`, and `execute_rows` adapts a general `ExecutionArgs` to it. - Skips the all-valid check in `finalize_kernel_output` when the output is not nullable, and marks the `CachedId` read `#[inline]`. Adds five regression cases covering invalid payloads, null short circuits, lazy division, device-backed validity, and device-backed primitive input. They pass on the existing execution flow, so #10675 shows that its single-dispatch path keeps that behavior. --------- Signed-off-by: Connor Tsui <[email protected]>
d3e04bb to
37c7a3e
Compare
37c7a3e to
b0a64dd
Compare
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
c83a86c to
a8403bc
Compare
Finish all-null, null-free, and dense batches in the planning visit, instead of dispatching a second time and checking that both dispatches chose the same plan. Hand every other batch back as a PartiallyValidBatch for dense retry or valid-row execution, which still dispatch again. Signed-off-by: Connor Tsui <[email protected]>
A static AVX-512 build inlines collect_bool_words_avx512 only when rustc emits that copy in the caller's codegen unit. This branch moved it out of the RowFn deferred Boolean kernel's unit, adding 15% more instructions per row. Such a build has nothing wider to detect, so use collect_bool_words_inline, which already packs with AVX-512. Signed-off-by: Connor Tsui <[email protected]>
a8403bc to
9beb5db
Compare
Tracking Issue: #9128 ## Summary Moves primitive comparisons onto the shared RowFn executor from #10675, so comparison code only selects the predicate while the framework handles decoding, constants, validity, and Boolean output. ## Changes Preserves the public `Binary` identity and floating-point total ordering. Uses borrowed operands and dtypes, and moves a constant left operand to the right with the swapped operator. Relies on the existing `primitive_comparisons_across_bitmap_words` test for all six swapped operators and keeps the existing comparison benchmarks. Based on #10675. Keeps the wide-type, sliced-input, bitmap-boundary, NaN, and signed-zero tests added upstream in #10843. ## Performance CodSpeed compares this PR with #10675 (`b0a64dd`). Nine primitive comparison benchmarks are 12% to 31% faster in walltime, led by `compare_u64_avx2` and `compare_float_avx2` at 31% and 28%, and no comparison benchmark regresses. These gains depend on #10675. #11220 measured the same two commits directly on `develop`. Without #10675, comparisons on small arrays were 10% to 16% slower than `develop` in CPU simulation, and only two walltime gains remained, at 16% and 18%. The #10675 description has the full comparison. The baseline already includes the private NEON packing kernel from #10843, which this PR replaces with RowFn execution. So the gains above are measured against that improved baseline, and the old packing gains aren't credited to this PR. <details> <summary>CodSpeed comparison benchmarks against #10675</summary> CodSpeed marks the walltime runners as unreliable, but all nine cases move in the same direction. | Benchmark | Faster by | | --- | ---: | | `compare_u64_avx2` | 30.8% | | `compare_float_avx2` | 28.2% | | `lt_i64_nullable_avx2` | 18.4% | | `compare_int_constant_neon` | 16.2% | | `compare_int_eq_avx2` | 15.4% | | `compare_f32_avx2` | 15.4% | | `compare_int_constant_avx2` | 15.3% | | `compare_int_avx2` | 13.5% | | `compare_int_nullable_avx2` | 12.5% | </details> <details> <summary>Historical EC2 accumulated-stack measurements</summary> These runs measured an earlier accumulated stack, not this layer in isolation or the current head. They showed gains on some configurations and regressions on others. Ratios are the median of 21 paired candidate/baseline elapsed-time ratios. Below 1 is faster. This compares the earlier complete migration, M0 `ddf8a16df6b7d7c20cfd860e5f23438d97ca4238`, with D `0291778db36809f77ce0219bf9850506d3cba3b6`. That baseline predates #10843. The array-pair cases use all-valid nullable inputs. The addition case is an arithmetic control whose algorithm was unchanged. | Workload | C7g NEON | C6a AVX2 | C7i AVX2 build | C7i AVX-512 build | | --- | ---: | ---: | ---: | ---: | | `f32` array-pair Gte, 8,192 rows | 1.014 | 1.040 | 0.873 | 1.036 | | `i64` array-pair Gte, 8,192 rows | 0.900 | 1.139 | 0.929 | 0.943 | | Nullable `i64` add, 32,768 rows | 1.002 | 1.123 | 1.154 | 1.004 | For example, the old stack took about 10% less time for the ARM `i64` comparison, but about 14% more for the same case on AVX2-only C6a. C6a nullable `i64` addition took about 12% more time. These whole-executable measurements do not establish that the comparison algorithm caused movement in arithmetic controls. In the four-label matrix, D ran before M0 in 11 blocks and after it in ten. The experimental one-pass decoder variants are excluded from this table. EC2 C7g (Neoverse V1), C6a (EPYC 7R13), and C7i (Xeon Platinum 8488C). Rust 1.99.1 / LLVM 23.1.1, bench profile with 16 CGUs and no LTO, MiMalloc, CPU 2 pinned. The same 455-case public-path microbenchmark suite ran serially across hosts, with 21 paired blocks, 10 ms per case, A/A calibration, and a separate control executable. These are microbenchmarks, not query results. C7i can dispatch to AVX-512 at runtime even in its AVX2 build. Saved evidence is in the local investigation archive `row-fn-evaluation-20261026`: `manifest.json`, `*-matrix-*-medians.csv`, `*-matrix-*-summary.json`, and `results/<host>/results/matrix-*.json`. </details> <details> <summary>Confirmed independent packing gains, now in develop via #10843</summary> The ARM-only private-kernel change reduced long array-pair cohort time on Graviton3 as follows. Each cohort contains 32 cases, including partially valid inputs and lengths around 16K. | Cohort | Confirmation elapsed-time ratio | Paired 95% interval | Time reduction | | --- | ---: | --- | ---: | | `i64` | 0.846 | 0.843 to 0.856 | 15.4% | | `u64` | 0.860 | 0.858 to 0.864 | 14.0% | | `f64` | 0.882 | 0.880 to 0.886 | 11.8% | Baseline `a59294e35e7109fac894a7a38a68f42db0c3c967`, candidate `f018aaeef739145e1bc50079a441c5d9fd53404a`. The cohort uses a geometric mean of case ratios within each block, then the median across blocks. Two 21-block sessions used complementary AB/BA orders, the same full 455-case suite, and 1,000 single-iteration samples per case. The initial fixed-order source comparisons were discarded. The AVX2 builds had identical `.text` bytes, so no x86 speedup was attributed to this change. See #10843 for the harness and full methodology. These gains required neither this PR nor #10675. </details> On `e724610`, CI passed all-target, all-feature Clippy with warnings denied and the Linux, CPU feature, sanitizer, and Miri test jobs. --------- Signed-off-by: Connor Tsui <[email protected]>
Tracking Issue: #9128
Summary
Combines planning and execution for eligible RowFn batches in one type dispatch. This is the shared framework layer for #10676, which contains the primitive comparison migration. It builds on #10899, which separates batch inspection from planning and adds the regression cases this PR must keep passing.
Changes
BatchProbehas classified all of them, so a later null literal still skips an earlier fallible expression or device transfer. Fallible decoders keep valid-row execution.PartiallyValidBatch, the only input that dense retry and valid-row execution accept, so every caller runs the fast paths first.collect_bool_words_multiversioneduses the inline packing loop instead of the#[target_feature]copy, so its inlining no longer depends on codegen-unit placement.Performance
This PR lowers the fixed cost of each RowFn call, so CodSpeed shows no gain for it alone. The gain shows up in #10676, which moves primitive comparison onto the RowFn executor. To isolate this PR, #11220 applied the two #10676 commits directly on
develop. Without this PR, comparisons on small arrays are 10% to 16% slower thandevelopin CPU simulation, including the zone map pruning benchmarks,i64_lt, andtimestamp_gte. The added time barely changes between 16 and 1,024 zones (18.0 µs and 17.3 µs forint_gt), so it is paid per call, not per row. On top of this PR, #10676 shows none of these regressions, and its walltime comparison gains are larger:compare_u64_avx2improves by 31% instead of 16%, and seven more comparison benchmarks improve by 12% to 18%.In the AVX-512 build, the RowFn changes moved the deferred Boolean kernel's AVX-512 packing loop out of its caller's codegen unit, so LLVM could no longer inline it. CodSpeed reported the two partially valid
deferred_bool_avx512cases as 42% and 16% slower. With the last commit, CodSpeed no longer flags either case. Builds that detect AVX-512 at runtime compile the same code as before. #11220 also showscompare_u8_avx512andcompare_f32_avx512as 14% and 10% slower without this PR, which matches the same packing problem, but that build's assembly wasn't inspected.CodSpeed on the current head flags three varbin comparison cases and
random_i8[0.01]as 12% to 18% slower, and onetake_filtercase as 13% faster. None of them runs a RowFn. #10676 reverses the varbin andtake_filterchanges without touching their code, so they follow code placement.random_i8[0.01]isn't a stable signal either: itsdevelopbaseline moved from 99.6 µs to 119.3 µs betweenbc58587andc9a3fbe, and #11220 moved it 13% the other way.CodSpeed results for #10676 with and without this PR
The first column is #11220, which compares the #10676 commits on
developwithdevelop(c9a3fbe). The second column is the #10676 report, which compares the same commits on this PR with this PR (b0a64dd). Againstdevelop(bc58587), this PR flags none of these benchmarks. CodSpeed marks the walltime runners as unreliable, and the two columns come from different runs.int_gt[16]int_gt[1024]float_gt[16]float_gt[1024]or_chain[16]or_chain[1024]is_not_null_pred[16]is_not_null_pred[1024]i64_lttimestamp_gtebench_compare_sliced_dict_primitive[(1000, 10000)]encode_primitives[f32, (2000, 2)]compare_u8_avx512compare_u8_neoncompare_f32_avx512compare_float_avx2compare_u64_avx2lt_i64_nullable_avx2compare_int_constant_neoncompare_int_eq_avx2compare_f32_avx2compare_int_constant_avx2compare_int_avx2compare_int_nullable_avx2AVX-512 investigation on EC2
C7i metal (Xeon Platinum 8488C) with the CodSpeed AVX-512 build flags, the bench profile, one isolated CPU, and 21 paired blocks of 1,000 samples. Ratios are the median paired ratio to
develop(8c87c4c457). Below 1 is faster. An A/A comparison of two identical binaries flags none of the 100 cases.deferred_bool_avx512casedevelopConstantLhs, PartialAcceptedColumns, PartialAcceptedBefore the fix, the kernel executed 15% more instructions per row. LLVM inline remarks show
collect_bool_words_avx512inlined ondevelopand "definition is unavailable" on this branch.Across all 100
#[cpu_features]cases, the fix also returnscompare_string_ltto thedevelopspeed. It makes twointersect_by_rankcases and onestarts_withcase 3.5% to 5.6% slower, but those execute the same instructions as before, so the difference is code placement. The AVX2 build is unchanged.Historical EC2 framework measurements
These numbers belong to an older framework replay and do not establish the performance of this revision. Ratios are the median of 21 paired candidate/baseline elapsed-time ratios. Below 1 is faster. This compares the framework alone, F
94be28140d5bd7e6b654af5e462253add2e49044, with D0291778db36809f77ce0219bf9850506d3cba3b6.u64multiply, 8,192 rowsi64add, 32,768 rowsThe actual framework pair order was imbalanced: D ran before F in 16 blocks and after it in five. Treat these figures as diagnostic observations, not reliable causal effect sizes.
EC2 C7g (Neoverse V1), C6a (EPYC 7R13), and C7i (Xeon Platinum 8488C). Rust 1.99.1 / LLVM 23.1.1, bench profile with 16 CGUs and no LTO, MiMalloc, CPU 2 pinned. The same 455-case public-path microbenchmark suite ran serially across hosts, with 21 paired blocks, 10 ms per case, A/A calibration, and a separate control executable. These are microbenchmarks, not query results. C7i can dispatch to AVX-512 at runtime even in its AVX2 build.
The separately confirmed NEON comparison gains were published in #10843, which has merged. They do not measure this framework.
CI passed on
b0a64dd, including all-target, all-feature Clippy with warnings denied and the Linux, Windows, sanitizer, and Miri test jobs. The CodSpeed check fails on the flags described above.