Skip to content

perf: collapse single-value partitions instead of a direct struct projection path - #10536

Merged
gatesn merged 3 commits into
developfrom
claude/stoic-cannon-p024ce
Oct 28, 2026
Merged

gatesn merged 3 commits into
developfrom
claude/stoic-cannon-p024ce

Conversation

@robert3005

@robert3005 robert3005 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Replace StructReader's Partitioned::Direct special case with a general
fix in partitioned evaluation:

  • BoundPartitionedExpr::collapse_single_partitions unwraps partitions that
    hold one sub-expression (pack(a_0: e) becomes e) and rewrites the root's
    $.a.a_0 accesses to $.a. Field readers then evaluate $ directly
    instead of wrapping every field in a one-field struct.
  • Partitions are collected in the order the expression references them
    rather than hash order.
  • The new PartitionedEval caches the collapsed expression together with
    RootFields. When the root only packs partition results into a
    non-nullable struct, projection assembles the StructArray directly
    instead of applying the root expression on every split.

…jection path

Replace `StructReader`'s `Partitioned::Direct` special case with a general
fix in partitioned evaluation:

- `BoundPartitionedExpr::collapse_single_partitions` unwraps partitions that
  hold one sub-expression (`pack(a_0: e)` becomes `e`) and rewrites the root's
  `$.a.a_0` accesses to `$.a`. Field readers then evaluate `$` directly
  instead of wrapping every field in a one-field struct.
- Partitions are collected in the order the expression references them
  rather than hash order.
- The new `PartitionedEval` caches the collapsed expression together with
  `RootFields`. When the root only packs partition results into a
  non-nullable struct, projection assembles the `StructArray` directly
  instead of applying the root expression on every split.

The struct and row-index readers both use it. Nullable structs and filters
get the collapsed partitions too, and `filter_evaluation` no longer has a
variant it must reject.

Signed-off-by: Claude <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AKKCqG37w4E5DGkkiRfR4k
@robert3005 robert3005 added the action/bench-compress Run only the compression benchmark on this PR label Oct 16, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Oct 16, 2026
@codspeed

codspeed Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 16.45%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

❌ 3 regressed benchmarks
✅ 2201 untouched benchmarks
⏩ 370 skipped benchmarks1
🗄️ 30 archived benchmarks run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation test_scan_scalar_at[int_float] 715.1 µs 873.1 µs -18.1%
❌ CPU Simulation test_scan_scalar_at[int_str] 748 µs 904.5 µs -17.3%
❌ CPU Simulation test_scan_scalar_at[int_str_float] 948.8 µs 1,101.9 µs -13.89%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/stoic-cannon-p024ce (e70e6fd) with develop (487bdba)

Open in CodSpeed

Footnotes

  1. 370 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 30 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 11db00d compress-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR 11db00d2 vs base 52a1d156


vortex / vortex-file-compressed / ns (1.048x ➖, 1↑ 6↓)
name ns (PR / base / %diff)
compress time/Arade 771280812 / 788053650 / -2.1%
compress time/Bimbo 3585184039 / 3652527354 / -1.8%
compress time/CMSprovider 2532848590 / 2556482563 / -0.9%
compress time/Euro2016 270699764 / 269786521 / +0.3%
compress time/Food 318859210 / 319042378 / -0.1%
compress time/HashTags 468317810 / 467081595 / +0.3%
compress time/TPC-H l_comment canonical 605555469 / 600865831 / +0.8%
compress time/TPC-H l_comment chunked 606696333 / 604916682 / +0.3%
compress time/taxi 391075219 / 390864424 / +0.1%
compress time/wide table cols=100 chunks=1 rows=1000 7117303 / 7301044 / -2.5%
compress time/wide table cols=100 chunks=50 rows=1000 7228717 / 7360837 / -1.8%
compress time/wide table cols=1000 chunks=1 rows=1000 69914353 / 72093701 / -3.0%
compress time/wide table cols=1000 chunks=50 rows=1000 70785264 / 72117694 / -1.8%
compress time/wide table cols=10000 chunks=1 rows=1000 770547542 / 797452011 / -3.4%
compress time/wide table cols=10000 chunks=50 rows=1000 769483350 / 795853842 / -3.3%
decompress time/Arade 8927313 / 9132966 / -2.3%
decompress time/Bimbo 34948273 / 36488435 / -4.2%
decompress time/CMSprovider 27167717 / 26727085 / +1.6%
decompress time/Euro2016 9940798 / 10054072 / -1.1%
decompress time/Food 2366633 / 2753177 / -14.0% 🟢
decompress time/HashTags 42471284 / 42647749 / -0.4%
decompress time/TPC-H l_comment canonical 11169501 / 11061602 / +1.0%
decompress time/TPC-H l_comment chunked 10851961 / 10919612 / -0.6%
decompress time/taxi 5288587 / 5112378 / +3.4%
decompress time/wide table cols=100 chunks=1 rows=1000 1748983 / 1338955 / +30.6% 🔴
decompress time/wide table cols=100 chunks=50 rows=1000 1756654 / 1366649 / +28.5% 🔴
decompress time/wide table cols=1000 chunks=1 rows=1000 16997354 / 12422280 / +36.8% 🔴
decompress time/wide table cols=1000 chunks=50 rows=1000 16980359 / 12451622 / +36.4% 🔴
decompress time/wide table cols=10000 chunks=1 rows=1000 183670446 / 132947047 / +38.2% 🔴
decompress time/wide table cols=10000 chunks=50 rows=1000 183454504 / 133592979 / +37.3% 🔴
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/Arade 142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo 458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider 420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016 164130260 / 164130260 / +0.0%
vortex-file-compressed size/Food 38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags 196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical 178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked 178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi 49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93277680 / 93277680 / +0.0%
vortex / vortex-file-compressed / ratio (1.032x ➖, 1↑ 6↓)
name ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade 0.343494823 / 0.349431422 / -1.7%
vortex:parquet-zstd ratio compress time/Bimbo 0.30453392 / 0.30892022 / -1.4%
vortex:parquet-zstd ratio compress time/CMSprovider 0.396939855 / 0.402140742 / -1.3%
vortex:parquet-zstd ratio compress time/Euro2016 0.25033971 / 0.249675253 / +0.3%
vortex:parquet-zstd ratio compress time/Food 0.429114568 / 0.430878429 / -0.4%
vortex:parquet-zstd ratio compress time/HashTags 0.256807216 / 0.254923412 / +0.7%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical 0.231164308 / 0.228651855 / +1.1%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked 0.230805415 / 0.229853672 / +0.4%
vortex:parquet-zstd ratio compress time/taxi 0.357771461 / 0.357824835 / -0.0%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000 1.78850346 / 1.85316362 / -3.5%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000 1.83777405 / 1.86774732 / -1.6%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000 1.56480833 / 1.61752091 / -3.3%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000 1.5992867 / 1.59043169 / +0.6%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000 1.64833871 / 1.73272467 / -4.9%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000 1.67810741 / 1.75095231 / -4.2%
vortex:parquet-zstd ratio decompress time/Arade 0.0160404187 / 0.0163907878 / -2.1%
vortex:parquet-zstd ratio decompress time/Bimbo 0.021824572 / 0.0227041416 / -3.9%
vortex:parquet-zstd ratio decompress time/CMSprovider 0.0199952476 / 0.0196528651 / +1.7%
vortex:parquet-zstd ratio decompress time/Euro2016 0.0298337505 / 0.030148597 / -1.0%
vortex:parquet-zstd ratio decompress time/Food 0.0139464165 / 0.0161616417 / -13.7% 🟢
vortex:parquet-zstd ratio decompress time/HashTags 0.0825154578 / 0.0824896285 / +0.0%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical 0.0224835567 / 0.0221464357 / +1.5%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked 0.0218690534 / 0.0218507864 / +0.1%
vortex:parquet-zstd ratio decompress time/taxi 0.0243630476 / 0.0234797973 / +3.8%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000 0.843062956 / 0.650966297 / +29.5% 🔴
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000 0.851936148 / 0.672365613 / +26.7% 🔴
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000 0.722011947 / 0.527201579 / +37.0% 🔴
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000 0.741273835 / 0.515167429 / +43.9% 🔴
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000 0.729894085 / 0.546956812 / +33.4% 🔴
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000 0.735575013 / 0.547930033 / +34.2% 🔴
vortex:parquet-zstd size/Arade 0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo 1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider 1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016 1.33461969 / 1.33461969 / +0.0%
vortex:parquet-zstd size/Food 1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags 1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical 1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked 1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi 0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000 1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000 1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000 1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000 1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000 1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000 1.00040408 / 1.00040408 / -0.0%
vortex / parquet / ns (1.000x ➖, 0↑ 0↓)
name ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade 2245392826 / 2255245520 / -0.4%
parquet_rs-zstd compress time/Bimbo 11772692002 / 11823529560 / -0.4%
parquet_rs-zstd compress time/CMSprovider 6380937969 / 6357183679 / +0.4%
parquet_rs-zstd compress time/Euro2016 1081329700 / 1080549704 / +0.1%
parquet_rs-zstd compress time/Food 743063121 / 740446392 / +0.4%
parquet_rs-zstd compress time/HashTags 1823616244 / 1832242837 / -0.5%
parquet_rs-zstd compress time/TPC-H l_comment canonical 2619588962 / 2627863355 / -0.3%
parquet_rs-zstd compress time/TPC-H l_comment chunked 2628605278 / 2631746866 / -0.1%
parquet_rs-zstd compress time/taxi 1093086680 / 1092334533 / +0.1%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 3979474 / 3939773 / +1.0%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 3933409 / 3941024 / -0.2%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 44679180 / 44570491 / +0.2%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 44260522 / 45344729 / -2.4%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000 467469177 / 460230079 / +1.6%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000 458542371 / 454526280 / +0.9%
parquet_rs-zstd decompress time/Arade 556551120 / 557201160 / -0.1%
parquet_rs-zstd decompress time/Bimbo 1601326842 / 1607126826 / -0.4%
parquet_rs-zstd decompress time/CMSprovider 1358708703 / 1359958704 / -0.1%
parquet_rs-zstd decompress time/Euro2016 333206447 / 333483910 / -0.1%
parquet_rs-zstd decompress time/Food 169694702 / 170352558 / -0.4%
parquet_rs-zstd decompress time/HashTags 514707003 / 517007408 / -0.4%
parquet_rs-zstd decompress time/TPC-H l_comment canonical 496785324 / 499475498 / -0.5%
parquet_rs-zstd decompress time/TPC-H l_comment chunked 496224541 / 499735424 / -0.7%
parquet_rs-zstd decompress time/taxi 217074115 / 217735185 / -0.3%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 2074558 / 2056873 / +0.9%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 2061955 / 2032598 / +1.4%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 23541652 / 23562676 / -0.1%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 22906999 / 24170049 / -5.2%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000 251639861 / 243066809 / +3.5%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000 249402849 / 243813938 / +2.3%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/Arade 258014282 / 258014282 / +0.0%
parquet size/Bimbo 384517292 / 384517292 / +0.0%
parquet size/CMSprovider 376887911 / 376887911 / +0.0%
parquet size/Euro2016 122979049 / 122979049 / +0.0%
parquet size/Food 35699500 / 35699500 / +0.0%
parquet size/HashTags 133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical 158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked 158358238 / 158358238 / +0.0%
parquet size/taxi 55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.951x ➖, 5↑ 0↓)
name ns (PR / base / %diff)
arrow-ipc compress time/Arade 99018052 / 100280549 / -1.3%
arrow-ipc compress time/Bimbo 619782126 / 624334906 / -0.7%
arrow-ipc compress time/CMSprovider 390993464 / 406144379 / -3.7%
arrow-ipc compress time/Euro2016 38442045 / 38665406 / -0.6%
arrow-ipc compress time/Food 33752339 / 33994913 / -0.7%
arrow-ipc compress time/HashTags 78772385 / 79071409 / -0.4%
arrow-ipc compress time/TPC-H l_comment canonical 949544435 / 993045597 / -4.4%
arrow-ipc compress time/TPC-H l_comment chunked 958235379 / 998332638 / -4.0%
arrow-ipc compress time/taxi 74778929 / 75762115 / -1.3%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000 211102 / 208709 / +1.1%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000 201412 / 207072 / -2.7%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000 2247956 / 2297895 / -2.2%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000 2226459 / 2274984 / -2.1%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000 25617530 / 25837297 / -0.9%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000 25696633 / 25776663 / -0.3%
arrow-ipc decompress time/Arade 140364764 / 181341685 / -22.6% 🟢
arrow-ipc decompress time/Bimbo 416533045 / 411290214 / +1.3%
arrow-ipc decompress time/CMSprovider 473388465 / 581345057 / -18.6% 🟢
arrow-ipc decompress time/Euro2016 63835917 / 74908512 / -14.8% 🟢
arrow-ipc decompress time/Food 48774617 / 58461553 / -16.6% 🟢
arrow-ipc decompress time/HashTags 120481828 / 143065409 / -15.8% 🟢
arrow-ipc decompress time/TPC-H l_comment canonical 633387881 / 635000961 / -0.3%
arrow-ipc decompress time/TPC-H l_comment chunked 635795065 / 633792097 / +0.3%
arrow-ipc decompress time/taxi 45165129 / 48827972 / -7.5%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000 318603 / 323199 / -1.4%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000 320749 / 320704 / +0.0%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000 3493899 / 3842328 / -9.1%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000 3611254 / 3871294 / -6.7%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000 36747337 / 37163802 / -1.1%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000 36568670 / 37251619 / -1.8%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/Arade 691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo 3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider 2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016 360582666 / 360582666 / +0.0%
arrow-ipc size/Food 255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags 704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical 7317023586 / 7317023586 / +0.0%
arrow-ipc size/TPC-H l_comment chunked 7317023586 / 7317023586 / +0.0%
arrow-ipc size/taxi 482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 125752506 / 125752506 / +0.0%

@robert3005 robert3005 added the changelog/chore A trivial change label Oct 22, 2026
claude and others added 2 commits October 22, 2026 13:47
Resolve the expression module relocation and preserve AND/OR pruning with the shared partition evaluator.

Signed-off-by: Nicholas Gates <[email protected]>
@gatesn
gatesn enabled auto-merge (squash) October 28, 2026 00:39
@gatesn gatesn added changelog/fix A bug fix and removed changelog/chore A trivial change labels Oct 28, 2026
@gatesn
gatesn merged commit 20cc6e2 into develop Oct 28, 2026
112 of 114 checks passed
@gatesn
gatesn deleted the claude/stoic-cannon-p024ce branch October 28, 2026 00:45
ChristopherWirt added a commit to quantcast/vortex that referenced this pull request Oct 28, 2026
Take upstream's StructReader, whose collapsed partitions replace this
branch's direct_projection extension. Keep the segment-count tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Christopher Wirt <[email protected]>
ChristopherWirt added a commit to quantcast/vortex that referenced this pull request Oct 28, 2026
## Summary

Closes #9

Spark asks the scan for only the nested fields a query uses, but the
connector read every nested column whole. `SELECT s.b` decoded all of
`s`.

Struct columns are now pruned all the way down to the native scan, at
any depth. Fields inside arrays and maps are still read whole; that
needs `list_transform`
([vortex-data#10503](vortex-data#10503)).

## Changes

**Spark connector**
- `pruneColumns` keeps Spark's pruned type for struct columns.
- `NestedSchemaPruning` turns the pruned read schema into a native
projection:
- `select([b], $.s)` when only leaf fields are pruned. This keeps the
struct's own nulls.
- `mask(pack(..), is_not_null($.s))` when a child struct is pruned too.
`pack` has no nulls, so the mask carries them over. Non-nullable structs
skip the mask.

**JNI**
- Adds an `Expression.mask` binding.

**`StructReader`**
- No changes. Upstream
[vortex-data#10536](vortex-data#10536)
collapses single-field partitions, so the Spark projections are read
field by field without a special case. An earlier version of this PR
extended `StructReader`'s direct projection path, which vortex-data#10536 removed.

**Tests**
- Spark: null structs, null leaves, nulls at each level of a struct
inside a struct, and a pushed filter on a field that was pruned away.
- Layout: segment-count tests, now run against vortex-data#10536's partitioned
path. A pruned nullable struct column reads only its kept fields plus
its validity.
- `CountingSegmentSource` now lives in `segments/test.rs`, replacing
three copies.

## Benchmark

Measured before the merge of vortex-data#10536. JMH, 2 forks × 10 iterations,
release native library, Spark 4.1 local[4]. The table has 2M rows, with
a nullable 20-field struct `s` and the same struct nested at
`outer.inner`. `nestedSchemaPruning=false` is the old behaviour, where
the whole struct is read.

| Query | Pruning on | Pruning off |
|---|---|---|
| `max(s.f3)` | 26.2 ± 2.3 ms | 40.2 ± 3.4 ms |
| `max(outer.inner.f3)` | 22.0 ± 1.1 ms | 37.1 ± 2.3 ms |

## Checks

- `cargo test -p vortex-layout --lib`: 696 passed
- `cargo clippy -p vortex-layout --all-targets --all-features -- -D
warnings`: clean
- `cargo +nightly-2026-09-10 fmt --check -p vortex-layout -p
vortex-jni`: clean
- `./gradlew spotlessCheck :vortex-jni:test :vortex-spark_2.13:test
:vortex-spark_2.12:test`: all pass (JNI 97, Spark 2.13 225, Spark 2.12
225 with 2 Spark-4-only tests skipped).
- Not run since vortex-data#10536: the benchmark, the full workspace suite.

## Known gaps

- `Expression` factories don't keep their inputs reachable while the
native call runs. In theory GC can free a temporary first. This affects
every factory, not just the new `mask`, so it's left for a separate fix.
- `count(*)`, or a query that selects only partition columns, still
reads every column. This predates the PR.
- Pruning two levels inside a nullable struct reads the whole inner
struct. For `nullable_outer.inner.c`, `inner.a` and `inner.b` are still
read; sibling fields of `inner` are skipped. The cause is upstream:
`StructReader` masks `$.inner` with the parent's validity before the
`select`, so the `select` can't be pushed down into `inner`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Christopher Wirt <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants