-
Type:
Sub-task
-
Resolution: Fixed
-
Priority:
Major
-
Affects Version/s: 5.0.0
-
Component/s: SQL
Replaces per-group in.slice(bitWidth) (one ByteBuffer allocation per 8 values) with a single bulk slice for the entire RLE PACKED run in the Parquet vectorized reader.
Changes
- Batch ByteBuffer slice: Instead of allocating a new ByteBuffer for each 8-value group in a PACKED run, allocate a single slice covering all groups and advance a position cursor.
- Eliminates ~128K short-lived ByteBuffer allocations per 1M-value page.
Benchmark Results (GHA, AMD EPYC 7763)
| bitWidth | Speedup (readIntegers) | Speedup (skipIntegers) |
|---|---|---|
| 4 | 2.0x | 2.1x |
| 8 | 2.0x | 2.4x |
| 12 | 1.6x | 1.6x |
| 20 | 1.4x | 1.4x |
PR: https://github.com/apache/spark/pull/55922
Parent issue: https://github.com/apache/spark/issues/56011
- links to