fix(arrow-select): bounds-check take_record_batch once, with a faster check_bounds for null indices by mbutrovich · Pull Request #11829 · apache/arrow-rs

mbutrovich · 2026-10-22T20:06:47Z

Which issue does this PR close?

Closes take_record_batch reads out of bounds when the first column's type has no width #11825.

Rationale for this change

#10945 is not in a release yet, so this fix should go in 60.1.0 (#10525).

What changes are included in this PR?

take_record_batch now checks the indices once against the row count with check_bounds, then takes every column unchecked. It used to rely on a checked take of the first column, which reads no index for some types.

In local testing, the new check_bounds call made take_record_batch with null indices noticeably slower, because check_bounds checks null indices one at a time. So this PR also makes check_bounds run its vectorized loop first, and fall back to the null-aware loop only if that finds a value out of bounds. New benchmarks cover this path.

Are these changes tested?

take_record_batch had no tests of its own, only its doc example and one test through BatchCoalescer::push_batch_with_indices. The new test_take_record_batch* tests in arrow-select/src/take.rs cover columns whose take reads no index, out-of-bounds and negative indices, null index slots holding out-of-bounds values, empty batches, and batches with no columns.

On the benchmark bot, the check_bounds optimization makes take with null indices 1.7-1.9x faster (default, aligned). take_record_batch is within 3% of main, except 5 primitive columns, which is 7-8% slower (default, aligned). Against 60.0.0 it is 1-4% faster, except string and binary columns (run).

Are there any significant user-facing changes?

take_record_batch returns an error for an out-of-bounds index. 60.0.0 panicked for most column types, and accepted the index when no column read it, for example in a batch of only Null columns.

Closes apache#11825.

…indices have nulls

mbutrovich · 2026-10-22T20:08:02Z

run benchmark take_kernels

env:
  BENCH_FILTER: "check bounds|take_record_batch"
baseline:
  ref: 6512c6f72214834a7e9b30c80dc09aa4e8394b56

mbutrovich · 2026-10-22T20:08:19Z

run benchmark take_kernels

env:
  BENCH_FILTER: "check bounds|take_record_batch"
  RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
  ref: 6512c6f72214834a7e9b30c80dc09aa4e8394b56

mbutrovich · 2026-10-22T20:08:28Z

run benchmark take_kernels

env:
  BENCH_FILTER: "check bounds|take_record_batch"
baseline:
  ref: cd2f1a41036f2563eb02b4f557f7fea6b1fa020f

mbutrovich · 2026-10-22T20:08:34Z

run benchmark take_kernels

env:
  BENCH_FILTER: "check bounds|take_record_batch"
  RUSTFLAGS: "-C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6"
baseline:
  ref: cd2f1a41036f2563eb02b4f557f7fea6b1fa020f

adriangbot · 2026-10-22T20:14:21Z

🤖 Benchmark completed (GKE) | trigger

Target: take_kernels

Comparing fix-take-record-batch (28ad286) to 6512c6f diff

Run configuration

run benchmark take_kernels
baseline:
  ref: 6512c6f72214834a7e9b30c80dc09aa4e8394b56
env:
  BENCH_FILTER: check bounds|take_record_batch
shards: 1

Details

group                                                           base                                   changed
-----                                                           ----                                   -------
take check bounds i32 1024                                      1.00    651.1±2.15ns        ? ?/sec    1.00    654.1±2.33ns        ? ?/sec
take check bounds i32 512                                       1.00    363.8±0.93ns        ? ?/sec    1.01    366.3±1.84ns        ? ?/sec
take check bounds i32 null indices 1024                         1.86   1739.7±1.36ns        ? ?/sec    1.00    936.5±1.77ns        ? ?/sec
take check bounds i32 null indices out of bounds 1024           1.05   1791.5±1.83ns        ? ?/sec    1.00   1701.0±3.08ns        ? ?/sec
take_record_batch 5 primitive cols no nulls 1024                1.00      2.9±0.00µs        ? ?/sec    1.03      2.9±0.00µs        ? ?/sec
take_record_batch 6 string/binary cols no nulls 1024            1.00     26.8±0.04µs        ? ?/sec    1.00     26.9±0.02µs        ? ?/sec
take_record_batch 7 mixed cols null values 1024                 1.00     13.9±0.10µs        ? ?/sec    1.00     13.8±0.08µs        ? ?/sec
take_record_batch 7 mixed cols null values null indices 1024    1.07     14.2±0.07µs        ? ?/sec    1.00     13.3±0.03µs        ? ?/sec

Per-runner information

take_kernels — shard 1/1

Node: gk3-benchmark-cluster-nap-bqth6n19-8ff92dbe-5flt

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6350193595-3767-q8mpw 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench take_kernels

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	80.0s
Peak memory	17.6 MiB
Avg memory	13.0 MiB
CPU user	73.8s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	75.0s
Peak memory	17.6 MiB
Avg memory	13.1 MiB
CPU user	70.1s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-10-22T20:14:29Z

🤖 Benchmark completed (GKE) | trigger

Target: take_kernels

Comparing fix-take-record-batch (28ad286) to cd2f1a4 diff

Run configuration

run benchmark take_kernels
baseline:
  ref: cd2f1a41036f2563eb02b4f557f7fea6b1fa020f
env:
  BENCH_FILTER: check bounds|take_record_batch
shards: 1

Details

group                                                           base                                   changed
-----                                                           ----                                   -------
take check bounds i32 1024                                      1.01    657.4±4.33ns        ? ?/sec    1.00    653.7±2.40ns        ? ?/sec
take check bounds i32 512                                       1.00    365.5±3.57ns        ? ?/sec    1.00    366.0±1.13ns        ? ?/sec
take check bounds i32 null indices 1024                         1.86   1741.6±3.24ns        ? ?/sec    1.00    936.3±2.99ns        ? ?/sec
take check bounds i32 null indices out of bounds 1024           1.06   1794.2±3.91ns        ? ?/sec    1.00   1699.6±2.46ns        ? ?/sec
take_record_batch 5 primitive cols no nulls 1024                1.00      2.7±0.02µs        ? ?/sec    1.07      2.9±0.00µs        ? ?/sec
take_record_batch 6 string/binary cols no nulls 1024            1.00     26.5±0.02µs        ? ?/sec    1.03     27.2±0.03µs        ? ?/sec
take_record_batch 7 mixed cols null values 1024                 1.00     13.7±0.07µs        ? ?/sec    1.02     14.0±0.07µs        ? ?/sec
take_record_batch 7 mixed cols null values null indices 1024    1.00     13.2±0.03µs        ? ?/sec    1.00     13.3±0.04µs        ? ?/sec

Per-runner information

take_kernels — shard 1/1

Node: gk3-benchmark-cluster-nap-9990rfe7-f33a191e-8tm9

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6350201685-3769-6kwb4 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench take_kernels

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	80.0s
Peak memory	17.8 MiB
Avg memory	13.2 MiB
CPU user	73.8s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	75.0s
Peak memory	17.6 MiB
Avg memory	13.4 MiB
CPU user	71.1s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-10-22T20:14:51Z

🤖 Benchmark completed (GKE) | trigger

Target: take_kernels

Comparing fix-take-record-batch (28ad286) to 6512c6f diff

Run configuration

run benchmark take_kernels
baseline:
  ref: 6512c6f72214834a7e9b30c80dc09aa4e8394b56
env:
  BENCH_FILTER: check bounds|take_record_batch
  RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1

Details

group                                                           base                                   changed
-----                                                           ----                                   -------
take check bounds i32 1024                                      1.00    651.3±2.12ns        ? ?/sec    1.00    653.5±2.06ns        ? ?/sec
take check bounds i32 512                                       1.00    363.8±1.66ns        ? ?/sec    1.01    366.0±1.01ns        ? ?/sec
take check bounds i32 null indices 1024                         1.70   1590.1±1.46ns        ? ?/sec    1.00    936.9±3.05ns        ? ?/sec
take check bounds i32 null indices out of bounds 1024           1.00  1662.0±34.36ns        ? ?/sec    1.14  1897.5±40.54ns        ? ?/sec
take_record_batch 5 primitive cols no nulls 1024                1.00      2.8±0.00µs        ? ?/sec    1.01      2.9±0.00µs        ? ?/sec
take_record_batch 6 string/binary cols no nulls 1024            1.00     26.7±0.05µs        ? ?/sec    1.01     27.0±0.04µs        ? ?/sec
take_record_batch 7 mixed cols null values 1024                 1.00     13.8±0.10µs        ? ?/sec    1.00     13.9±0.09µs        ? ?/sec
take_record_batch 7 mixed cols null values null indices 1024    1.05     14.4±0.05µs        ? ?/sec    1.00     13.8±0.04µs        ? ?/sec

Per-runner information

take_kernels — shard 1/1

Node: gk3-benchmark-cluster-nap-bqth6n19-13544216-c487

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6350199242-3768-fvd22 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench take_kernels

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	80.0s
Peak memory	19.4 MiB
Avg memory	14.4 MiB
CPU user	73.7s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	75.0s
Peak memory	19.4 MiB
Avg memory	14.8 MiB
CPU user	71.1s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-10-22T20:14:56Z

🤖 Benchmark completed (GKE) | trigger

Target: take_kernels

Comparing fix-take-record-batch (28ad286) to cd2f1a4 diff

Run configuration

run benchmark take_kernels
baseline:
  ref: cd2f1a41036f2563eb02b4f557f7fea6b1fa020f
env:
  BENCH_FILTER: check bounds|take_record_batch
  RUSTFLAGS: -C llvm-args=-align-all-functions=6 -C llvm-args=-align-all-nofallthru-blocks=6
shards: 1

Details

group                                                           base                                   changed
-----                                                           ----                                   -------
take check bounds i32 1024                                      1.00    652.0±2.06ns        ? ?/sec    1.00    651.5±2.50ns        ? ?/sec
take check bounds i32 512                                       1.00    361.7±0.72ns        ? ?/sec    1.01    364.3±1.27ns        ? ?/sec
take check bounds i32 null indices 1024                         1.70   1595.1±1.34ns        ? ?/sec    1.00    935.7±3.42ns        ? ?/sec
take check bounds i32 null indices out of bounds 1024           1.00   1656.0±2.35ns        ? ?/sec    1.15  1908.9±62.88ns        ? ?/sec
take_record_batch 5 primitive cols no nulls 1024                1.00      2.7±0.00µs        ? ?/sec    1.08      2.9±0.01µs        ? ?/sec
take_record_batch 6 string/binary cols no nulls 1024            1.00     28.0±0.04µs        ? ?/sec    1.00     27.8±0.08µs        ? ?/sec
take_record_batch 7 mixed cols null values 1024                 1.00     13.8±0.11µs        ? ?/sec    1.02     14.1±0.44µs        ? ?/sec
take_record_batch 7 mixed cols null values null indices 1024    1.00     13.6±0.05µs        ? ?/sec    1.01     13.8±0.15µs        ? ?/sec

Per-runner information

take_kernels — shard 1/1

Node: gk3-benchmark-cluster-nap-9990rfe7-b5ee8466-njp7

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6350203663-3770-sq8vk 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench take_kernels

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	80.0s
Peak memory	19.4 MiB
Avg memory	14.5 MiB
CPU user	73.9s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	75.0s
Peak memory	19.4 MiB
Avg memory	14.8 MiB
CPU user	71.1s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

mbutrovich · 2026-10-22T20:38:29Z

run benchmark take_kernels

env:
  BENCH_FILTER: "^take (check bounds )?i32 1024$|take_record_batch 5 primitive"

adriangbot · 2026-10-22T20:43:23Z

🤖 Benchmark completed (GKE) | trigger

Target: take_kernels

Comparing fix-take-record-batch (28ad286) to 01a9917 (merge-base) diff

Run configuration

run benchmark take_kernels
env:
  BENCH_FILTER: ^take (check bounds )?i32 1024$|take_record_batch 5 primitive
shards: 1

Details

group                                               base                                   changed
-----                                               ----                                   -------
take check bounds i32 1024                          1.00    566.7±0.85ns        ? ?/sec    1.01    573.8±3.41ns        ? ?/sec
take i32 1024                                       1.00    508.7±0.81ns        ? ?/sec    1.00    510.7±1.66ns        ? ?/sec
take_record_batch 5 primitive cols no nulls 1024    1.00      2.8±0.01µs        ? ?/sec    1.05      2.9±0.01µs        ? ?/sec

Per-runner information

take_kernels — shard 1/1

Node: gk3-benchmark-cluster-nap-155is15w-5ae84a9b-5k6v

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6350793941-3775-5zh7l 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench take_kernels

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	35.0s
Peak memory	13.9 MiB
Avg memory	10.0 MiB
CPU user	29.6s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	30.0s
Peak memory	13.8 MiB
Avg memory	10.5 MiB
CPU user	27.0s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

mbutrovich · 2026-10-22T20:44:36Z

run benchmark take_kernels

env:
  BENCH_FILTER: "take_record_batch"
baseline:
  ref: ef1fa157977633f0ba21aa921f9ef3d5c669235b

adriangbot · 2026-10-22T20:50:27Z

🤖 Benchmark completed (GKE) | trigger

Target: take_kernels

Comparing fix-take-record-batch (28ad286) to ef1fa15 diff

Run configuration

run benchmark take_kernels
baseline:
  ref: ef1fa157977633f0ba21aa921f9ef3d5c669235b
env:
  BENCH_FILTER: take_record_batch
shards: 1

Details

group                                                           base                                   changed
-----                                                           ----                                   -------
take_record_batch 5 primitive cols no nulls 1024                1.04      2.9±0.02µs        ? ?/sec    1.00      2.8±0.01µs        ? ?/sec
take_record_batch 6 string/binary cols no nulls 1024            1.00     26.8±0.02µs        ? ?/sec    1.04     27.8±0.03µs        ? ?/sec
take_record_batch 7 mixed cols null values 1024                 1.01     14.0±0.10µs        ? ?/sec    1.00     13.8±0.08µs        ? ?/sec
take_record_batch 7 mixed cols null values null indices 1024    1.04     13.7±0.04µs        ? ?/sec    1.00     13.2±0.04µs        ? ?/sec

Per-runner information

take_kernels — shard 1/1

Node: gk3-benchmark-cluster-nap-bhbuxd0n-d730891d-c8vh

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6350909780-3776-4zhph 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench take_kernels

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	40.0s
Peak memory	16.0 MiB
Avg memory	11.4 MiB
CPU user	33.8s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	40.0s
Peak memory	17.6 MiB
Avg memory	13.7 MiB
CPU user	37.0s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey · 2026-10-22T23:27:04Z

+    // Null slots usually hold in-bounds values, so first check every value with a loop that
+    // vectorizes, and look at validity only if that finds one out of bounds
+    let in_bounds = indices.values().iter().fold(true, |in_bounds, &i| {
+        in_bounds & (i >= T::Native::ZERO) & (i < len)


using bit and here is intentional to enable vectorization?

using bit and here is intentional to enable vectorization?

Yes. This is the existing fold from #8966, moved unchanged from the no-nulls branch on main so that it also runs when the indices have nulls. #8966 links a godbolt example of the loop vectorizing.

mbutrovich · 2026-10-23T00:09:00Z

Thanks for the review @Jefffrey!

…d_batch (apache#11831) # Which issue does this PR close? - Closes apache#11824. # Rationale for this change apache#10945 is not in a release yet, so this fix should go in 60.1.0 (apache#10525). This PR changes the same lines of `take_record_batch` as apache#11829, so it should merge after apache#11829. I will rebase it then. # What changes are included in this PR? `take_record_batch` builds its result with `RecordBatch::try_new_with_options` instead of `RecordBatch::new_unchecked`, so it checks nullability as 60.0.0 did. It passes the row count so a batch with no columns still works. # Are these changes tested? `take_record_batch` had no tests of its own, only its doc example and one test through `BatchCoalescer::push_batch_with_indices`. The new `test_take_record_batch_*` tests in `arrow-select/src/take.rs` cover a null index into non-nullable primitive, string, dictionary, struct, and fixed-size list columns, indices whose validity buffer has no nulls, and a batch with no columns. The benchmark bot shows `take_record_batch` within 2% of `main` ([default](apache#11831 (comment)), [aligned](apache#11831 (comment))). # Are there any significant user-facing changes? No. This restores the 60.0.0 behavior. --------- Co-authored-by: Jefffrey <[email protected]>

Rich-T-kid

approving to remove from my queue (ignore)

mbutrovich added 3 commits October 22, 2026 15:50

bench(arrow): add take check bounds benchmarks with null indices

cd2f1a4

fix(arrow-select): check take_record_batch indices against the row count

6512c6f

Closes apache#11825.

perf(arrow-select): check index bounds with the vectorized loop when …

28ad286

…indices have nulls

This comment was marked as duplicate.

Sign in to view

github-actions Bot added arrow-core Core crates related to representing arrow data arrow-kernels Crates that transform arrow data (e.g. compute, cast) labels Oct 22, 2026

mbutrovich mentioned this pull request Oct 22, 2026

fix(arrow-select): reject nulls in non-nullable columns in take_record_batch #11831

Merged

This comment was marked as duplicate.

Sign in to view

mbutrovich marked this pull request as ready for review October 22, 2026 21:04

mbutrovich mentioned this pull request Oct 22, 2026

Release arrow-rs / parquet Minor version 60.1.0 (October 2026) #10525

Closed

11 of 12 tasks

Jefffrey requested a review from Rich-T-kid October 22, 2026 23:17

Jefffrey approved these changes Oct 22, 2026

View reviewed changes

Jefffrey added the bug label Oct 22, 2026

Merge branch 'main' into fix-take-record-batch

c56da14

Jefffrey enabled auto-merge October 23, 2026 05:29

Jefffrey added this pull request to the merge queue Oct 23, 2026

Merged via the queue into apache:main with commit b90914f Oct 23, 2026
35 checks passed

mbutrovich deleted the fix-take-record-batch branch October 23, 2026 11:21

mbutrovich mentioned this pull request Nov 3, 2026

[DISCUSSION] 2026 Q4 Roadmap apache/datafusion#26684

Open

Rich-T-kid approved these changes Nov 13, 2026

View reviewed changes

Conversation

mbutrovich commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any significant user-facing changes?

Uh oh!

mbutrovich commented Oct 22, 2026

Uh oh!

This comment was marked as duplicate.

mbutrovich commented Oct 22, 2026

Uh oh!

This comment was marked as duplicate.

mbutrovich commented Oct 22, 2026

Uh oh!

This comment was marked as duplicate.

mbutrovich commented Oct 22, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Oct 22, 2026

Uh oh!

adriangbot commented Oct 22, 2026

Uh oh!

adriangbot commented Oct 22, 2026

Uh oh!

adriangbot commented Oct 22, 2026

Uh oh!

mbutrovich commented Oct 22, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Oct 22, 2026

Uh oh!

mbutrovich commented Oct 22, 2026

Uh oh!

This comment was marked as duplicate.

adriangbot commented Oct 22, 2026

Uh oh!

Jefffrey Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

mbutrovich Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich commented Oct 23, 2026

Uh oh!

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mbutrovich commented Oct 22, 2026 •

edited

Loading

mbutrovich Oct 23, 2026 •

edited

Loading