fix(arrow-array): deprecate FixedSizeListArray::value_offset, which wraps past i32::MAX by Cintu07 · Pull Request #11070 · apache/arrow-rs

Cintu07 · 2026-09-13T07:47:04Z

Which issue does this PR close?

Closes FixedSizeListArray::value_offset wraps past i32::MAX, and the avro writer panics on it #11059.

Rationale for this change

FixedSizeListArray::value_offset casts a usize offset to i32, so once a row starts past i32::MAX child values it returns a negative number. the avro FixedSizeList encoder cast that back to usize and panicked writing the batch.

What changes are included in this PR?

deprecate value_offset in favour of i * value_length() as usize, same as FixedSizeBinaryArray::value_offset in [arrow-array] use usize arithmetic in FixedSizeBinaryArray, aggressive overflow checks #9910, and panic instead of wrapping
the avro encoder uses idx * elem_len instead
take_value_indices_from_fixed_size_list computes the offset in usize and returns an error when it doesn't fit u32
the value_offset asserts come out of the tests, and the slice test checks row values instead

Are these changes tested?

yes. fixed_size_list_encoder_int32_sliced is new and covers the avro change. the overflow itself needs 384 MiB so it isn't a unit test, but the repro from the issue panics on 59.3.0 and writes the batch on this branch. arrow-array 972, arrow-select 446 and arrow-avro 515 tests pass, and clippy is clean.

Are there any user-facing changes?

FixedSizeListArray::value_offset is deprecated and panics past i32::MAX. take on a FixedSizeList returns an error past u32::MAX instead of wrapping.

…::MAX value_offset narrows a usize offset to i32 with as, so past i32::MAX child values it returns a negative number. The avro FixedSizeList encoder cast that back to usize and panicked writing the batch. Deprecate it in favour of i * value_length() as usize, as was done for FixedSizeBinaryArray::value_offset, and panic instead of wrapping. The avro encoder and take work the offset out in usize, and take returns an error when it does not fit its u32 indices. Closes apache#11059.

alamb

Thanks @Cintu07 and @Jefffrey

alamb · 2026-09-15T20:24:09Z

    #[inline]
    pub fn value_offset(&self, i: usize) -> i32 {
-        self.value_offset_at(i) as i32
+        i32::try_from(self.value_offset_at(i)).expect("offset overflow")


It would be somewhat annoying if I were a user to start seeing panics (even though you could argue that is better than silent overflows 🤔 )

fair. the deprecated method only panics past i32::MAX, where it used to hand back a negative offset and the avro writer turned that into an index near u64::MAX. FixedSizeBinaryArray::value_offset documents the same panic since 59.0.0. anyone who wants no panic at all can move to value_offset_at, which stays in usize. happy to leave the wrapping cast alone and only deprecate if you would rather not add a panic at all.

for a minor release I think we should just deprecate the function and not change its behavior

Jefffrey asked for 60.0.0 rather than 60.1.0, and alamb asked for a public function that works the offset out correctly, so value_offset_at is public now and the deprecation note points at it.

alamb

I don't think we should change the implementation of

    pub fn value_offset(&self, i: usize) -> i32 {

Other than that this PR looks good to go to me -- thank you @Cintu07

alamb asked for a deprecation only in a minor release, so the wrapping cast comes back and the doc says what happens past i32::MAX instead. value_offset_at stays public as the accessor that never wraps.

Jefffrey asked for the removed asserts back rather than dropped, and for take to use the accessor too, so the tests and take both go through value_offset_at.

Cintu07 · 2026-09-22T05:45:16Z

Both restored through value_offset_at in 7977909, and take uses it now too.

changes made

alamb · 2026-09-22T10:27:42Z

run benchmark take_kernels

alamb · 2026-09-22T10:27:52Z

Thank you @Cintu07

adriangbot · 2026-09-22T10:31:15Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5774905477-2577-4zfb2 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix/fixed-size-list-value-offset (7977909) to 4cd8be9 (merge-base) diff

Run configuration

run benchmark take_kernels

BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench take_kernels
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-22T10:51:19Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing fix/fixed-size-list-value-offset (7977909) to 4cd8be9 (merge-base) diff

Run configuration

run benchmark take_kernels

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                       fix_fixed-size-list-value-offset       main
-----                                                                       --------------------------------       ----
take bool 1024                                                              1.00    602.7±0.47ns        ? ?/sec    1.01    607.0±1.05ns        ? ?/sec
take bool 512                                                               1.00    349.1±0.42ns        ? ?/sec    1.00    349.5±1.33ns        ? ?/sec
take bool null indices 1024                                                 1.00    806.5±1.09ns        ? ?/sec    1.00    806.0±2.27ns        ? ?/sec
take bool null values 1024                                                  1.00   1099.5±0.74ns        ? ?/sec    1.01   1108.5±1.83ns        ? ?/sec
take bool null values null indices 1024                                     1.00   1129.0±0.96ns        ? ?/sec    1.01   1144.6±3.85ns        ? ?/sec
take check bounds i32 1024                                                  1.00    584.1±0.80ns        ? ?/sec    1.13    659.2±1.18ns        ? ?/sec
take check bounds i32 512                                                   1.00    380.9±0.75ns        ? ?/sec    1.19    454.9±1.08ns        ? ?/sec
take dense union 1024                                                       1.02      7.5±0.01µs        ? ?/sec    1.00      7.4±0.01µs        ? ?/sec
take dense union 512                                                        1.00      4.9±0.01µs        ? ?/sec    1.05      5.1±0.01µs        ? ?/sec
take fixed_size_list<i32>[22] 1024                                          1.02      3.5±0.01µs        ? ?/sec    1.00      3.4±0.01µs        ? ?/sec
take fixed_size_list<i32>[22] null indices 1024                             1.00      6.3±0.01µs        ? ?/sec    1.01      6.4±0.01µs        ? ?/sec
take fixed_size_list<i32>[22] null values 1024                              1.00      7.5±0.02µs        ? ?/sec    1.00      7.6±0.01µs        ? ?/sec
take fixed_size_list<i32>[8] 1024                                           1.00      2.5±0.00µs        ? ?/sec    1.00      2.5±0.00µs        ? ?/sec
take fixed_size_list<i32>[8] null indices 1024                              1.00      4.2±0.01µs        ? ?/sec    1.00      4.2±0.01µs        ? ?/sec
take fixed_size_list<i32>[8] null values 1024                               1.01      5.2±0.01µs        ? ?/sec    1.00      5.1±0.01µs        ? ?/sec
take fsb value len: 12, indices: 1024                                       1.00   1712.2±1.26ns        ? ?/sec    1.00   1709.1±1.36ns        ? ?/sec
take fsb value len: 12, null values, indices: 1024                          1.00      2.3±0.00µs        ? ?/sec    1.03      2.4±0.00µs        ? ?/sec
take fsb value optimized len: 16, indices: 1024                             1.00    589.0±1.22ns        ? ?/sec    1.00    588.2±0.80ns        ? ?/sec
take fsb value optimized len: 16, null values, indices: 1024                1.00   1202.4±1.50ns        ? ?/sec    1.00   1198.5±1.26ns        ? ?/sec
take i32 1024                                                               1.00    508.9±0.74ns        ? ?/sec    1.03    523.4±1.22ns        ? ?/sec
take i32 512                                                                1.00    346.9±0.92ns        ? ?/sec    1.00    346.7±1.15ns        ? ?/sec
take i32 null indices 1024                                                  1.02    871.5±1.50ns        ? ?/sec    1.00    856.0±1.80ns        ? ?/sec
take i32 null values 1024                                                   1.00   1118.6±2.13ns        ? ?/sec    1.00   1121.4±1.90ns        ? ?/sec
take i32 null values null indices 1024                                      1.04   1684.9±2.31ns        ? ?/sec    1.00   1625.5±2.19ns        ? ?/sec
take list i32 1024                                                          1.01      5.1±0.06µs        ? ?/sec    1.00      5.1±0.06µs        ? ?/sec
take list i32 512                                                           1.00      2.7±0.03µs        ? ?/sec    1.01      2.8±0.02µs        ? ?/sec
take list i32 null indices 1024                                             1.00      8.3±0.03µs        ? ?/sec    1.01      8.4±0.02µs        ? ?/sec
take list i32 null values 1024                                              1.00      3.9±0.02µs        ? ?/sec    1.01      4.0±0.02µs        ? ?/sec
take list i32 null values null indices 1024                                 1.00      6.9±0.05µs        ? ?/sec    1.00      6.9±0.06µs        ? ?/sec
take listview i32 1024                                                      1.00    962.1±4.50ns        ? ?/sec    1.08   1038.2±1.86ns        ? ?/sec
take listview i32 512                                                       1.00    597.0±4.02ns        ? ?/sec    1.01    604.0±1.00ns        ? ?/sec
take listview i32 null indices 1024                                         1.01   1622.6±3.94ns        ? ?/sec    1.00   1609.1±1.35ns        ? ?/sec
take listview i32 null values 1024                                          1.01   1554.5±3.27ns        ? ?/sec    1.00   1542.4±1.90ns        ? ?/sec
take listview i32 null values null indices 1024                             1.03      2.4±0.00µs        ? ?/sec    1.00      2.4±0.00µs        ? ?/sec
take map<str, i32> 1024                                                     1.00     33.3±0.10µs        ? ?/sec    1.00     33.3±0.04µs        ? ?/sec
take map<str, i32> 512                                                      1.01     18.9±0.03µs        ? ?/sec    1.00     18.8±0.02µs        ? ?/sec
take map<str, i32> null indices 1024                                        1.00     19.8±0.04µs        ? ?/sec    1.00     19.8±0.03µs        ? ?/sec
take map<str, i32> null values 1024                                         1.01     19.9±0.03µs        ? ?/sec    1.00     19.8±0.03µs        ? ?/sec
take primitive run logical len: 1024, physical len: 512, indices: 1024      1.00     17.9±0.02µs        ? ?/sec    1.02     18.2±0.02µs        ? ?/sec
take sparse union 1024                                                      1.00      7.6±0.01µs        ? ?/sec    1.00      7.7±0.01µs        ? ?/sec
take sparse union 512                                                       1.00      4.7±0.01µs        ? ?/sec    1.03      4.8±0.01µs        ? ?/sec
take str 1024                                                               1.00      8.0±0.01µs        ? ?/sec    1.00      8.0±0.00µs        ? ?/sec
take str 512                                                                1.00      3.7±0.01µs        ? ?/sec    1.00      3.7±0.01µs        ? ?/sec
take str null indices 1024                                                  1.00      4.4±0.05µs        ? ?/sec    1.03      4.5±0.04µs        ? ?/sec
take str null indices 512                                                   1.02      2.1±0.03µs        ? ?/sec    1.00      2.1±0.03µs        ? ?/sec
take str null values 1024                                                   1.00      4.5±0.06µs        ? ?/sec    1.00      4.5±0.05µs        ? ?/sec
take str null values null indices 1024                                      1.01      3.1±0.01µs        ? ?/sec    1.00      3.1±0.02µs        ? ?/sec
take string run logical len: 1024, physical len: 128, indices: 1024         1.00     23.5±0.02µs        ? ?/sec    1.00     23.4±0.03µs        ? ?/sec
take string run logical len: 1024, physical len: 512, indices: 1024         1.00     23.8±0.03µs        ? ?/sec    1.01     24.0±0.02µs        ? ?/sec
take string run logical len: 1024, physical len: 512, null indices: 1024    1.00     16.3±0.07µs        ? ?/sec    1.00     16.3±0.05µs        ? ?/sec
take stringview 1024                                                        1.08    903.9±1.05ns        ? ?/sec    1.00    837.9±1.22ns        ? ?/sec
take stringview 512                                                         1.14    560.2±1.43ns        ? ?/sec    1.00    491.3±0.40ns        ? ?/sec
take stringview null indices 1024                                           1.00    849.9±1.02ns        ? ?/sec    1.00    848.5±0.93ns        ? ?/sec
take stringview null indices 512                                            1.00    504.4±1.16ns        ? ?/sec    1.00    505.3±1.00ns        ? ?/sec
take stringview null values 1024                                            1.04   1474.2±1.18ns        ? ?/sec    1.00   1419.4±4.14ns        ? ?/sec
take stringview null values null indices 1024                               1.00   1680.0±1.17ns        ? ?/sec    1.00   1678.0±2.81ns        ? ?/sec
take_record_batch 5 primitive cols no nulls 1024                            1.02      3.0±0.02µs        ? ?/sec    1.00      2.9±0.01µs        ? ?/sec
take_record_batch 6 string/binary cols no nulls 1024                        1.00     27.4±0.08µs        ? ?/sec    1.00     27.5±0.05µs        ? ?/sec
take_record_batch 7 mixed cols null values 1024                             1.00     14.1±0.09µs        ? ?/sec    1.02     14.4±0.45µs        ? ?/sec
take_record_batch 7 mixed cols null values null indices 1024                1.00     13.9±0.05µs        ? ?/sec    1.01     14.1±0.14µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	580.1s
Peak memory	18.5 MiB
Avg memory	14.3 MiB
CPU user	575.4s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	570.1s
Peak memory	18.5 MiB
Avg memory	15.0 MiB
CPU user	567.6s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Jefffrey · 2026-09-23T02:29:08Z

thanks @Cintu07 & @alamb

…pache/arrow-rs#11070) Co-Authored-By: Claude Fable 5.1 <[email protected]>

github-actions Bot added arrow Changes to the arrow crate arrow-avro arrow-avro crate arrow-select arrow-array labels Sep 13, 2026

Jefffrey approved these changes Sep 15, 2026

View reviewed changes

Comment thread arrow-array/src/array/fixed_size_list_array.rs Resolved

Jefffrey added the bug label Sep 15, 2026

alamb reviewed Sep 15, 2026

View reviewed changes

review: deprecate since 60.0.0 and make value_offset_at public

8aab2dd

Jefffrey asked for 60.0.0 rather than 60.1.0, and alamb asked for a public function that works the offset out correctly, so value_offset_at is public now and the deprecation note points at it.

alamb previously requested changes Sep 17, 2026

View reviewed changes

review: deprecate value_offset without changing what it does

e4b48cf

alamb asked for a deprecation only in a minor release, so the wrapping cast comes back and the doc says what happens past i32::MAX instead. value_offset_at stays public as the accessor that never wraps.

Jefffrey reviewed Sep 21, 2026

View reviewed changes

Comment thread arrow-array/src/array/fixed_size_list_array.rs Resolved

Comment thread arrow-array/src/builder/fixed_size_list_builder.rs Resolved

Comment thread arrow-select/src/take.rs Resolved

alamb self-requested a review September 21, 2026 19:45

review: keep the offset asserts, through value_offset_at

7977909

Jefffrey asked for the removed asserts back rather than dropped, and for take to use the accessor too, so the tests and take both go through value_offset_at.

Cintu07 force-pushed the fix/fixed-size-list-value-offset branch from 7dc218b to 7977909 Compare September 22, 2026 05:29

This comment was marked as outdated.

Sign in to view

Jefffrey approved these changes Sep 23, 2026

View reviewed changes

Jefffrey merged commit 132371a into apache:main Sep 23, 2026
41 checks passed

alamb added a commit to alamb/datafusion that referenced this pull request Oct 15, 2026

Use FixedSizeListArray::value_offset_at (value_offset deprecated in a…

4fd151c

…pache/arrow-rs#11070) Co-Authored-By: Claude Fable 5.1 <[email protected]>

This was referenced Oct 15, 2026

[DO NOT MERGE] Test DataFusion against arrow-rs main (pre-release testing, 2026-10-15) apache/datafusion#26384

Closed

[DO NOT MERGE] Test DataFusion against arrow/parquet 60.1.0 release candidate (apache/arrow-rs#11750) apache/datafusion#26543

Closed

alamb mentioned this pull request Oct 29, 2026

chore: Update to arrow/parquet 60.1.0 apache/datafusion#26867

Merged

Conversation

Cintu07 commented Sep 13, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

alamb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Cintu07 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cintu07 commented Sep 22, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

alamb commented Sep 22, 2026

Uh oh!

alamb commented Sep 22, 2026

Uh oh!

adriangbot commented Sep 22, 2026

Uh oh!

adriangbot commented Sep 22, 2026

Uh oh!

Uh oh!

Jefffrey commented Sep 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alamb left a comment •

edited

Loading

alamb Sep 16, 2026 •

edited

Loading