arrow-ipc: honour StreamDecoder::with_skip_validation for record batches by Dandandan · Pull Request #11552 · apache/arrow-rs

Dandandan · 2026-10-14T12:31:35Z

This is significantly faster:

StreamDecoder/no_validation/read_10 │ 241 µs → 9.8 µs (−96%)

StreamDecoder passed its skip_validation flag to dictionary batches only, so record batches were always validated. Pass it to the RecordBatchDecoder too, and cover StreamDecoder in the validation tests. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PpXao5thfsWERyA3UQfLRf

Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PpXao5thfsWERyA3UQfLRf

Dandandan · 2026-10-14T12:43:12Z

run benchmark ipc_reader

adriangbot · 2026-10-14T12:43:17Z

🤖 Benchmark starting (GKE) | trigger

Target: ipc_reader

Sharding factor: 1 (1 workers).

Comparing stream-decoder-skip-validation (05ae983) to bfdcd26 (merge-base) diff

Run configuration

run benchmark ipc_reader
shards: 1

Results will be posted when all workers finish.

File an issue against this benchmark runner

adriangbot · 2026-10-14T12:50:32Z

🤖 Benchmark completed (GKE) | trigger

Target: ipc_reader

Comparing stream-decoder-skip-validation (05ae983) to bfdcd26 (merge-base) diff

Run configuration

run benchmark ipc_reader
shards: 1

Details

group                                                       base                                   changed
-----                                                       ----                                   -------
arrow_ipc_reader/FileReader/no_validation/read_10           1.00    146.1±4.46µs        ? ?/sec    1.09    158.7±4.28µs        ? ?/sec
arrow_ipc_reader/FileReader/no_validation/read_10/mmap      1.00     58.2±0.61µs        ? ?/sec    1.00     58.3±0.50µs        ? ?/sec
arrow_ipc_reader/FileReader/read_10                         1.00   471.2±31.91µs        ? ?/sec    1.05   494.5±31.61µs        ? ?/sec
arrow_ipc_reader/FileReader/read_10/mmap                    1.00   498.7±30.63µs        ? ?/sec    1.00   499.3±30.11µs        ? ?/sec
arrow_ipc_reader/StreamDecoder/no_validation/read_10                                               1.00     14.1±0.08µs        ? ?/sec
arrow_ipc_reader/StreamDecoder/read_10                                                             1.00   307.9±30.34µs        ? ?/sec
arrow_ipc_reader/StreamReader/no_validation/read_10         1.00    153.1±4.63µs        ? ?/sec    1.02    156.8±5.31µs        ? ?/sec
arrow_ipc_reader/StreamReader/no_validation/read_10/zstd    1.00      2.5±0.02ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
arrow_ipc_reader/StreamReader/read_10                       1.02   477.4±33.21µs        ? ?/sec    1.00   467.3±34.13µs        ? ?/sec
arrow_ipc_reader/StreamReader/read_10/zstd                  1.00      2.8±0.01ms        ? ?/sec    1.00      2.8±0.01ms        ? ?/sec

Per-runner information

ipc_reader — shard 1/1

Node: gk3-benchmark-cluster-nap-4vpacm1r-837509ea-cqz7

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6168610238-3383-2p7xj 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench ipc_reader

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Resource Usage

base (merge-base)

Metric	Value
Wall time	90.0s
Peak memory	16.8 MiB
Avg memory	13.1 MiB
CPU user	74.2s
CPU sys	10.6s
Peak spill	0 B

branch

Metric	Value
Wall time	110.0s
Peak memory	21.1 MiB
Avg memory	14.9 MiB
CPU user	94.2s
CPU sys	9.8s
Peak spill	0 B

File an issue against this benchmark runner

comphead · 2026-10-24T03:36:53Z

@Dandandan are you planning to move on with this? I think it resolves some of DF perf issues with spilling

comphead · 2026-10-28T00:43:33Z

wondering if I can steal this PR 🤔

comphead · 2026-11-05T20:33:30Z

@Dandandan I picked this changes in the separate PR #12094

Rich-T-kid · 2026-11-05T21:54:51Z

closing in favor of #12094

@Dandandan

# Which issue does this PR close? - Relates to apache/datafusion#26687. # Rationale for this change `StreamDecoder` has no benchmarks. As CONTRIBUTING.md asks, this adds them in their own PR so the benchmark runner can measure the `StreamDecoder::with_skip_validation` fix from apache#11552 against `main`. # What changes are included in this PR? Two benchmarks next to the `StreamReader` ones, `StreamDecoder/read_10` and `StreamDecoder/no_validation/read_10`. Both decode the same 10-batch stream from a single `Buffer`. The code is from apache#11552 by @Dandandan. # Are these changes tested? This PR only adds benchmarks. On `main` both take about 129 µs on an M5 Max, because `with_skip_validation` does not reach record batches yet. # Are there any significant user-facing changes? No. AI disclosure: The benchmarks come from apache#11552. Porting them to `main` and writing this description were done with Claude Code. I have reviewed every change and the benchmark results. Co-authored-by: Daniël Heres <[email protected]>

@Dandandan

…oder (apache#12094) # Which issue does this PR close? - Relates to apache/datafusion#26687. This picks up apache#11552 by @Dandandan with the same fix, rebased on `main`. The benchmarks are split out into apache#12093. # Rationale for this change `StreamDecoder::with_skip_validation(true)` only applies to dictionary batches, so record batches are still fully validated. `StreamReader`, `FileDecoder` and the `arrow-flight` `FlightDataDecoder` all pass the flag through for record batches too. DataFusion's spill reader has called `StreamDecoder::new().with_skip_validation(true)` since apache/datafusion#21882. Because of this bug, every spilled batch is validated again when it is read back, including a UTF-8 check of every string. That undoes apache/datafusion#15454. # What changes are included in this PR? - Pass `skip_validation` to `RecordBatchDecoder` in `StreamDecoder::decode`. - Cover `StreamDecoder` in `expect_ipc_validation_error`. # Are these changes tested? Yes. `expect_ipc_validation_error` now checks that `StreamDecoder` rejects invalid data by default and accepts it when validation is skipped. Without the fix, the five `test_validation_of_invalid_*` tests fail on the new check. Benchmarks from apache#12093 on an M5 Max, `main` against this PR: | Benchmark | main | this PR | Change | |---|---|---|---| | `StreamDecoder/no_validation/read_10` | 129.0 µs | 6.18 µs | −95.2% | | `StreamDecoder/read_10` | 128.9 µs | 128.8 µs | no change | The `StreamReader` and `FileReader` benchmarks do not change. With arrow patched to this branch, DataFusion's `spill_io/StreamReader/read_100` drops from 9.59 ms to 8.48 ms (−11.5%). That benchmark reads 100 spilled batches of 8192 rows through `StreamDecoder`. After apache#12093 merges, rebasing this PR lets `run benchmark ipc_reader` confirm these numbers on the shared runner. # Are there any significant user-facing changes? No API changes, and validation is still on by default. Callers that pass `true` now skip validation for record batches too, as documented. AI disclosure: The fix, the test and this description were written with Claude Code. I have reviewed every change and the benchmark results. Co-authored-by: RIchard Baah <[email protected]>

github-actions Bot added the arrow-ipc arrow-ipc crate label Oct 14, 2026

arrow-ipc: add StreamDecoder benchmarks

05ae983

Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PpXao5thfsWERyA3UQfLRf

alamb added the performance label Oct 16, 2026

This was referenced Oct 24, 2026

fix: skip validating spilled record batches again apache/datafusion#26688

Closed

Spill files are validated again on read since 55.0.0 apache/datafusion#26687

Open

Release arrow-rs / parquet Minor version 60.2.0 (November 2026) #10526

Open

This was referenced Oct 31, 2026

perf: spill byte arrays as lengths and coalesce spill writes apache/datafusion#26960

Merged

bench(arrow-ipc): add StreamDecoder benchmarks #12093

Merged

fix(arrow-ipc): honor skip_validation for record batches in StreamDecoder #12094

Merged

Rich-T-kid closed this Nov 5, 2026

Navigation Menu

Repository navigation

arrow-ipc: honour StreamDecoder::with_skip_validation for record batches - #11552

arrow-ipc: honour StreamDecoder::with_skip_validation for record batches#11552
Dandandan wants to merge 2 commits into
apache:mainfrom
Dandandan:stream-decoder-skip-validation

Dandandan commented Oct 14, 2026 •

edited

Loading

Uh oh!

Dandandan commented Oct 14, 2026

Uh oh!

adriangbot commented Oct 14, 2026

Uh oh!

adriangbot commented Oct 14, 2026

Uh oh!

comphead commented Oct 24, 2026

Uh oh!

comphead commented Oct 28, 2026

Uh oh!

comphead commented Nov 5, 2026

Uh oh!

Rich-T-kid commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Dandandan commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Oct 14, 2026

Uh oh!

adriangbot commented Oct 14, 2026

Uh oh!

adriangbot commented Oct 14, 2026

Uh oh!

comphead commented Oct 24, 2026

Uh oh!

comphead commented Oct 28, 2026

Uh oh!

comphead commented Nov 5, 2026

Uh oh!

Rich-T-kid commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Dandandan commented Oct 14, 2026 •

edited

Loading