Repository navigation
Commit 56b1895
fix(arrow-ipc): honor skip_validation for record batches in StreamDecoder (apache#12094)
# Which issue does this PR close?
- Relates to apache/datafusion#26687.
This picks up apache#11552 by @Dandandan with the same fix, rebased on `main`.
The benchmarks are split out into apache#12093.
# Rationale for this change
`StreamDecoder::with_skip_validation(true)` only applies to dictionary
batches, so record batches are still fully validated. `StreamReader`,
`FileDecoder` and the `arrow-flight` `FlightDataDecoder` all pass the
flag through for record batches too.
DataFusion's spill reader has called
`StreamDecoder::new().with_skip_validation(true)` since
apache/datafusion#21882. Because of this bug, every spilled batch is
validated again when it is read back, including a UTF-8 check of every
string. That undoes apache/datafusion#15454.
# What changes are included in this PR?
- Pass `skip_validation` to `RecordBatchDecoder` in
`StreamDecoder::decode`.
- Cover `StreamDecoder` in `expect_ipc_validation_error`.
# Are these changes tested?
Yes. `expect_ipc_validation_error` now checks that `StreamDecoder`
rejects invalid data by default and accepts it when validation is
skipped. Without the fix, the five `test_validation_of_invalid_*` tests
fail on the new check.
Benchmarks from apache#12093 on an M5 Max, `main` against this PR:
| Benchmark | main | this PR | Change |
|---|---|---|---|
| `StreamDecoder/no_validation/read_10` | 129.0 µs | 6.18 µs | −95.2% |
| `StreamDecoder/read_10` | 128.9 µs | 128.8 µs | no change |
The `StreamReader` and `FileReader` benchmarks do not change. With arrow
patched to this branch, DataFusion's `spill_io/StreamReader/read_100`
drops from 9.59 ms to 8.48 ms (−11.5%). That benchmark reads 100 spilled
batches of 8192 rows through `StreamDecoder`.
After apache#12093 merges, rebasing this PR lets `run benchmark ipc_reader`
confirm these numbers on the shared runner.
# Are there any significant user-facing changes?
No API changes, and validation is still on by default. Callers that pass
`true` now skip validation for record batches too, as documented.
AI disclosure: The fix, the test and this description were written with
Claude Code. I have reviewed every change and the benchmark results.
Co-authored-by: RIchard Baah <[email protected]>1 parent 71a9574 commit 56b1895
2 files changed
Lines changed: 16 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2956 | 2956 | | |
2957 | 2957 | | |
2958 | 2958 | | |
| 2959 | + | |
| 2960 | + | |
| 2961 | + | |
| 2962 | + | |
| 2963 | + | |
| 2964 | + | |
| 2965 | + | |
| 2966 | + | |
| 2967 | + | |
| 2968 | + | |
2959 | 2969 | | |
2960 | 2970 | | |
2961 | 2971 | | |
| |||
4035 | 4045 | | |
4036 | 4046 | | |
4037 | 4047 | | |
| 4048 | + | |
| 4049 | + | |
| 4050 | + | |
| 4051 | + | |
| 4052 | + | |
4038 | 4053 | | |
4039 | 4054 | | |
4040 | 4055 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
| 252 | + | |
252 | 253 | | |
253 | 254 | | |
254 | 255 | | |
| |||
0 commit comments