Skip to content

Commit 56b1895

Browse files
compheadRich-T-kid
andauthored
fix(arrow-ipc): honor skip_validation for record batches in StreamDecoder (apache#12094)
# Which issue does this PR close? - Relates to apache/datafusion#26687. This picks up apache#11552 by @Dandandan with the same fix, rebased on `main`. The benchmarks are split out into apache#12093. # Rationale for this change `StreamDecoder::with_skip_validation(true)` only applies to dictionary batches, so record batches are still fully validated. `StreamReader`, `FileDecoder` and the `arrow-flight` `FlightDataDecoder` all pass the flag through for record batches too. DataFusion's spill reader has called `StreamDecoder::new().with_skip_validation(true)` since apache/datafusion#21882. Because of this bug, every spilled batch is validated again when it is read back, including a UTF-8 check of every string. That undoes apache/datafusion#15454. # What changes are included in this PR? - Pass `skip_validation` to `RecordBatchDecoder` in `StreamDecoder::decode`. - Cover `StreamDecoder` in `expect_ipc_validation_error`. # Are these changes tested? Yes. `expect_ipc_validation_error` now checks that `StreamDecoder` rejects invalid data by default and accepts it when validation is skipped. Without the fix, the five `test_validation_of_invalid_*` tests fail on the new check. Benchmarks from apache#12093 on an M5 Max, `main` against this PR: | Benchmark | main | this PR | Change | |---|---|---|---| | `StreamDecoder/no_validation/read_10` | 129.0 µs | 6.18 µs | −95.2% | | `StreamDecoder/read_10` | 128.9 µs | 128.8 µs | no change | The `StreamReader` and `FileReader` benchmarks do not change. With arrow patched to this branch, DataFusion's `spill_io/StreamReader/read_100` drops from 9.59 ms to 8.48 ms (−11.5%). That benchmark reads 100 spilled batches of 8192 rows through `StreamDecoder`. After apache#12093 merges, rebasing this PR lets `run benchmark ipc_reader` confirm these numbers on the shared runner. # Are there any significant user-facing changes? No API changes, and validation is still on by default. Callers that pass `true` now skip validation for record batches too, as documented. AI disclosure: The fix, the test and this description were written with Claude Code. I have reviewed every change and the benchmark results. Co-authored-by: RIchard Baah <[email protected]>
1 parent 71a9574 commit 56b1895

2 files changed

Lines changed: 16 additions & 0 deletions

File tree

‎arrow-ipc/src/reader.rs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2956,6 +2956,16 @@ mod tests {
29562956
reader.next().unwrap()
29572957
}
29582958

2959+
/// Return the first record batch read from the IPC Stream buffer
2960+
/// using the StreamDecoder API
2961+
fn read_stream_with_decoder(
2962+
buf: &[u8],
2963+
skip_validation: bool,
2964+
) -> Result<RecordBatch, ArrowError> {
2965+
let mut decoder = unsafe { StreamDecoder::new().with_skip_validation(skip_validation) };
2966+
Ok(decoder.decode(&mut Buffer::from(buf))?.unwrap())
2967+
}
2968+
29592969
fn roundtrip_ipc_stream(rb: &RecordBatch) -> RecordBatch {
29602970
let buf = write_stream(rb);
29612971
read_stream(&buf).unwrap()
@@ -4035,6 +4045,11 @@ mod tests {
40354045
let err = read_stream(&buf).unwrap_err();
40364046
assert_eq!(err.to_string(), expected_err);
40374047

4048+
// IPC Stream format with StreamDecoder
4049+
read_stream_with_decoder(&buf, true).unwrap();
4050+
let err = read_stream_with_decoder(&buf, false).unwrap_err();
4051+
assert_eq!(err.to_string(), expected_err);
4052+
40384053
// IPC File format
40394054
let buf = write_ipc(&rb); // write is ok
40404055
read_ipc_skip_validation(&buf).unwrap();

‎arrow-ipc/src/reader/stream.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,7 @@ impl StreamDecoder {
249249
&version,
250250
)?
251251
.with_require_alignment(self.require_alignment)
252+
.with_skip_validation(self.skip_validation.clone())
252253
.read_record_batch()?;
253254
self.state = DecoderState::default();
254255
return Ok(Some(batch));

0 commit comments

Comments
 (0)