Repository navigation
fix(parquet): validate UTF-8 when a supplied schema reads binary as a string - #11144
hassaanch23 wants to merge 6 commits into
Conversation
kita-renji
left a comment
There was a problem hiding this comment.
Thanks for picking this up. For anyone following along, this is the root cause of apache/datafusion#25509 (binary_as_string segfault).
I checked out fcdfa20 and ran a matrix against it and against its merge base: an unannotated BYTE_ARRAY column written as PLAIN, RLE_DICTIONARY, dictionary with fallback to PLAIN part way through the chunk, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY, read with a supplied schema of Utf8, LargeUtf8, Utf8View and Dictionary(Int32, Utf8 | LargeUtf8 | Utf8View), plus Struct and List children. The data covers 0xFF among nulls and empty strings, a 3-byte sequence split across two values and across a page boundary, a truncated tail with a shared prefix (for DELTA_BYTE_ARRAY), and values over 12 bytes for the view buffers.
On the merge base none of the 165 invalid cases returns an error: debug panics in 86 and returns an array that fails validate_full in the rest (the view paths go through new_unchecked); release returns an invalid array in 156. With the PR, 162 of 165 return encountered non UTF-8 data in both debug and release, and the plumbing looks right to me. Every decoder built in set_data/set_dict reads self.validate_utf8, set_page_reader sets it before any page is decoded, and validation stays one check_valid_utf8 per decoded batch, so annotated UTF8 columns cost the same as before.
The other 3 cases, and 6 valid-data cases, fail because of a gap on the dictionary target. DictionaryDecoder::new still takes the dictionary value type from the annotation (byte_array_dictionary.rs:324-329), so for an unannotated column read as Dictionary(_, Utf8) from dictionary-encoded pages, the dictionary values are built as Binary. With valid data, debug panics at dictionary_buffer.rs:182 with Child type mismatch for Dictionary(Int32, Utf8). Expected Utf8 but child data had Binary. Release returns a DictionaryArray with a Binary child through build_unchecked (dictionary_buffer.rs:183), and Dictionary(_, Utf8View) panics on as_string at byte_array_dictionary.rs:154. main behaves the same way. The new test doesn't catch this because it lists Dictionary(Int32, Utf8) only with invalid data, which errors in set_dict before the mismatch is reached.
Since it has the same root cause (the decision comes from the converted type instead of the target Arrow type), I think it belongs in this PR. This change on top of the branch made all 30 valid-data cases read correctly and all 165 invalid cases error, in debug and release:
fn set_validate_utf8(&mut self, validate_utf8: bool) {
self.validate_utf8 |= validate_utf8;
if self.validate_utf8 {
self.value_type = match V::IS_LARGE {
true => ArrowType::LargeUtf8,
false => ArrowType::Utf8,
};
}
}A separate hint for the value type would be cleaner than piggybacking on the UTF-8 flag. Either way, a valid-data Dictionary(Int32, Utf8) case in test_read_binary_as_utf8 (tests/schema.rs:567) would cover it.
One side effect to mention in the description: JSON-annotated BYTE_ARRAY maps to Utf8 by default (schema/primitive.rs:279,286), but its converted type is JSON, not UTF8, so on main it was never validated, even with no supplied schema (the base returns an invalid StringArray for PLAIN, dictionary and DELTA_LENGTH). The PR fixes that too because the flag now comes from the Arrow type. A test for it would help.
On tests generally: test_read_non_utf8_binary_as_utf8 writes three values in a single page with default properties, so it only reaches the dictionary decoder and never the PLAIN, DELTA or view decoders. Since the fix is flag plumbing, a later change could drop the flag for one decoder without the test noticing. Could you run it over a few WriterProperties (dictionary disabled with PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY) and add a null and an empty string?
Nit: GenericRecordReader::set_validate_utf8 assigns with = while the decoders OR it in. That's fine, but the trait doc at decoder.rs:95 could say that passing false doesn't turn validation off for annotated columns.
I also patched DataFusion main (arrow 60.0.0) to this branch through [patch.crates-io]. The datafusion#25509 repro now fails cleanly with encountered non UTF-8 data on both the Utf8View and Utf8 paths instead of segfaulting, and valid multibyte data still reads.
Once the dictionary value type is handled, this looks good to me.
|
@kita-renji Thank you for this. Running the matrix against the merge base as well is what made the dictionary gap visible. Everything is addressed in 67e56f4 and 315e6d1: Dictionary value type. Agreed that it's the same root cause. Instead of setting I checked the opposite direction too, a JSON. Added Encodings. The tests now write each column five ways: dictionary pages; dictionary pages that fall back to PLAIN after the first value; and PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY without a dictionary. The write helper asserts each chunk used the intended encodings, so a change in writer defaults can't quietly stop a decoder from being exercised. The invalid values now follow an empty value and a null, so in the fallback case they land in the PLAIN pages rather than the dictionary. They are read into all six targets ( Doc nit. The trait doc now says that passing
|
|
could you rebase this branch |
… string The byte array, byte view and dictionary decoders enabled UTF-8 validation only when the Parquet column was annotated `ConvertedType::UTF8`. A schema given through `ArrowReaderOptions::with_schema` can map a plain `BYTE_ARRAY` column to `Utf8`, `LargeUtf8`, `Utf8View` or a dictionary of those, so the annotation does not decide whether a string array will be built. Reading such a column skipped validation entirely and produced a string array over arbitrary bytes: a panic in debug, undefined behaviour in release. Give `ColumnValueDecoder` a `set_validate_utf8` method, defaulting to a no-op for decoders that never produce string data, and have the array readers turn it on from the type they are about to build. `GenericRecordReader` carries the flag so it also reaches decoders it creates later for subsequent pages. Validation is only ever turned on, never off, so a column annotated as a string is still validated when read as `Binary`. Co-Authored-By: Claude Opus 5 <[email protected]>
…ted as UTF-8 The dictionary decoder chose the type of its dictionary values from the Parquet annotation, separately from UTF-8 validation. An unannotated column read into a dictionary with string values was validated, but its values were still built as `Binary`. That panics in debug and returns an invalid array in release. Derive the value type from the validation flag, so the values are strings exactly when they are validated. Test the readers over dictionary, PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY pages: valid data read into every string target, invalid data next to a null and an empty value, and a `JSON` column, which is read as `Utf8` but was never validated. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…to PLAIN Write the test columns a fifth way: dictionary pages abandoned after the first value, so that the column chunk holds both RLE_DICTIONARY and PLAIN pages. Order the invalid values after an empty value and a null so they land in the PLAIN pages. Co-Authored-By: Claude Opus 5.5 <[email protected]>
315e6d1 to
61131ab
Compare
|
Rebased onto current The one conflict was with #11262, which added FLBA decoding to Locally, all of these pass:
The three new tests still fail when run against |
|
your forks CI seems to be failing still |
|
Sorry for the slow reply. The one failure was |
alamb
left a comment
There was a problem hiding this comment.
Thank you @hassaanch23
This PR feels to me like it is overly complicated for the problem it is trying to solve. I left some comments. Let me know what you think and if there is any way to make this simpler
| /// when the column is annotated as a string, or is read into a dictionary | ||
| /// with string values even though its annotation does not say so. | ||
| fn value_type(&self) -> ArrowType { | ||
| match (V::IS_LARGE, self.validate_utf8) { |
There was a problem hiding this comment.
This seems backwards -- the determination of validating UTF8 should be done based on the type of the column. Maybe we need to store the value type instead of validate_utf8?
There was a problem hiding this comment.
On the other hand, I see this is just moved to a functions (the logic is pre-existing) so it is ok
There was a problem hiding this comment.
Right, the mapping itself is unchanged. It now reads the flag instead of the annotation, so the values are built as strings exactly when they were validated. That's what fixes Dictionary<_, Utf8> over an unannotated column, which used to come back with Binary values.
| impl<K, V: OffsetSizeTrait> DictionaryDecoder<K, V> { | ||
| /// The type of the dictionary values. | ||
| /// | ||
| /// Values are built as strings exactly when they are validated as UTF-8: |
There was a problem hiding this comment.
this paragraph is pretty hard to read -- can you find some way to make ti more concise?
Also the code does not check dictionaries so I am not sure what it is trying to say
There was a problem hiding this comment.
Shortened in a2eb29e to: "The type of the dictionary values: strings if they are validated as UTF-8, binary otherwise."
| }; | ||
|
|
||
| // A schema given to the reader can map this column to a string type even | ||
| // though the Parquet annotation does not describe one, in which case the |
There was a problem hiding this comment.
Can you give an example here of when the reader would map to a string type rather than just restating?
There was a problem hiding this comment.
Done in a2eb29e. The comments in the three readers now give an example: a schema passed to ArrowReaderOptions::with_schema reading an unannotated BYTE_ARRAY column as Utf8View (or Utf8, or Dictionary<_, Utf8>).
| // A schema given to the reader can map this column to a string type even | ||
| // though the Parquet annotation does not describe one, in which case the | ||
| // decoder still has to validate the data as UTF-8. | ||
| let validate_utf8 = matches!(data_type, ArrowType::Utf8View); |
There was a problem hiding this comment.
Do we also have to check for Utf8?
There was a problem hiding this comment.
No. This reader only builds BinaryView and Utf8View, and any other type returns an error a few lines below. Utf8 and LargeUtf8 go through make_byte_array_reader, which checks both.
| } | ||
|
|
||
| /// A supplied schema that reads a `Binary` column as a string type must still | ||
| /// validate UTF-8. The readers used to enable validation only when the Parquet |
There was a problem hiding this comment.
Why do we need to document what the readers used to do? "the readers used to enable ..."
| /// UTF-8 even though its converted type is `JSON` rather than `UTF8`. | ||
| #[test] | ||
| fn test_read_non_utf8_json() { | ||
| let message_type = "message schema { REQUIRED BYTE_ARRAY json (JSON); }"; |
There was a problem hiding this comment.
Can you pleae update this test to use the ArrowWriter rather than the low level SerializedFileWriter? It is quite verbose
There was a problem hiding this comment.
Done in a2eb29e. It now writes a BinaryArray with ArrowWriter and uses ArrowWriterOptions::with_parquet_schema to give the column the JSON annotation.
| self.padding_threshold = Some(threshold); | ||
| } | ||
|
|
||
| /// Require the values decoder to validate the data as UTF-8, for a column |
There was a problem hiding this comment.
why does the record reader need to know this? Why isn't the check localized to the values reader (which knows the resulting arrow type)?
There was a problem hiding this comment.
The values decoder can't see the Arrow type. GenericRecordReader::set_page_reader creates a new decoder with CV::new(&column_desc) for every column chunk, and that constructor only gets the column descriptor. So the flag has to live on something that outlives each decoder, and the array reader only holds the record reader. I added a sentence to the setter's doc saying so.
The alternatives I could see:
- Validate in the array readers after decoding, where the Arrow type is known. That works for
Utf8,LargeUtf8andUtf8View. But dictionary values are decoded once per dictionary page and shared between batches, so they would need checking again in every batch, or caching. - Pass the Arrow type to
ColumnValueDecoder::new, which changes every decoder.
The field on the record reader seemed the smallest of the three, but I'm happy to switch if you prefer one of the others.
| // A schema given to the reader can map this column to a string type even | ||
| // though the Parquet annotation does not describe one, in which case the | ||
| // decoder still has to validate the data as UTF-8. | ||
| let validate_utf8 = matches!(data_type, ArrowType::Utf8 | ArrowType::LargeUtf8); |
There was a problem hiding this comment.
This seems like the core check to me -- when the arrow type is Utf8/LargeUtf8 then we need to validate utf8
There was a problem hiding this comment.
Yes, that's the whole fix for this reader. The view and dictionary readers make the same check for their own types.
| } | ||
|
|
||
| fn set_validate_utf8(&mut self, validate_utf8: bool) { | ||
| self.validate_utf8 |= validate_utf8; |
There was a problem hiding this comment.
|= means this can only ever turn validation on. The record reader passes its flag to every new decoder, and the flag is false by default. With =, a reader that never called set_validate_utf8 would switch validation off for a column annotated as a string.
Today = would behave the same, because a column annotated as a string can only be read as a string type, so all three readers pass true for it. I kept |= as the safer choice, but can change it if you'd rather have =.
Give the comments a concrete example of a string type the annotation does not describe, trim the trait and decoder docs, and write the JSON test with `ArrowWriter` and an explicit Parquet schema instead of the low-level `SerializedFileWriter`. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Thanks for the review @alamb. a2eb29e shortens the docs and comments and moves the JSON test to |
…rs (apache#12119) # Which issue does this PR close? - Part of apache#7044. # Rationale for this change A few `build_unchecked` calls in the parquet array readers had no comment explaining why they're safe. # What changes are included in this PR? - Added `SAFETY` comments in the list, fixed size list and fixed len byte array readers. - In the fixed len byte array reader, the buffer length check was a `debug_assert`, so it's now a real check that returns an error in release builds too. I left `OffsetBuffer::into_array` out for now, since apache#11144 is fixing the UTF-8 issue there (apache#11140). Happy to add the comment once that's merged. # Are these changes tested? Covered by existing tests. The new check shouldn't trigger for valid files. # Are there any significant user-facing changes? No. --------- Co-authored-by: Richard Baah <[email protected]>
apache#12161 decodes an unannotated BYTE_ARRAY dictionary column as Binary and converts the values with StringArray::try_from_binary, which validates UTF-8. That covers the dictionary reader this PR used to change, so the merge takes main's byte_array_dictionary.rs unchanged. The remaining changes are in the non-dictionary readers. test_read_non_utf8_binary_as_utf8 now matches "non UTF-8 data", like apache#12161's tests, because the dictionary path reports "Encountered non UTF-8 data" and the decoders "encountered non UTF-8 data". Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
I merged
@alamb @Rich-T-kid, could you take another look when you have time? |
| Arc::new(BinaryArray::from_opt_vec(vec![ | ||
| Some(b"".as_ref()), | ||
| None, | ||
| Some(b"\xDE\x00\xFF".as_ref()), |
There was a problem hiding this comment.
Could we add [b"\xC2", b"\xA2"] as two separate values here? Each value is invalid UTF-8, but together they form a valid sequence. With those values, DELTA_LENGTH_BYTE_ARRAY read as Utf8View returns an array that fails validate_full, while Utf8 panics; the Utf8View DELTA_BYTE_ARRAY path also returns an invalid array. These paths need the per-value boundary check used by OffsetBuffer::try_push.
|
|
||
| let builder = ParquetRecordBatchReaderBuilder::try_new(file.try_clone().unwrap()).unwrap(); | ||
| for column in builder.metadata().row_group(0).columns() { | ||
| let written: Vec<_> = column.encodings().collect(); |
There was a problem hiding this comment.
Could this inspect the data-page encodings instead of column.encodings()? To reproduce, write the same batch once with default properties and once with set_dictionary_page_size_limit(1), then print column.encodings().collect::<Vec<_>>(). Both report [PLAIN, RLE, RLE_DICTIONARY] because the dictionary page itself is PLAIN, even though only the second file has a PLAIN data page.
| } | ||
|
|
||
| /// Validate decoded values as UTF-8 even if the column is not annotated as a string. | ||
| pub fn set_validate_utf8(&mut self, validate_utf8: bool) { |
There was a problem hiding this comment.
Could we add a two row group test for this? A small reproducer is [b"valid", b"\xFF"] written with set_max_row_group_row_count(Some(1)), then read as Utf8 with batch size 1. The first next() should succeed and the second should return the UTF-8 error. That exercises the new decoder created by the second set_page_reader call.
Which issue does this PR close?
ArrowReaderOptions::with_schemaoverride fromBinaryto a string type skips UTF-8 validation #11140.This is also the root cause of apache/datafusion#25509, where a query with
binary_as_stringsegfaults.Rationale for this change
ByteArrayColumnValueDecoderandByteViewArrayColumnValueDecoderdecide whether to validate UTF-8 from the Parquet annotation alone:But a schema passed to
ArrowReaderOptions::with_schemacan read a plainBYTE_ARRAYcolumn asUtf8,LargeUtf8orUtf8View. That is a supported conversion, andtest_read_binary_as_utf8covers it. On that path validation is skipped, so the reader builds a string array directly over whatever bytes the column holds.OffsetBuffer::into_arraythen panics in debug builds, and in release it returns aStringArraythat is not valid UTF-8, which is undefined behaviour for anything that reads it as&str.#12161 fixed the dictionary reader's side of this. It decodes an unannotated dictionary column as
Binaryand converts the values withStringArray::try_from_binary, which validates them. This PR therefore no longer changesbyte_array_dictionary.rs; it covers the two non-dictionary readers.What changes are included in this PR?
ColumnValueDecodergainsset_validate_utf8, a no-op by default. The byte array and byte view decoders override it.make_byte_array_readerandmake_byte_view_array_readerturn it on when the Arrow type they build is a string type. That type already reflects the supplied schema.GenericRecordReaderkeeps the flag and applies it inset_page_reader, because that creates a new values decoder for every column chunk.The flag only turns validation on, so a column annotated as a string is still validated when it is read as
Binary. Because the flag comes from the Arrow type,JSONcolumns are validated too: they are read asUtf8by default, but their converted type isJSON, so onmaininvalid data in aJSONcolumn also produces an invalidStringArray.Are these changes tested?
Yes, in
parquet/src/arrow/arrow_reader/tests/schema.rs. Each test writes the column five ways: dictionary pages, dictionary pages that fall back to PLAIN part way through the chunk, and PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY pages without a dictionary.test_read_binary_as_utf8: valid data read intoUtf8,LargeUtf8,Utf8Viewand dictionaries of each, checked withvalidate_full.test_read_non_utf8_binary_as_utf8: onmainit is#[should_panic], asserting the bug. It now reads invalid data, next to a null and an empty value, into the same six types and expects a "non UTF-8 data" error. The dictionary cases pass through fix(parquet): honor requested dictionary value type #12161's validation.test_read_non_utf8_json: aJSONcolumn with invalid data, read without a supplied schema.With the four source files reverted to
main,test_read_non_utf8_binary_as_utf8andtest_read_non_utf8_jsonfail: the invalid bytes reach the panic atoffset_buffer.rs:129. With the change,cargo test -p parquet --all-featurespasses (2248 tests),cargo clippy -p parquet --all-targets --all-features -- -D warningsis clean, andcargo fmt --all -- --checkpasses. I did not runtypos.Are there any user-facing changes?
Utf8,LargeUtf8orUtf8Viewthrough a supplied schema now returns an "encountered non UTF-8 data" error, instead of panicking in debug builds or producing an invalid array in release builds.JSONcolumns with invalid UTF-8 now fail to read, instead of producing an invalidStringArray.There are no public API changes.
ColumnValueDecoderis in apub(crate)module andGenericRecordReaderin a private one.AI disclosure
Per CONTRIBUTING.md: this PR was written with AI assistance (Claude Code). The diagnosis, the design, the verification steps above and this description were all reviewed by me, and I ran every command quoted here locally.
🤖 Generated with Claude Code