Skip to content

fix(parquet): validate UTF-8 when a supplied schema reads binary as a string - #11144

Open
hassaanch23 wants to merge 6 commits into
apache:mainfrom
hassaanch23:fix/schema-hint-utf8-validation
Open

hassaanch23 wants to merge 6 commits into
apache:mainfrom
hassaanch23:fix/schema-hint-utf8-validation

Conversation

@hassaanch23

@hassaanch23 hassaanch23 commented Sep 20, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

This is also the root cause of apache/datafusion#25509, where a query with binary_as_string segfaults.

Rationale for this change

ByteArrayColumnValueDecoder and ByteViewArrayColumnValueDecoder decide whether to validate UTF-8 from the Parquet annotation alone:

let validate_utf8 = col.converted_type() == ConvertedType::UTF8;

But a schema passed to ArrowReaderOptions::with_schema can read a plain BYTE_ARRAY column as Utf8, LargeUtf8 or Utf8View. That is a supported conversion, and test_read_binary_as_utf8 covers it. On that path validation is skipped, so the reader builds a string array directly over whatever bytes the column holds. OffsetBuffer::into_array then panics in debug builds, and in release it returns a StringArray that is not valid UTF-8, which is undefined behaviour for anything that reads it as &str.

#12161 fixed the dictionary reader's side of this. It decodes an unannotated dictionary column as Binary and converts the values with StringArray::try_from_binary, which validates them. This PR therefore no longer changes byte_array_dictionary.rs; it covers the two non-dictionary readers.

What changes are included in this PR?

  • ColumnValueDecoder gains set_validate_utf8, a no-op by default. The byte array and byte view decoders override it.
  • make_byte_array_reader and make_byte_view_array_reader turn it on when the Arrow type they build is a string type. That type already reflects the supplied schema.
  • GenericRecordReader keeps the flag and applies it in set_page_reader, because that creates a new values decoder for every column chunk.

The flag only turns validation on, so a column annotated as a string is still validated when it is read as Binary. Because the flag comes from the Arrow type, JSON columns are validated too: they are read as Utf8 by default, but their converted type is JSON, so on main invalid data in a JSON column also produces an invalid StringArray.

Are these changes tested?

Yes, in parquet/src/arrow/arrow_reader/tests/schema.rs. Each test writes the column five ways: dictionary pages, dictionary pages that fall back to PLAIN part way through the chunk, and PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY pages without a dictionary.

  • test_read_binary_as_utf8: valid data read into Utf8, LargeUtf8, Utf8View and dictionaries of each, checked with validate_full.
  • test_read_non_utf8_binary_as_utf8: on main it is #[should_panic], asserting the bug. It now reads invalid data, next to a null and an empty value, into the same six types and expects a "non UTF-8 data" error. The dictionary cases pass through fix(parquet): honor requested dictionary value type #12161's validation.
  • test_read_non_utf8_json: a JSON column with invalid data, read without a supplied schema.

With the four source files reverted to main, test_read_non_utf8_binary_as_utf8 and test_read_non_utf8_json fail: the invalid bytes reach the panic at offset_buffer.rs:129. With the change, cargo test -p parquet --all-features passes (2248 tests), cargo clippy -p parquet --all-targets --all-features -- -D warnings is clean, and cargo fmt --all -- --check passes. I did not run typos.

Are there any user-facing changes?

  • Reading a non-string column as Utf8, LargeUtf8 or Utf8View through a supplied schema now returns an "encountered non UTF-8 data" error, instead of panicking in debug builds or producing an invalid array in release builds.
  • JSON columns with invalid UTF-8 now fail to read, instead of producing an invalid StringArray.

There are no public API changes. ColumnValueDecoder is in a pub(crate) module and GenericRecordReader in a private one.

AI disclosure

Per CONTRIBUTING.md: this PR was written with AI assistance (Claude Code). The diagnosis, the design, the verification steps above and this description were all reviewed by me, and I ran every command quoted here locally.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Sep 20, 2026

@kita-renji kita-renji left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up. For anyone following along, this is the root cause of apache/datafusion#25509 (binary_as_string segfault).

I checked out fcdfa20 and ran a matrix against it and against its merge base: an unannotated BYTE_ARRAY column written as PLAIN, RLE_DICTIONARY, dictionary with fallback to PLAIN part way through the chunk, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY, read with a supplied schema of Utf8, LargeUtf8, Utf8View and Dictionary(Int32, Utf8 | LargeUtf8 | Utf8View), plus Struct and List children. The data covers 0xFF among nulls and empty strings, a 3-byte sequence split across two values and across a page boundary, a truncated tail with a shared prefix (for DELTA_BYTE_ARRAY), and values over 12 bytes for the view buffers.

On the merge base none of the 165 invalid cases returns an error: debug panics in 86 and returns an array that fails validate_full in the rest (the view paths go through new_unchecked); release returns an invalid array in 156. With the PR, 162 of 165 return encountered non UTF-8 data in both debug and release, and the plumbing looks right to me. Every decoder built in set_data/set_dict reads self.validate_utf8, set_page_reader sets it before any page is decoded, and validation stays one check_valid_utf8 per decoded batch, so annotated UTF8 columns cost the same as before.

The other 3 cases, and 6 valid-data cases, fail because of a gap on the dictionary target. DictionaryDecoder::new still takes the dictionary value type from the annotation (byte_array_dictionary.rs:324-329), so for an unannotated column read as Dictionary(_, Utf8) from dictionary-encoded pages, the dictionary values are built as Binary. With valid data, debug panics at dictionary_buffer.rs:182 with Child type mismatch for Dictionary(Int32, Utf8). Expected Utf8 but child data had Binary. Release returns a DictionaryArray with a Binary child through build_unchecked (dictionary_buffer.rs:183), and Dictionary(_, Utf8View) panics on as_string at byte_array_dictionary.rs:154. main behaves the same way. The new test doesn't catch this because it lists Dictionary(Int32, Utf8) only with invalid data, which errors in set_dict before the mismatch is reached.

Since it has the same root cause (the decision comes from the converted type instead of the target Arrow type), I think it belongs in this PR. This change on top of the branch made all 30 valid-data cases read correctly and all 165 invalid cases error, in debug and release:

fn set_validate_utf8(&mut self, validate_utf8: bool) {
    self.validate_utf8 |= validate_utf8;
    if self.validate_utf8 {
        self.value_type = match V::IS_LARGE {
            true => ArrowType::LargeUtf8,
            false => ArrowType::Utf8,
        };
    }
}

A separate hint for the value type would be cleaner than piggybacking on the UTF-8 flag. Either way, a valid-data Dictionary(Int32, Utf8) case in test_read_binary_as_utf8 (tests/schema.rs:567) would cover it.

One side effect to mention in the description: JSON-annotated BYTE_ARRAY maps to Utf8 by default (schema/primitive.rs:279,286), but its converted type is JSON, not UTF8, so on main it was never validated, even with no supplied schema (the base returns an invalid StringArray for PLAIN, dictionary and DELTA_LENGTH). The PR fixes that too because the flag now comes from the Arrow type. A test for it would help.

On tests generally: test_read_non_utf8_binary_as_utf8 writes three values in a single page with default properties, so it only reaches the dictionary decoder and never the PLAIN, DELTA or view decoders. Since the fix is flag plumbing, a later change could drop the flag for one decoder without the test noticing. Could you run it over a few WriterProperties (dictionary disabled with PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY) and add a null and an empty string?

Nit: GenericRecordReader::set_validate_utf8 assigns with = while the decoders OR it in. That's fine, but the trait doc at decoder.rs:95 could say that passing false doesn't turn validation off for annotated columns.

I also patched DataFusion main (arrow 60.0.0) to this branch through [patch.crates-io]. The datafusion#25509 repro now fails cleanly with encountered non UTF-8 data on both the Utf8View and Utf8 paths instead of segfaulting, and valid multibyte data still reads.

Once the dictionary value type is handled, this looks good to me.

@hassaanch23

Copy link
Copy Markdown
Author

@kita-renji Thank you for this. Running the matrix against the merge base as well is what made the dictionary gap visible. Everything is addressed in 67e56f4 and 315e6d1:

Dictionary value type. Agreed that it's the same root cause. Instead of setting value_type inside set_validate_utf8, I removed the value_type field. DictionaryDecoder::value_type() now derives it from validate_utf8, so dictionary values are strings exactly when they are validated, and the two can't drift apart again. With only that change reverted, the new valid-data test fails with the Child type mismatch for Dictionary(Int32, Utf8). Expected Utf8 but child data had Binary you reported.

I checked the opposite direction too, a UTF8 column read into Dictionary(_, Binary). The schema check already rejects it (requested Dictionary(Int32, Binary) but found Utf8), so a validated column can never need Binary values.

JSON. Added test_read_non_utf8_json: a JSON column holding invalid UTF-8, read without a supplied schema. It fails on the merge base and returns encountered non UTF-8 data here. The description now mentions it.

Encodings. The tests now write each column five ways: dictionary pages; dictionary pages that fall back to PLAIN after the first value; and PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY without a dictionary. The write helper asserts each chunk used the intended encodings, so a change in writer defaults can't quietly stop a decoder from being exercised.

The invalid values now follow an empty value and a null, so in the fallback case they land in the PLAIN pages rather than the dictionary. They are read into all six targets (Utf8, LargeUtf8, Utf8View, and a dictionary of each). Valid data goes into the same six and is checked with validate_full.

Doc nit. The trait doc now says that passing false does not turn validation off for a column annotated as a string.

cargo test -p parquet (with and without --all-features) and clippy are clean.

@Rich-T-kid

Copy link
Copy Markdown
Contributor

could you rebase this branch

hassaanch23 and others added 3 commits October 27, 2026 19:38
… string

The byte array, byte view and dictionary decoders enabled UTF-8 validation
only when the Parquet column was annotated `ConvertedType::UTF8`. A schema
given through `ArrowReaderOptions::with_schema` can map a plain `BYTE_ARRAY`
column to `Utf8`, `LargeUtf8`, `Utf8View` or a dictionary of those, so the
annotation does not decide whether a string array will be built. Reading such
a column skipped validation entirely and produced a string array over
arbitrary bytes: a panic in debug, undefined behaviour in release.

Give `ColumnValueDecoder` a `set_validate_utf8` method, defaulting to a no-op
for decoders that never produce string data, and have the array readers turn
it on from the type they are about to build. `GenericRecordReader` carries the
flag so it also reaches decoders it creates later for subsequent pages.

Validation is only ever turned on, never off, so a column annotated as a
string is still validated when read as `Binary`.

Co-Authored-By: Claude Opus 5 <[email protected]>
…ted as UTF-8

The dictionary decoder chose the type of its dictionary values from the
Parquet annotation, separately from UTF-8 validation. An unannotated column
read into a dictionary with string values was validated, but its values were
still built as `Binary`. That panics in debug and returns an invalid array in
release. Derive the value type from the validation flag, so the values are
strings exactly when they are validated.

Test the readers over dictionary, PLAIN, DELTA_LENGTH_BYTE_ARRAY and
DELTA_BYTE_ARRAY pages: valid data read into every string target, invalid
data next to a null and an empty value, and a `JSON` column, which is read as
`Utf8` but was never validated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…to PLAIN

Write the test columns a fifth way: dictionary pages abandoned after the
first value, so that the column chunk holds both RLE_DICTIONARY and PLAIN
pages. Order the invalid values after an empty value and a null so they land
in the PLAIN pages.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@hassaanch23
hassaanch23 force-pushed the fix/schema-hint-utf8-validation branch from 315e6d1 to 61131ab Compare October 27, 2026 14:46
@hassaanch23

Copy link
Copy Markdown
Author

Rebased onto current main.

The one conflict was with #11262, which added FLBA decoding to DictionaryDecoder. I kept its flba_column_descr, and the value type is still derived from validate_utf8. An FLBA column can't be read into a string dictionary, because a supplied schema asking for that is rejected as incompatible. So the new fixed-width path doesn't need UTF-8 validation.

Locally, all of these pass:

  • cargo test -p parquet --all-features
  • cargo clippy -p parquet --all-targets --all-features -- -D warnings
  • cargo fmt

The three new tests still fail when run against main's readers.

@Rich-T-kid

Copy link
Copy Markdown
Contributor

your forks CI seems to be failing still

@hassaanch23

Copy link
Copy Markdown
Author

Sorry for the slow reply. The one failure was rust / Test on Mac. GitHub never assigned it a runner ("The job was not acquired by Runner of type hosted even after multiple attempts"), so it was cancelled without running. The other 28 jobs, including the parquet tests and clippy, passed. I've started that job again.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @hassaanch23

This PR feels to me like it is overly complicated for the problem it is trying to solve. I left some comments. Let me know what you think and if there is any way to make this simpler

/// when the column is annotated as a string, or is read into a dictionary
/// with string values even though its annotation does not say so.
fn value_type(&self) -> ArrowType {
match (V::IS_LARGE, self.validate_utf8) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems backwards -- the determination of validating UTF8 should be done based on the type of the column. Maybe we need to store the value type instead of validate_utf8?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, I see this is just moved to a functions (the logic is pre-existing) so it is ok

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the mapping itself is unchanged. It now reads the flag instead of the annotation, so the values are built as strings exactly when they were validated. That's what fixes Dictionary<_, Utf8> over an unannotated column, which used to come back with Binary values.

impl<K, V: OffsetSizeTrait> DictionaryDecoder<K, V> {
/// The type of the dictionary values.
///
/// Values are built as strings exactly when they are validated as UTF-8:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this paragraph is pretty hard to read -- can you find some way to make ti more concise?

Also the code does not check dictionaries so I am not sure what it is trying to say

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shortened in a2eb29e to: "The type of the dictionary values: strings if they are validated as UTF-8, binary otherwise."

};

// A schema given to the reader can map this column to a string type even
// though the Parquet annotation does not describe one, in which case the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give an example here of when the reader would map to a string type rather than just restating?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a2eb29e. The comments in the three readers now give an example: a schema passed to ArrowReaderOptions::with_schema reading an unannotated BYTE_ARRAY column as Utf8View (or Utf8, or Dictionary<_, Utf8>).

// A schema given to the reader can map this column to a string type even
// though the Parquet annotation does not describe one, in which case the
// decoder still has to validate the data as UTF-8.
let validate_utf8 = matches!(data_type, ArrowType::Utf8View);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also have to check for Utf8?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. This reader only builds BinaryView and Utf8View, and any other type returns an error a few lines below. Utf8 and LargeUtf8 go through make_byte_array_reader, which checks both.

}

/// A supplied schema that reads a `Binary` column as a string type must still
/// validate UTF-8. The readers used to enable validation only when the Parquet

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to document what the readers used to do? "the readers used to enable ..."

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in a2eb29e.

/// UTF-8 even though its converted type is `JSON` rather than `UTF8`.
#[test]
fn test_read_non_utf8_json() {
let message_type = "message schema { REQUIRED BYTE_ARRAY json (JSON); }";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you pleae update this test to use the ArrowWriter rather than the low level SerializedFileWriter? It is quite verbose

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a2eb29e. It now writes a BinaryArray with ArrowWriter and uses ArrowWriterOptions::with_parquet_schema to give the column the JSON annotation.

Comment thread parquet/src/arrow/record_reader/mod.rs Outdated
self.padding_threshold = Some(threshold);
}

/// Require the values decoder to validate the data as UTF-8, for a column

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does the record reader need to know this? Why isn't the check localized to the values reader (which knows the resulting arrow type)?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The values decoder can't see the Arrow type. GenericRecordReader::set_page_reader creates a new decoder with CV::new(&column_desc) for every column chunk, and that constructor only gets the column descriptor. So the flag has to live on something that outlives each decoder, and the array reader only holds the record reader. I added a sentence to the setter's doc saying so.

The alternatives I could see:

  • Validate in the array readers after decoding, where the Arrow type is known. That works for Utf8, LargeUtf8 and Utf8View. But dictionary values are decoded once per dictionary page and shared between batches, so they would need checking again in every batch, or caching.
  • Pass the Arrow type to ColumnValueDecoder::new, which changes every decoder.

The field on the record reader seemed the smallest of the three, but I'm happy to switch if you prefer one of the others.

// A schema given to the reader can map this column to a string type even
// though the Parquet annotation does not describe one, in which case the
// decoder still has to validate the data as UTF-8.
let validate_utf8 = matches!(data_type, ArrowType::Utf8 | ArrowType::LargeUtf8);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like the core check to me -- when the arrow type is Utf8/LargeUtf8 then we need to validate utf8

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's the whole fix for this reader. The view and dictionary readers make the same check for their own types.

}

fn set_validate_utf8(&mut self, validate_utf8: bool) {
self.validate_utf8 |= validate_utf8;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why |= ? Why not =?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

|= means this can only ever turn validation on. The record reader passes its flag to every new decoder, and the flag is false by default. With =, a reader that never called set_validate_utf8 would switch validation off for a column annotated as a string.

Today = would behave the same, because a column annotated as a string can only be read as a string type, so all three readers pass true for it. I kept |= as the safer choice, but can change it if you'd rather have =.

Give the comments a concrete example of a string type the annotation does
not describe, trim the trait and decoder docs, and write the JSON test with
`ArrowWriter` and an explicit Parquet schema instead of the low-level
`SerializedFileWriter`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@hassaanch23

Copy link
Copy Markdown
Author

Thanks for the review @alamb. a2eb29e shortens the docs and comments and moves the JSON test to ArrowWriter. I replied inline on the two design questions: why the flag goes through the record reader, and |= vs =. If you'd rather validate in the array readers after decoding, I can rework it that way.

mbutrovich pushed a commit to mbutrovich/arrow-rs that referenced this pull request Nov 10, 2026
…rs (apache#12119)

# Which issue does this PR close?

- Part of apache#7044.

# Rationale for this change

A few `build_unchecked` calls in the parquet array readers had no
comment explaining why they're safe.

# What changes are included in this PR?

- Added `SAFETY` comments in the list, fixed size list and fixed len
byte array readers.
- In the fixed len byte array reader, the buffer length check was a
`debug_assert`, so it's now a real check that returns an error in
release builds too.

I left `OffsetBuffer::into_array` out for now, since apache#11144 is fixing
the UTF-8 issue there (apache#11140). Happy to add the comment once that's
merged.

# Are these changes tested?

Covered by existing tests. The new check shouldn't trigger for valid
files.

# Are there any significant user-facing changes?

No.

---------

Co-authored-by: Richard Baah <[email protected]>
hassaanch23 and others added 2 commits November 13, 2026 18:08
apache#12161 decodes an unannotated BYTE_ARRAY dictionary column as Binary and
converts the values with StringArray::try_from_binary, which validates
UTF-8. That covers the dictionary reader this PR used to change, so the
merge takes main's byte_array_dictionary.rs unchanged. The remaining
changes are in the non-dictionary readers.

test_read_non_utf8_binary_as_utf8 now matches "non UTF-8 data", like
apache#12161's tests, because the dictionary path reports "Encountered non
UTF-8 data" and the decoders "encountered non UTF-8 data".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@hassaanch23

Copy link
Copy Markdown
Author

I merged main now that #12161 is in. #12161 validates the dictionary reader's values with StringArray::try_from_binary, so this PR no longer touches byte_array_dictionary.rs. It now only covers the two non-dictionary readers, which makes it quite a bit smaller. I also shortened the comments and updated the description.

cargo test -p parquet --all-features passes (2248 tests), and clippy with -D warnings is clean. With the four source files reverted to main, the new tests fail at the offset_buffer.rs panic.

@alamb @Rich-T-kid, could you take another look when you have time?

@Rich-T-kid Rich-T-kid left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this is rebased, could you trim the description to match the resulting diff?

Arc::new(BinaryArray::from_opt_vec(vec![
Some(b"".as_ref()),
None,
Some(b"\xDE\x00\xFF".as_ref()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add [b"\xC2", b"\xA2"] as two separate values here? Each value is invalid UTF-8, but together they form a valid sequence. With those values, DELTA_LENGTH_BYTE_ARRAY read as Utf8View returns an array that fails validate_full, while Utf8 panics; the Utf8View DELTA_BYTE_ARRAY path also returns an invalid array. These paths need the per-value boundary check used by OffsetBuffer::try_push.


let builder = ParquetRecordBatchReaderBuilder::try_new(file.try_clone().unwrap()).unwrap();
for column in builder.metadata().row_group(0).columns() {
let written: Vec<_> = column.encodings().collect();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this inspect the data-page encodings instead of column.encodings()? To reproduce, write the same batch once with default properties and once with set_dictionary_page_size_limit(1), then print column.encodings().collect::<Vec<_>>(). Both report [PLAIN, RLE, RLE_DICTIONARY] because the dictionary page itself is PLAIN, even though only the second file has a PLAIN data page.

}

/// Validate decoded values as UTF-8 even if the column is not annotated as a string.
pub fn set_validate_utf8(&mut self, validate_utf8: bool) {

@Rich-T-kid Rich-T-kid Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a two row group test for this? A small reproducer is [b"valid", b"\xFF"] written with set_max_row_group_row_count(Some(1)), then read as Utf8 with batch size 1. The first next() should succeed and the second should return the UTF-8 error. That exercises the new decoder created by the second set_page_reader call.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parquet: an ArrowReaderOptions::with_schema override from Binary to a string type skips UTF-8 validation

4 participants