Repository navigation
Parquet reader: option to replace invalid UTF-8 instead of failing #12047
Description
Is your feature request related to a problem or challenge?
The Parquet reader fails the whole read when a STRING column holds bytes that are not valid UTF-8:
Parquet error: encountered non UTF-8 data: invalid utf-8 sequence of 1 bytes from index 80822
Files like this are unfortunately common in practice, for example Latin-1 text written as STRING by older producers. Other readers handle them:
- Arrow C++ (and so pyarrow and R) and arrow-go don't fail. They put the raw bytes into the string array and leave the UTF-8 check to later steps (ValidateFull). Users can still get at the data and clean it up.
- The JVM side doesn't fail either. parquet-java and iceberg-java turn the bytes into strings with U+FFFD for invalid sequences, and Spark reads them without error.
- arrow-rs fails, and there is no workaround:
- The pass-through approach of C++ and Go is not possible: StringArray must hold valid UTF-8, which is why feat: thread flag throughout reader code #11013 was closed.
- Reading the column as Binary is rejected (requested Binary but found Utf8), because validation is keyed on the Parquet UTF8 annotation, not on the requested Arrow type.
- Stripping the annotation reads the bytes unchecked, which is parquet: an
ArrowReaderOptions::with_schemaoverride fromBinaryto a string type skips UTF-8 validation #11140. - The error doesn't name the column.
This blocks the Rust engines that need to read what the JVM ecosystem reads. DataFusion Comet is stuck on exactly this: its native scan fails where Spark succeeds (apache/datafusion-comet#4121, tracked in apache/datafusion-comet#4764).
Describe the solution you'd like
A reader option that picks what happens on invalid UTF-8:
pub enum InvalidUtf8 {
/// Fail the read (current behavior, default)
Error,
/// Replace invalid sequences with U+FFFD per value, as String::from_utf8_lossy does
Replace,
}
ArrowReaderOptions::with_invalid_utf8(InvalidUtf8)
- Output is always valid UTF-8
- New behaviour is opt-in, so by default nothing changes
- The replacement is consistent with what the
stdRust library already does inString::from_utf8_lossy - Results match the Java/Spark ecosystem, which already replaces, and this unblocks Comet.
- No cost on valid data. Validation stays as it is. Only the existing error branch changes.
Describe alternatives you've considered
- Pass the bytes through, as C++ and Go do: that would be Proposal: Add
unsafeoption to disable UTF8 validation on parquet read #6701, already discarded in feat: thread flag throughout reader code #11013. - Read as Binary and convert in the caller: the hint is rejected for STRING columns. Even if it were allowed, the column loses its string type, so string predicates and statistics pruning stop working.
- Strip the UTF8 annotation from the metadata, then read as Utf8: the bytes are not validated and end up in a StringArray (parquet: an
ArrowReaderOptions::with_schemaoverride fromBinaryto a string type skips UTF-8 validation #11140). Once fix(parquet): validate UTF-8 when a supplied schema reads binary as a string #11144 lands, this fails exactly as today.
Additional context
No response
Activity
andygrove commentedon Nov 4, 2026
+1 to this. DataFusion Comet would use
Replace, and it would let us close several issues tracked in apache/datafusion-comet#4764.Two points on the design.
The check should key on the target type, not only the annotation. The byte array decoders validate UTF-8 only when the column is annotated
UTF8. A BINARY column without the annotation that is read asUtf8,LargeUtf8,Utf8View, or a dictionary of those, throughArrowReaderOptions::with_schema, is never checked, so it produces an invalid string array today (#12063). With the check keyed on the annotation alone,Replacewould not apply there, and that is the other half of what Comet needs: Spark reads such columns as strings withspark.sql.parquet.binaryAsString. DataFusion also hits this through its binary-to-string schema coercion (apache/datafusion#27065).I had opened #12064 to fix that case on its own (validate whenever the target is a string type, and error on invalid UTF-8). I have closed it in favor of folding it into this option, since
Error/Replaceis the better place to decide what happens. The branch may still be useful: it threads the flag through the byte array, byte view and dictionary decoders, fixes the dictionary decoder's value type for string dictionaries over unannotated columns (they returnedBinaryvalues, or panicked forUtf8View), and has a test over the six string types with plain and dictionary encoding.Replacement semantics.
String::from_utf8_lossyfollows the Unicode "maximal subpart" practice. The JVM's UTF-8 decoder, which Spark uses, emits a different number of U+FFFD for some ill-formed sequences, encoded surrogates (ED A0..BF ..) among them. For Comet it would help ifReplacematched the JVM, or if the replacement were pluggable. If that is out of scope, matchingfrom_utf8_lossyis still a big improvement, and we can document the difference.Are you planning to implement this? If not, I'd be happy to put up a PR building on the #12064 branch.
alamb commentedon Nov 4, 2026
This seems like the right behavior to me. If the file is invalid utf8, we shouldn't be able to put it in a StringArray that requires utf8 for safetly
It would be nice to make a small PR to improve the error message
alamb commentedon Nov 4, 2026
I suggest we
It seems to me they are both worthwhile and independent
antonireus commentedon Nov 4, 2026
@andygrove
@alamb
I agree, I mentioned it only to make aware that stripping the annotation is a dead end, as #11140 should eventually be fixed
hassaanch23 commentedon Nov 6, 2026
For step 1, #11144 is that standalone fix, for #11140. It has been open since September, and I rebased it onto
mainlast week. It takes the same approach as #12064:set_validate_utf8based on the Arrow type they build:Utf8,LargeUtf8,Utf8View, or a dictionary of one of those.Dictionary<_, Utf8View>included, now get string values.The tests read valid and invalid data into each string type, over five kinds of pages: dictionary, a dictionary that falls back to PLAIN, PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY. They also cover a
JSONcolumn, which is read asUtf8but was never validated. @kita-renji tested it against 165 cases, and @Rich-T-kid is reviewing it now.If that works for everyone, it could land as step 1, and the
InvalidUtf8option could then build on the same flag. cc @andygrove, since #12064 makes the same change.alamb commentedon Nov 8, 2026
Thanks -- I left some comments (I am hoping we can simplify that PR before merge, but it is a good start)