Skip to content

Commit 621b9cc

Browse files
manuzhangclaude
andauthored
fix: decode invalid UTF-8 in nested Parquet BINARY values read as strings (#7106)
* fix: decode invalid UTF-8 in nested Parquet BINARY values read as strings When a Parquet BINARY column without the UTF8 annotation is read as a string, through a read schema or spark.sql.parquet.binaryAsString, the native scan converts nested values (struct fields, list elements, map keys and values) in parquet_convert_array. That fell through to Arrow's safe cast, which turns invalid UTF-8 into NULL, so those values silently became NULL where Spark keeps the bytes. Convert Binary to Utf8 with Spark's cast instead. It replaces each ill-formed sequence with U+FFFD, the decode CAST(binary AS string) and native shuffle already share (#4764). Top-level columns are unchanged: DataFusion's file-schema coercion still decodes them straight into Utf8 (#7032). To keep the zero-copy conversion Arrow's cast gave valid nested values, cast_binary_to_string checks the bytes the rows span once for the default and UTF8 styles. It doesn't check the whole value buffer, which a slice shares with its parent. When the bytes are valid, it returns the input's value buffer narrowed to them, with a slice's offsets rebased. Otherwise rows inside the valid prefix are copied without being checked again, and only the rest go through the U+FFFD decoder. This also speeds up plain CAST(binary AS string). Split out of #6246. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: skip the UTF-8 fast path when nulls hide most of the payload cast_binary_to_string validated every byte the rows span, including the payload of null rows, which the row-by-row decoder skipped. After nullif or a CASE with column branches, a mostly-null batch can keep most of its bytes under nulls, so the cast scanned all of them and the reused buffer kept them alive downstream. Count the bytes of the non-null rows first. When null rows span more than half of the bytes, skip the check and decode the live rows one at a time as before, into a value buffer sized by the live bytes instead of the whole span. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: decode invalid UTF-8 in nested LargeBinary and BinaryView values The Parquet reader keeps the types a file's ARROW:schema stores, and DataFusion's file-schema coercion only changes top-level columns. So when an Arrow writer stores a list element, struct field or map value as LargeBinary or BinaryView, the native scan hands it to parquet_convert_array as that type. Only the Binary arm used Spark's cast, so these values still went through Arrow's safe cast and a malformed value became NULL. Convert LargeBinary and BinaryView to Binary first and use the same Spark cast. For LargeBinary that rewrites the offsets and keeps the value buffer. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: copy a valid slice that fills little of its value buffer A slice shares its parent's value buffer, and Buffer::slice_with_length narrows the view but keeps the whole allocation. When cast_binary_to_string reused a valid slice's bytes, every output batch cut from one large array, such as aggregate output, held that array's whole allocation. DataFusion's sort charges each batch the capacity of its buffers, so it counted the allocation once per batch and spilled where a compact copy didn't. Reuse the buffer only when the rows' bytes fill at least half of its allocation. Otherwise copy those already validated bytes into a compact buffer with fresh offsets, which still skips the row-by-row decoder. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor: share offsets when casting valid binary to string cast_binary_to_string rebuilt the offsets whenever it copied the values of valid input, even for an unsliced array whose offsets needed no rebasing. Decide each buffer on its own: reuse the offsets when they start at zero and fill at least half of their allocation, and the values when the rows' bytes fill at least half of theirs. An unsliced array keeps its offsets when only its value buffer is oversized, while the first slice of a large array still gets offsets that don't hold the parent's allocation. Also update the function's doc comment, which still said the bytes are taken as is, and pass parquet_convert_array_impl's Binary to Utf8 cast only the options it reads. Add rows to the nested ParquetReadSuite fixture: a null list element, struct field and map value ahead of the invalid row, and a row whose list, struct and map are null. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: decode invalid UTF-8 in nested dictionary-encoded BINARY values A file's ARROW:schema can dictionary-encode a nested BINARY leaf, which the reader then returns as Dictionary(_, Binary | LargeBinary). Those fell through to Arrow's safe cast and turned malformed values into NULL. Decode the dictionary values through the Binary arm and expand the keys. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: copy the validity bitmap a small slice shares with its parent The fast path for valid input compacted the offsets and values of a slice that fills little of their allocations, but cloned the input's validity bitmap, which still held the parent's whole allocation. A sort that charges every batch for what it retains then spilled where the base did not, for example on nullable binary group keys emitted in small slices. Apply the same rule to the bitmap: reuse it when the slice's bits fill at least half of its allocation, and copy them otherwise. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: keep the bytes of a map key read from unannotated BINARY Decoding turned the keys FF and EF BF BD into the same U+FFFD, so a lookup of one returned the other's value and map_from_entries failed with DUPLICATED_MAP_KEY where Spark, which tells keys apart by their bytes, succeeds. A key now keeps its bytes at any depth under a map's key field, unchecked, as a top-level value does. Map values still decode. Decoding a dictionary now touches only the entries the batch references, so a wide malformed entry that one batch uses is not decoded again for every other batch of the row group. A batch whose decoded values would not fit a string column is reported with the byte count instead of an offset overflow. In the binary-to-string fallback, count the live bytes over runs of non-null rows instead of row by row, so sparse nulls don't cost a pass over the batch. In the row that holds the first ill-formed sequence, copy its validated prefix and decode only the rest. Size the builder for the validated prefix plus three bytes per remaining byte, and copy the result compact when a replacement still left it filling under half of its allocation. Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix: copy the bytes of a valid binary cast outside a scan A plain CAST(binary AS string) can run on a batch imported over the Arrow C stream, where every buffer keeps the whole batch alive, wide sibling columns included, for as long as a sort or a join holds the result. Share the input's buffers only when a scan adapts its schema, where the reader owns them, and copy the validated bytes once otherwise, as the cast did before. Also cover a Dictionary(Int32, Binary) map key, link the tracking issue for the per-batch dictionary overflow from the code, and list that limit under the known issues with the batch size and the growth of ill-formed bytes. Co-Authored-By: Claude Fable 5.1 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent a828088 commit 621b9cc

6 files changed

Lines changed: 1025 additions & 32 deletions

File tree

‎docs/source/user-guide/latest/compatibility/index.md‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,8 @@ codegen dispatcher covers only scalar expressions; see the
7979
Spark's `StringType` can hold arbitrary bytes, including sequences that are not valid UTF-8 (for
8080
example `CAST(X'FF' AS STRING)`). Arrow's string type requires valid UTF-8, so Comet cannot store
8181
the raw bytes natively. When Comet produces a string from arbitrary bytes (such as
82-
`CAST(binary AS string)` or a columnar shuffle), it decodes them the same way the JVM does
82+
`CAST(binary AS string)`, a columnar shuffle, or a nested Parquet BINARY value read as a string),
83+
it decodes them the same way the JVM does
8384
(`new String(bytes, UTF_8)`), replacing each ill-formed sequence with the Unicode replacement
8485
character `U+FFFD`. Spark itself applies the identical replacement whenever such a string is
8586
materialized (collected, printed, or passed to most string functions), so the rendered result
@@ -98,9 +99,15 @@ underlying bytes rather than on the rendered text:
9899
such as `contains` can therefore disagree with Spark when non-UTF-8 bytes are involved.
99100

100101
Both differences require string data that is not valid UTF-8. Comet's native Parquet scan never
101-
produces such strings (it rejects them, see below), and neither do most string expressions. They can
102-
reach Comet in these ways, which are the ones to check before relying on a result:
103-
102+
produces such strings from UTF8-annotated columns (it rejects them, see below), and neither do most
103+
string expressions. They can reach Comet in these ways, which are the ones to check before relying on
104+
a result:
105+
106+
- **Parquet BINARY columns read as strings.** A BINARY column without the UTF8 annotation can hold
107+
bytes that are not valid UTF-8 when it is read as `string`, through the read schema or
108+
`spark.sql.parquet.binaryAsString=true`. Comet decodes such a value in a struct field, list element
109+
or map value. Top-level values and map keys are not checked; see
110+
[Parquet](../parquet.md#reads-that-differ-from-spark).
104111
- **Text files.** Spark's Text reader keeps the bytes of each line as they are, so a Text file in an
105112
encoding other than UTF-8, such as Latin-1, holds strings that are not valid UTF-8. Comet decodes
106113
them when it converts a Text scan to Arrow, which it does only when

‎docs/source/user-guide/latest/parquet.md‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -288,12 +288,14 @@ for production, so do not turn it off to avoid an entry on this page.
288288
read back wrong inside lists written in a legacy layout, as Hive and
289289
`spark.sql.parquet.writeLegacyFormat=true` write them
290290
([#7029](https://github.com/apache/datafusion-comet/issues/7029)).
291-
- **Nested BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY
292-
column without the UTF8 annotation that is read as `string`, through the read schema or
293-
`spark.sql.parquet.binaryAsString=true`. Spark returns the bytes. Comet returns NULL for each
294-
such nested value. Top-level values match Spark in a release build, because the invalid bytes
295-
reach an Arrow string array unchecked. A debug build panics on them
296-
([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of
291+
- **BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY column
292+
without the UTF8 annotation that is read as `string`, through the read schema or
293+
`spark.sql.parquet.binaryAsString=true`. Spark keeps the bytes. In a struct field, list element
294+
or map value, Comet replaces each ill-formed sequence with `U+FFFD`, as described in
295+
[Strings with non-UTF-8 bytes](compatibility/index.md#strings-with-non-utf-8-bytes). Top-level
296+
values, and map keys at any depth, reach an Arrow string array unchecked, which keeps keys that
297+
differ only in such bytes distinct: they match Spark in a release build, and a debug build
298+
panics on top-level values ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of
297299
[#4764](https://github.com/apache/datafusion-comet/issues/4764)).
298300
- **Field-id reads where one file column matches a field by id and a sibling by name.** With
299301
`spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top
@@ -362,6 +364,15 @@ With `spark.comet.write.parquet.enabled=true`:
362364
([#7035](https://github.com/apache/datafusion-comet/issues/7035)).
363365
- **A `DATE` column read as `INTERVAL YEAR TO MONTH`.** Spark reads it, and Comet fails the read
364366
([#5188](https://github.com/apache/datafusion-comet/issues/5188)).
367+
- **A nested dictionary-encoded BINARY column, read as STRING, whose batch decodes to more than
368+
2 GiB.** In a struct field, list element, map key or map value, Comet expands the dictionary
369+
into one string array per batch, whose offsets hold at most 2,147,483,647 bytes. A batch of
370+
`spark.comet.batchSize` rows (8192 by default) fails the scan when the values its rows reference
371+
add up to more, counting each reference of a shared entry, even for a query that never reads the
372+
strings. Values that are not valid UTF-8 decode to up to three times their size, since each
373+
ill-formed sequence becomes `U+FFFD`. Spark keeps the dictionary and reads it row by row. Lower
374+
`spark.comet.batchSize` so that a batch holds fewer rows
375+
([#7254](https://github.com/apache/datafusion-comet/issues/7254)).
365376

366377
### Writes that fail where Spark succeeds
367378

‎native/core/src/parquet/parquet_exec.rs‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -834,6 +834,91 @@ mod tests {
834834
assert!(err.contains("rebase"), "unexpected error: {err}");
835835
}
836836

837+
/// The reader keeps the nested types of a file's ARROW:schema, so an Arrow writer's
838+
/// `LargeBinary`, `BinaryView` or dictionary-encoded list elements reach the conversion to
839+
/// Spark's string type as those types, not as `Binary`. A malformed value must still decode
840+
/// to U+FFFD.
841+
#[tokio::test]
842+
async fn nested_arrow_binary_types_read_as_string_decode_invalid_utf8() {
843+
use arrow::array::{Array, AsArray, BinaryArray, ListArray};
844+
use arrow::buffer::OffsetBuffer;
845+
846+
for binary_type in [
847+
DataType::LargeBinary,
848+
DataType::BinaryView,
849+
DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Binary)),
850+
DataType::Dictionary(Box::new(DataType::Int8), Box::new(DataType::LargeBinary)),
851+
] {
852+
let element = Arc::new(Field::new("element", binary_type.clone(), true));
853+
let values = arrow::compute::cast(
854+
&BinaryArray::from(vec![b"one".as_slice(), &[0xff]]),
855+
&binary_type,
856+
)
857+
.unwrap();
858+
let list = ListArray::new(element, OffsetBuffer::new(vec![0, 2].into()), values, None);
859+
let file_schema = Arc::new(Schema::new(vec![Field::new(
860+
"s",
861+
list.data_type().clone(),
862+
true,
863+
)]));
864+
let batch =
865+
RecordBatch::try_new(Arc::clone(&file_schema), vec![Arc::new(list)]).unwrap();
866+
867+
let filename = get_temp_filename()
868+
.as_path()
869+
.as_os_str()
870+
.to_str()
871+
.unwrap()
872+
.to_string();
873+
let file = File::create(&filename).unwrap();
874+
let mut writer = ArrowWriter::try_new(file, file_schema, None).unwrap();
875+
writer.write(&batch).unwrap();
876+
writer.close().unwrap();
877+
878+
let string_list = DataType::List(Arc::new(Field::new("element", DataType::Utf8, true)));
879+
let required_schema = Arc::new(Schema::new(vec![Field::new("s", string_list, true)]));
880+
let session_ctx = Arc::new(SessionContext::new());
881+
let scan = init_datasource_exec(
882+
required_schema,
883+
None,
884+
None,
885+
ObjectStoreUrl::local_filesystem(),
886+
ObjectStoreBackend::Local,
887+
vec![vec![PartitionedFile::from_path(filename).unwrap()]],
888+
None,
889+
None,
890+
None,
891+
"UTC",
892+
true,
893+
false,
894+
false,
895+
false,
896+
&session_ctx,
897+
false,
898+
false,
899+
false,
900+
true,
901+
"CORRECTED",
902+
"CORRECTED",
903+
)
904+
.unwrap();
905+
906+
let mut strings = Vec::new();
907+
let mut stream = scan.execute(0, session_ctx.task_ctx()).unwrap();
908+
while let Some(batch) = stream.next().await {
909+
let batch = batch.unwrap();
910+
let values = batch.column(0).as_list::<i32>().values();
911+
let values = values.as_string::<i32>();
912+
strings.extend(values.iter().map(|v| v.map(str::to_string)));
913+
}
914+
assert_eq!(
915+
strings,
916+
vec![Some("one".to_string()), Some("\u{fffd}".to_string())],
917+
"{binary_type}"
918+
);
919+
}
920+
}
921+
837922
/// Writes a metadata-free parquet file with one INT64 `TIMESTAMP_MICROS` column (`ts`) and
838923
/// one INT96 column (`ts96`) through the low-level writer (arrow's writer cannot emit
839924
/// INT96), then scans it with the given session read modes. `int96_days` is the day count

0 commit comments

Comments
 (0)