Skip to content

Commit c4a0b1f

Browse files
authored
fix: match Spark nullness for missing nested Parquet fields (#7085)
* fix: match Spark nullness for missing nested Parquet fields * fix: fall back for name-mapped nested Iceberg scans * fix: decline Iceberg scans when table properties cannot be read * docs: migrate nested-field notes to consolidated scan guides * docs: describe list and map nulling for all-missing nested leaves
1 parent f1778b9 commit c4a0b1f

10 files changed

Lines changed: 534 additions & 78 deletions

File tree

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,14 @@ The following scenarios will fall back to the JVM Iceberg reader:
221221
or renamed. The native reader cannot yet match such a field to data files written before the
222222
change. The check uses the table's schema history, so the fallback stays after those files are
223223
rewritten
224+
- Scans that read a struct, array, or map column from a table with
225+
`schema.name-mapping.default`, including tables imported with `snapshot`, `migrate`, or
226+
`add_files` ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The native reader
227+
does not apply name mappings to nested fields. Queries that read only primitive columns can
228+
still run natively. The fallback remains while the property is set, even if the data files
229+
have since been rewritten with field IDs. Iceberg's nullness rules are separate from the
230+
Parquet legacy setting; [#6934](https://github.com/apache/datafusion-comet/issues/6934) tracks
231+
the reader update and review of this fallback
224232
- Scans where no schema in the table's history gives a partition source or equality-delete key a
225233
name that the scan's other columns don't use (every task of a native scan must give such a
226234
column the same name)
@@ -792,12 +800,6 @@ Which entries apply depends on which native paths are enabled:
792800

793801
### Reads that differ from Spark
794802

795-
- **Migrated tables that use a name mapping.** The `migrate`, `snapshot` and `add_files`
796-
procedures set `schema.name-mapping.default` for files without Iceberg field ids. When a query
797-
also reads a top-level primitive column, the native scan returns NULL for nested fields (struct
798-
fields, list elements and map values). Filters on those fields then select the wrong rows
799-
([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix needs an iceberg-rust
800-
upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)).
801803
- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such
802804
a file as NULL, and the native scan returns the values stored in the file
803805
([#6794](https://github.com/apache/datafusion-comet/issues/6794)).

‎docs/source/user-guide/latest/parquet.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,14 @@ reads that fail where Spark succeeds are listed in
111111
([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's
112112
behavior.
113113

114+
### Missing Nested Fields
115+
116+
When all requested leaf fields under a nested column are missing from a Parquet file, Spark 3.4
117+
through 4.0 return NULL for the entire missing subtree, including nested arrays and maps. Comet
118+
follows this behavior. On Spark 4.1+, the default (`false`) preserves the file's parent nullness.
119+
Set `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` for legacy behavior.
120+
A field that exists in the file but contains only NULL values is not a missing field.
121+
114122
### Schema Mismatch Handling
115123

116124
The issues in this subsection apply only when the requested read schema differs from the schema written
@@ -272,11 +280,6 @@ for production, so do not turn it off to avoid an entry on this page.
272280
read back wrong inside lists written in a legacy layout, as Hive and
273281
`spark.sql.parquet.writeLegacyFormat=true` write them
274282
([#7029](https://github.com/apache/datafusion-comet/issues/7029)).
275-
- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the
276-
column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5
277-
and 4.0, and to Spark 4.1 and later with
278-
`spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true`
279-
([#7030](https://github.com/apache/datafusion-comet/issues/7030)).
280283
- **Nested BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY
281284
column without the UTF8 annotation that is read as `string`, through the read schema or
282285
`spark.sql.parquet.binaryAsString=true`. Spark returns the bytes. Comet returns NULL for each

0 commit comments

Comments
 (0)