You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c4a0b1f
Browse filesBrowse the repository at this point in the historyBrowse files
fix: match Spark nullness for missing nested Parquet fields (#7085)
* fix: match Spark nullness for missing nested Parquet fields
* fix: fall back for name-mapped nested Iceberg scans
* fix: decline Iceberg scans when table properties cannot be read
* docs: migrate nested-field notes to consolidated scan guides
* docs: describe list and map nulling for all-missing nested leaves
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/iceberg.md
+8-6Lines changed: 8 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -221,6 +221,14 @@ The following scenarios will fall back to the JVM Iceberg reader:
221
221
or renamed. The native reader cannot yet match such a field to data files written before the
222
222
change. The check uses the table's schema history, so the fallback stays after those files are
223
223
rewritten
224
+
- Scans that read a struct, array, or map column from a table with
225
+
`schema.name-mapping.default`, including tables imported with `snapshot`, `migrate`, or
226
+
`add_files` ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The native reader
227
+
does not apply name mappings to nested fields. Queries that read only primitive columns can
228
+
still run natively. The fallback remains while the property is set, even if the data files
229
+
have since been rewritten with field IDs. Iceberg's nullness rules are separate from the
- Scans where no schema in the table's history gives a partition source or equality-delete key a
225
233
name that the scan's other columns don't use (every task of a native scan must give such a
226
234
column the same name)
@@ -792,12 +800,6 @@ Which entries apply depends on which native paths are enabled:
792
800
793
801
### Reads that differ from Spark
794
802
795
-
-**Migrated tables that use a name mapping.** The `migrate`, `snapshot` and `add_files`
796
-
procedures set `schema.name-mapping.default` for files without Iceberg field ids. When a query
797
-
also reads a top-level primitive column, the native scan returns NULL for nested fields (struct
798
-
fields, list elements and map values). Filters on those fields then select the wrong rows
799
-
([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix needs an iceberg-rust
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/parquet.md
+8-5Lines changed: 8 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -111,6 +111,14 @@ reads that fail where Spark succeeds are listed in
111
111
([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's
112
112
behavior.
113
113
114
+
### Missing Nested Fields
115
+
116
+
When all requested leaf fields under a nested column are missing from a Parquet file, Spark 3.4
117
+
through 4.0 return NULL for the entire missing subtree, including nested arrays and maps. Comet
118
+
follows this behavior. On Spark 4.1+, the default (`false`) preserves the file's parent nullness.
119
+
Set `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` for legacy behavior.
120
+
A field that exists in the file but contains only NULL values is not a missing field.
121
+
114
122
### Schema Mismatch Handling
115
123
116
124
The issues in this subsection apply only when the requested read schema differs from the schema written
@@ -272,11 +280,6 @@ for production, so do not turn it off to avoid an entry on this page.
272
280
read back wrong inside lists written in a legacy layout, as Hive and
273
281
`spark.sql.parquet.writeLegacyFormat=true` write them
0 commit comments