You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 844e8a7
Browse filesBrowse the repository at this point in the historyBrowse files
docs: address review feedback on the known issues page
Drop #6264 and #6837, which are fixed on main. Scope the datetime rebase
entry to the native Parquet scan and say the Delta scan rebases. Add the
ignored Parquet writer properties from #5304 and note that the row group
failure did not reproduce. Explain how apache/iceberg#17320 covers the
identity partition source case. Reword several entries, break the list of
Spark SQL tests into one bullet each, and note that spark.comet.scan.enabled
is a testing setting.
-**Field-id reads in which one file column matches a field by id and a sibling by name.** With
77
+
-**Field-id reads where one file column matches a field by id and a sibling by name.** With
74
78
`spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top
75
-
level, and the values of the field matched by id inside a struct. Spark's own readers disagree
76
-
with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)).
79
+
level. Inside a struct, it returns the values of the field matched by id. Spark's own readers
80
+
disagree here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)).
77
81
-**`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and
78
82
in every column of a scan that has data filters, Comet returns NULL where Spark fails with
79
83
`ArithmeticException: long overflow`
@@ -85,20 +89,15 @@ Which entries apply depends on which native paths are enabled:
85
89
`INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and
-**Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic
89
-
partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of
90
-
the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as
91
-
with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan
92
-
as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)).
93
92
94
93
### Iceberg reads
95
94
96
-
-**Migrated tables that use a name mapping.**For data files without Iceberg field ids that are
97
-
read through`schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files`
98
-
procedures set, the native scan returns NULL for nested fields (struct fields, list elements and
99
-
map values) whenever the query also reads a top-level primitive column, so filters on them select
100
-
the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes
101
-
with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)).
95
+
-**Migrated tables that use a name mapping.**The `migrate`, `snapshot` and `add_files`
96
+
procedures set`schema.name-mapping.default` for files without Iceberg field ids. When a query
97
+
also reads a top-level primitive column, the native scan returns NULL for nested fields (struct
98
+
fields, list elements and map values). Filters on those fields then select the wrong rows
99
+
([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix needs an iceberg-rust
@@ -129,37 +128,39 @@ The native Parquet writer is experimental and disabled by default. With
129
128
comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ`
130
129
columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048),
-**Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema
139
-
([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR`
140
-
([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union
141
-
([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default
142
-
columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after
([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output
- CTAS with `CHAR_AS_VARCHAR` ([#3419](https://github.com/apache/datafusion-comet/issues/3419))
143
+
- CTAS with a union ([#3429](https://github.com/apache/datafusion-comet/issues/3429))
144
+
-`INSERT INTO` with default columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421))
145
+
-`INSERT INTO` after `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422))
146
+
-`INSERT INTO` from non-projection queries ([#3423](https://github.com/apache/datafusion-comet/issues/3423))
147
+
- the `INSERT OVERWRITE` output schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424))
-`INSERT OVERWRITE LOCAL DIRECTORY` ([#3420](https://github.com/apache/datafusion-comet/issues/3420))
-**`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed,
161
-
where iceberg-java rejects the value. It also trims or lowercases a few other settings that
162
-
iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)).
157
+
-**Brotli tables.** The native writer commits brotli-compressed files. JVM Parquet readers can
158
+
read them only if Hadoop's `BrotliCodec` is on the classpath. iceberg-java fails the write
[Parquet Compatibility](scans.md) describes the reads this affects.
183
-
-**A duplicate field id in a column the query does not read.** With
184
+
-**A duplicate field id in a column the query does not read.** With a valid read schema and
184
185
`spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same
185
186
field id fails the scan whenever the table schema has a field with that id, even when the query
186
187
does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)).
@@ -222,9 +223,6 @@ Spark, and fail where Spark succeeds:
222
223
-**Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other
223
224
configurations resolve differently from Hadoop's ABFS driver
@@ -258,7 +257,10 @@ In these cases Comet's result differs from Spark's, but the bug is in Spark or i
258
257
native Iceberg writer are not affected.
259
258
-**iceberg-java before 1.12: a NULL struct whose first field is an identity partition source.**
260
259
iceberg-java reads the struct as a struct of NULLs, and Comet returns NULL, as iceberg-java 1.12
261
-
does ([apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320)).
260
+
does. [apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320) is titled for nested
261
+
initial defaults, but it fixes this case too: it keeps one file column in the read, so the
262
+
struct's definition level survives when every projected field is a constant, and an identity
263
+
partition value is such a constant.
262
264
-**Field-id reads with a read schema that swaps two nested field names.** Spark's vectorized
263
265
reader pairs the file's columns with the read fields by the file's names, so it fails with a type
264
266
mismatch or returns no rows. Comet resolves the fields by id.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/iceberg-writes.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -495,11 +495,11 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed
495
495
- On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns
496
496
nested under a nullable struct count rows whose parent struct is null (they come from the
497
497
parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the
498
-
correct ones. iceberg-java's leave out the rows where the struct is null, so when every
499
-
non-null struct in a file holds a value, iceberg-java records a null count of 0 and `IS NULL`
500
-
prunes the file even though its null-struct rows match
498
+
correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null
499
+
struct in a file holds a value, iceberg-java records a null count of 0, and `IS NULL` prunes a
500
+
file that contains matching rows
501
501
([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts
502
-
never prune that file. `IS NOT NULL` pruning decisions are the same for both.
502
+
never prune such a file. `IS NOT NULL` pruning is the same for both.
503
503
504
504
All content not listed above — the logical data, statistics values, and manifest metadata —
505
505
must match iceberg-java exactly, or the write falls back. The open bugs in which a native write
0 commit comments