Skip to content

Commit 844e8a7

Browse files
committed
docs: address review feedback on the known issues page
Drop #6264 and #6837, which are fixed on main. Scope the datetime rebase entry to the native Parquet scan and say the Delta scan rebases. Add the ignored Parquet writer properties from #5304 and note that the row group failure did not reproduce. Explain how apache/iceberg#17320 covers the identity partition source case. Reword several entries, break the list of Spark SQL tests into one bullet each, and note that spark.comet.scan.enabled is a testing setting.
1 parent 9293815 commit 844e8a7

3 files changed

Lines changed: 70 additions & 66 deletions

File tree

‎docs/source/contributor-guide/data-source-audits/iceberg.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,9 @@
4646
holds a map, in more than 20 probes, CTAS included.
4747
- **Not Comet bugs:** on a table identity-partitioned by a struct field, iceberg-java before 1.12
4848
reads a NULL struct as a struct of NULLs when the partition source is the struct's first field.
49-
[apache/iceberg#17320] fixed it in 1.12.0. The native scan returns NULL, as 1.12 does.
49+
[apache/iceberg#17320] fixed it in 1.12.0. Its title names nested initial defaults, but the
50+
change also keeps a file column in the read when every projected field of a struct is a constant,
51+
and an identity partition value is such a constant. The native scan returns NULL, as 1.12 does.
5052
- **Not covered:** Variant ([#5438]), encrypted tables, and Iceberg versions other than 1.11.
5153

5254
## Native Iceberg writer

‎docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md‎

Lines changed: 63 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@
1919

2020
# Known Issues in Parquet and Iceberg Reads and Writes
2121

22-
This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to
23-
Spark does not cover, with a link to the issue that tracks each one:
22+
This page lists open issues in Comet's Parquet and Iceberg reads and writes where Comet does not
23+
fall back to Spark. Each entry links to its tracking issue. The entries are in two sections:
2424

2525
- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where
2626
Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written
@@ -40,40 +40,44 @@ Which entries apply depends on which native paths are enabled:
4040
| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` |
4141
| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) |
4242

43+
`spark.comet.scan.enabled` is a testing setting that Comet's own test suites use. It is not meant
44+
for production, so do not turn it off to avoid an entry on this page.
45+
4346
## Results that differ from Spark
4447

4548
### Parquet reads
4649

47-
- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and
48-
timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0
49-
wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet
50-
reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both
51-
projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and
52-
the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase
53-
([#5010](https://github.com/apache/datafusion-comet/issues/5010)).
50+
- **Dates and timestamps written with the legacy calendar.** Spark before 3.0, and later versions
51+
with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`, write these values in the hybrid
52+
Julian/Gregorian calendar. The native Parquet scan reads them as proleptic Gregorian, so values
53+
before October 15, 1582 are wrong in projections and predicates. It ignores
54+
`spark.sql.parquet.datetimeRebaseModeInRead` and the file's `org.apache.spark.legacyDateTime`
55+
metadata ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). The
56+
[Delta scan](../delta.md) rebases.
5457
- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262
5558
read back wrong inside lists written in a legacy layout, as Hive and
5659
`spark.sql.parquet.writeLegacyFormat=true` write them
5760
([#7029](https://github.com/apache/datafusion-comet/issues/7029)).
5861
- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the
59-
column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only
60-
behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with
62+
column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5
63+
and 4.0, and to Spark 4.1 and later with
6164
`spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true`
6265
([#7030](https://github.com/apache/datafusion-comet/issues/7030)).
63-
- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without
64-
the UTF8 annotation is read as STRING, because the read schema says `string` or because of
65-
`spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not
66-
valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation
66+
- **Nested BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY
67+
column without the UTF8 annotation that is read as `string`, through the read schema or
68+
`spark.sql.parquet.binaryAsString=true`. Spark returns the bytes. Comet returns NULL for each
69+
such nested value. Top-level values match Spark in a release build, because the invalid bytes
70+
reach an Arrow string array unchecked. A debug build panics on them
6771
([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of
6872
[#4764](https://github.com/apache/datafusion-comet/issues/4764)).
6973
- **Spark 3.4: a struct matched by field id under a different name.** With
7074
`spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to
7175
the file by field id but under another name reads as NULL when Spark prunes its nested fields
7276
([#7031](https://github.com/apache/datafusion-comet/issues/7031)).
73-
- **Field-id reads in which one file column matches a field by id and a sibling by name.** With
77+
- **Field-id reads where one file column matches a field by id and a sibling by name.** With
7478
`spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top
75-
level, and the values of the field matched by id inside a struct. Spark's own readers disagree
76-
with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)).
79+
level. Inside a struct, it returns the values of the field matched by id. Spark's own readers
80+
disagree here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)).
7781
- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and
7882
in every column of a scan that has data filters, Comet returns NULL where Spark fails with
7983
`ArithmeticException: long overflow`
@@ -85,20 +89,15 @@ Which entries apply depends on which native paths are enabled:
8589
`INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and
8690
Variant-annotated fields read as a plain struct
8791
([#5741](https://github.com/apache/datafusion-comet/issues/5741)).
88-
- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic
89-
partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of
90-
the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as
91-
with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan
92-
as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)).
9392

9493
### Iceberg reads
9594

96-
- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are
97-
read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files`
98-
procedures set, the native scan returns NULL for nested fields (struct fields, list elements and
99-
map values) whenever the query also reads a top-level primitive column, so filters on them select
100-
the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes
101-
with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)).
95+
- **Migrated tables that use a name mapping.** The `migrate`, `snapshot` and `add_files`
96+
procedures set `schema.name-mapping.default` for files without Iceberg field ids. When a query
97+
also reads a top-level primitive column, the native scan returns NULL for nested fields (struct
98+
fields, list elements and map values). Filters on those fields then select the wrong rows
99+
([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix needs an iceberg-rust
100+
upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)).
102101
- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such
103102
a file as NULL, and the native scan returns the values stored in the file
104103
([#6794](https://github.com/apache/datafusion-comet/issues/6794)).
@@ -129,37 +128,39 @@ The native Parquet writer is experimental and disabled by default. With
129128
comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ`
130129
columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048),
131130
[#3425](https://github.com/apache/datafusion-comet/issues/3425)).
132-
- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the
133-
write (file index refresh, cache refresh, partition refresh and statistics update) are skipped,
134-
so later queries can read the table's previous contents until `REFRESH TABLE`
131+
- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** Comet skips the steps Spark runs
132+
after a write: file index refresh, cache refresh, partition refresh and statistics update. Later
133+
queries can return the table's previous contents until `REFRESH TABLE`
135134
([#6315](https://github.com/apache/datafusion-comet/issues/6315)).
135+
- **Spark's Parquet writer properties are ignored.** `parquet.block.size`, page size, dictionary,
136+
writer version and statistics settings never reach the native writer
137+
([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
136138
- **An input with no partitions writes no file.** Spark writes one file that holds the schema
137139
([#5303](https://github.com/apache/datafusion-comet/issues/5303)).
138-
- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema
139-
([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR`
140-
([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union
141-
([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default
142-
columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after
143-
`ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)),
144-
`INSERT INTO` from non-projection queries
145-
([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output
146-
schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts
147-
([#3430](https://github.com/apache/datafusion-comet/issues/3430)),
148-
`INSERT OVERWRITE LOCAL DIRECTORY`
149-
([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files
150-
([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit`
151-
([#3428](https://github.com/apache/datafusion-comet/issues/3428)).
140+
- **Spark SQL tests that still fail with the native writer**, not yet classified:
141+
- CTAS schema ([#3418](https://github.com/apache/datafusion-comet/issues/3418))
142+
- CTAS with `CHAR_AS_VARCHAR` ([#3419](https://github.com/apache/datafusion-comet/issues/3419))
143+
- CTAS with a union ([#3429](https://github.com/apache/datafusion-comet/issues/3429))
144+
- `INSERT INTO` with default columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421))
145+
- `INSERT INTO` after `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422))
146+
- `INSERT INTO` from non-projection queries ([#3423](https://github.com/apache/datafusion-comet/issues/3423))
147+
- the `INSERT OVERWRITE` output schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424))
148+
- multiple inserts ([#3430](https://github.com/apache/datafusion-comet/issues/3430))
149+
- `INSERT OVERWRITE LOCAL DIRECTORY` ([#3420](https://github.com/apache/datafusion-comet/issues/3420))
150+
- empty files ([#3417](https://github.com/apache/datafusion-comet/issues/3417))
151+
- `onTaskCommit` ([#3428](https://github.com/apache/datafusion-comet/issues/3428))
152152

153153
### Iceberg writes
154154

155155
The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`:
156156

157-
- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can
158-
read unless Hadoop's `BrotliCodec` is on its classpath, where iceberg-java fails the write
159-
([#6960](https://github.com/apache/datafusion-comet/issues/6960)).
160-
- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed,
161-
where iceberg-java rejects the value. It also trims or lowercases a few other settings that
162-
iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)).
157+
- **Brotli tables.** The native writer commits brotli-compressed files. JVM Parquet readers can
158+
read them only if Hadoop's `BrotliCodec` is on the classpath. iceberg-java fails the write
159+
instead ([#6960](https://github.com/apache/datafusion-comet/issues/6960)).
160+
- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, and
161+
iceberg-java rejects the value. The native writer also trims or lowercases a few other settings
162+
that iceberg-java reads as written
163+
([#7008](https://github.com/apache/datafusion-comet/issues/7008)).
163164

164165
The physical differences that every native data file has, and why they are safe, are listed in
165166
[Accepted divergences](../iceberg-writes.md#accepted-divergences).
@@ -180,7 +181,7 @@ Comet does not fall back to Spark for them.
180181
([#5884](https://github.com/apache/datafusion-comet/issues/5884),
181182
[#5964](https://github.com/apache/datafusion-comet/issues/5964)).
182183
[Parquet Compatibility](scans.md) describes the reads this affects.
183-
- **A duplicate field id in a column the query does not read.** With
184+
- **A duplicate field id in a column the query does not read.** With a valid read schema and
184185
`spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same
185186
field id fails the scan whenever the table schema has a field with that id, even when the query
186187
does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)).
@@ -222,9 +223,6 @@ Spark, and fail where Spark succeeds:
222223
- **Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other
223224
configurations resolve differently from Hadoop's ABFS driver
224225
([#6605](https://github.com/apache/datafusion-comet/issues/6605)).
225-
- **`fs.comet.libhdfs.schemes` set only in the Hadoop configuration.** Comet's planner and its
226-
native scan read the setting from different places, so they can route a scheme differently
227-
([#6837](https://github.com/apache/datafusion-comet/issues/6837)).
228226
- **Iceberg tables whose metadata location has no host.** The native Iceberg scan skips the
229227
credential provider configured with `s3.comet.credential.provider.class`
230228
([#6536](https://github.com/apache/datafusion-comet/issues/6536)).
@@ -233,10 +231,11 @@ Spark, and fail where Spark succeeds:
233231

234232
With `spark.comet.write.parquet.enabled=true`:
235233

236-
- **Row groups that outgrow the memory pool.** The native writer caps row groups by row count
237-
only and can't spill, so with wide or nested rows the row group it is building can outgrow the
238-
off-heap pool and fail the task, where Spark's writer starts a new row group at
239-
`parquet.block.size` ([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
234+
- **Row groups that outgrow the memory pool.** The native writer cuts row groups by row count
235+
only, where Spark uses `parquet.block.size`, and it cannot spill. With wide or nested rows, the
236+
open row group can outgrow the off-heap pool and fail the task. The audit did not reproduce a
237+
failure, because a 239 MB row group fit in a 256 MB pool
238+
([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
240239

241240
### Iceberg writes that fail
242241

@@ -258,7 +257,10 @@ In these cases Comet's result differs from Spark's, but the bug is in Spark or i
258257
native Iceberg writer are not affected.
259258
- **iceberg-java before 1.12: a NULL struct whose first field is an identity partition source.**
260259
iceberg-java reads the struct as a struct of NULLs, and Comet returns NULL, as iceberg-java 1.12
261-
does ([apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320)).
260+
does. [apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320) is titled for nested
261+
initial defaults, but it fixes this case too: it keeps one file column in the read, so the
262+
struct's definition level survives when every projected field is a constant, and an identity
263+
partition value is such a constant.
262264
- **Field-id reads with a read schema that swaps two nested field names.** Spark's vectorized
263265
reader pairs the file's columns with the read fields by the file's names, so it fails with a type
264266
mismatch or returns no rows. Comet resolves the fields by id.

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -495,11 +495,11 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed
495495
- On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns
496496
nested under a nullable struct count rows whose parent struct is null (they come from the
497497
parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the
498-
correct ones. iceberg-java's leave out the rows where the struct is null, so when every
499-
non-null struct in a file holds a value, iceberg-java records a null count of 0 and `IS NULL`
500-
prunes the file even though its null-struct rows match
498+
correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null
499+
struct in a file holds a value, iceberg-java records a null count of 0, and `IS NULL` prunes a
500+
file that contains matching rows
501501
([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts
502-
never prune that file. `IS NOT NULL` pruning decisions are the same for both.
502+
never prune such a file. `IS NOT NULL` pruning is the same for both.
503503

504504
All content not listed above — the logical data, statistics values, and manifest metadata —
505505
must match iceberg-java exactly, or the write falls back. The open bugs in which a native write

0 commit comments

Comments
 (0)