Skip to content

Commit 093c175

Browse files
schenksjclaude
andcommitted
contrib(delta) repros for remaining regression failures (F3/F4/F6)
Confirmed, faithful local reproductions for every still-failing Delta 4.1 own-suite family after the F1/F2/apache#198 fixes. Triage collapsed the ~70 original failures into three remaining root causes (the rest are F1/F2, fixed): * F3 row tracking (dominant): native scan synthesizes row IDs from base_row_id+row_index instead of reading the materialized stable columns (_row-id-col-*/_row-commit-version-col-*), so IDs change across any rewrite. Covers all rowid/RowTracking{Merge,Delete,Compaction, ReadWrite}Suite failures (z-order, auto-compact, MERGE row-id stability, DELETE-with-DV-disabled, optimized writes, materialized/ conflicting columns). Repro: row IDs change across OPTIMIZE (v_0: 1->141). * F4 protobuf recursion: a WHERE of ~101 AND'd predicates builds a deep boolean expression that exceeds protobuf's recursion limit (100) when serialized to the native plan -- InvalidProtocolBufferException "too many levels of nesting". From DataSkippingDeltaTests. * F6 corrupted-file error compatibility (SC-8810): a 0-byte data file makes Comet's native reader throw "Requested range was invalid" instead of Spark's FAILED_READ_FILE. CometDeltaPendingReproSuite holds one minimal repro per family, each asserting the correct behavior and marked `ignore` so CI stays green (3 ignored, 0 failed); un-`ignore` to drive each fix. docs/08-known-limitations updated with B3 triage + new B5 (F4) and B6 (F6) entries. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent a4f2b07 commit 093c175

2 files changed

Lines changed: 204 additions & 18 deletions

File tree

‎contrib/delta/docs/08-known-limitations.md‎

Lines changed: 52 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -178,24 +178,58 @@ remains (A1 / #198).
178178
row-commit-version column, read it from parquet (don't synthesize).
179179
- **Tracking:** internal task #197 (F3).
180180

181-
### B4. Other untriaged failures — PENDING
182-
183-
Not yet root-caused; need triage (some may share a root cause with B2/B3 or be
184-
test-harness/transaction-visibility artifacts):
185-
186-
- MERGE family (not `isPartitioned`-gated): "DELETE MATCHED only MERGE",
187-
"UPDATE MATCHED only MERGE", "DELETE/UPDATE WHEN NOT MATCHED BY SOURCE MERGE",
188-
"UPDATE + DELETE WHEN NOT MATCHED BY SOURCE MERGE", "{DELETE,UPDATE} only with
189-
source rows matching multiple target rows", "Multiple merges into the same
190-
table", "Source and target referencing to the same table", "Target is accessed
191-
through a view", "MERGE preserves Row Tracking on tables enabled using
192-
backfill", "schema evolution, extra nested column in source - update".
193-
- DELETE with persistent DVs disabled (isPartitioned true/false).
194-
- Optimized writes (partitioned / unpartitioned / disabled).
195-
- Data skipping: "data skipping shouldn't use expressions involving a subquery",
196-
"remove redundant stats column references in data skipping expression"
197-
(and "old behavior with DataFrame schema" variants).
198-
- "SC-8810: skipping deleted file still throws on corrupted file".
181+
Repro: `CometDeltaPendingReproSuite` ("F3: row IDs are stable across
182+
OPTIMIZE"), `ignore`d until fixed.
183+
184+
### B4. Triage of remaining failures
185+
186+
The originally-"untriaged" failures resolve into:
187+
188+
- **Mostly B3 (row tracking).** "DELETE/UPDATE MATCHED only MERGE", "... WHEN NOT
189+
MATCHED BY SOURCE", "{DELETE,UPDATE} only with source rows matching multiple
190+
target rows", "Multiple merges into the same table", "Source and target
191+
referencing to the same table", "Target is accessed through a view", "MERGE
192+
preserves Row Tracking on tables enabled using backfill", "Optimized writes
193+
(partitioned/unpartitioned/disabled)", "DELETE with persistent DVs disabled"
194+
all live in `rowid/RowTracking{Merge,Delete,...}Suite` and fail the same way:
195+
a stable row ID changes across a rewrite (e.g. "Row ID has change for row with
196+
stored_id = 412"). Same root cause as B3 — the native scan synthesizes row IDs
197+
instead of reading the materialized columns. Fixing B3/F3 should clear these.
198+
- **F1 (already fixed).** "data skipping shouldn't use expressions involving a
199+
subquery" is the same `CometSubqueryAdaptiveBroadcastExec` DPP crash as B2 —
200+
fixed by `64cd878a`. Confirm via re-run.
201+
202+
### B5. Deeply-nested data-skipping expression — protobuf recursion limit — PENDING
203+
204+
- **Test:** "remove redundant stats column references in data skipping
205+
expression" (+ "old behavior with DataFrame schema" variant), from
206+
`DataSkippingDeltaTests`.
207+
- **Symptom:** A WHERE with ~101 AND'd predicates produces a very deep boolean
208+
expression; serializing it to Comet's native proto exceeds protobuf's default
209+
recursion limit (100): `InvalidProtocolBufferException: Protocol message had
210+
too many levels of nesting` from `ExprOuterClass$BinaryExpr.mergeFrom`.
211+
- **Fix direction:** raise the proto parse recursion limit
212+
(`CodedInputStream.setRecursionLimit`) where Comet parses the plan/expr, or
213+
rebalance/flatten deep AND/OR chains before serialization.
214+
- **Repro:** `CometDeltaPendingReproSuite` ("F4: deeply-nested data-skipping
215+
filter ..."), `ignore`d until fixed.
216+
217+
### B6. Corrupted-file read error compatibility (SC-8810) — PENDING
218+
219+
- **Test:** "SC-8810: skipping deleted file still throws on corrupted file"
220+
(`DeltaSuite`).
221+
- **Symptom:** With one data file truncated to 0 bytes, vanilla Spark+Delta
222+
throws `[FAILED_READ_FILE.NO_HINT]`; Comet's native reader throws
223+
`CometNativeException: External: Generic LocalFileSystem error: Requested range
224+
was invalid` instead, so the test's message assertion fails.
225+
- **Fix direction:** map the native corrupted/short-file error to Spark's
226+
`FAILED_READ_FILE` (or decline native and let Spark's reader produce it).
227+
- **Repro:** `CometDeltaPendingReproSuite` ("F6: reading a corrupted file ..."),
228+
`ignore`d until fixed.
229+
230+
All pending-failure repros live in `CometDeltaPendingReproSuite` (marked
231+
`ignore`; un-`ignore` to drive a fix). They are confirmed failing as of this
232+
writing.
199233

200234
---
201235

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.comet.contrib.delta
21+
22+
import org.apache.spark.sql.functions._
23+
24+
// Local reproductions for the Delta 4.1 own-suite regression families that are
25+
// not yet fixed on this branch. One faithful, minimal repro per root cause so
26+
// each can be diagnosed and fixed (and kept as a regression guard afterward).
27+
//
28+
// Status when written (after F1 DPP / F2 time-travel / #198 fixes landed):
29+
// F3 (row tracking) -- PENDING; dominant family (all RowTracking* suites)
30+
// F4 (protobuf recursion)-- PENDING
31+
// F6 (corrupted file) -- PENDING
32+
//
33+
// Each test asserts the EXPECTED (correct) behavior and is marked `ignore` so
34+
// CI stays green while the bug is open. It is a confirmed repro: un-`ignore` it
35+
// (change `ignore(` -> `test(`) to watch it fail today and pass once the
36+
// corresponding fix lands. Verified failing at the time of writing (see commit
37+
// message / docs/08-known-limitations.md).
38+
class CometDeltaPendingReproSuite extends CometDeltaTestBase {
39+
40+
// === F3: row tracking -- materialized stable row IDs ======================
41+
//
42+
// Mirrors RowTracking{Merge,Compaction,ReadWrite}Suite. When a file is
43+
// rewritten (OPTIMIZE / z-order / compaction / MERGE), Delta preserves stable
44+
// row IDs by MATERIALIZING them into real parquet columns
45+
// `_row-id-col-<uuid>` / `_row-commit-version-col-<uuid>`. The Spark plan then
46+
// reads `coalesce(_metadata.row_id, base_row_id + row_index)`. The native scan
47+
// classifies those names as synthetic (`isExtraSyntheticName`) and synthesizes
48+
// from `base_row_id + row_index` instead of reading the persisted values --
49+
// so after a rewrite the row IDs CHANGE (new file => new base+index) instead
50+
// of staying stable. Expected: row IDs are stable across OPTIMIZE.
51+
52+
ignore("F3: row IDs are stable across OPTIMIZE (materialized row-id columns)") {
53+
assume(deltaSparkAvailable, "delta-spark not on the test classpath; skipping")
54+
withDeltaTable("f3_rowid_optimize") { tablePath =>
55+
val ss = spark
56+
import ss.implicits._
57+
// Multiple files so OPTIMIZE actually compacts/rewrites.
58+
(0 until 100)
59+
.map(i => (i.toLong, s"v_$i"))
60+
.toDF("id", "v")
61+
.repartition(5)
62+
.write
63+
.format("delta")
64+
.option("delta.enableRowTracking", "true")
65+
.option("delta.minReaderVersion", "3")
66+
.option("delta.minWriterVersion", "7")
67+
.save(tablePath)
68+
69+
def rowIdsByValue(): Map[String, Long] =
70+
spark.read.format("delta").load(tablePath)
71+
.select(col("v"), col("_metadata.row_id").as("rid"))
72+
.collect()
73+
.map(r => r.getString(0) -> r.getLong(1))
74+
.toMap
75+
76+
val before = rowIdsByValue()
77+
// OPTIMIZE compacts the 5 files into 1, rewriting + materializing row IDs.
78+
spark.sql(s"OPTIMIZE delta.`$tablePath`")
79+
val after = rowIdsByValue()
80+
81+
assert(before.nonEmpty && before.size == 100, s"setup: ${before.size} rows")
82+
val changed = before.keys.filter(k => before(k) != after.getOrElse(k, -1L)).toSeq.sorted
83+
assert(
84+
changed.isEmpty,
85+
s"row IDs must be stable across OPTIMIZE; changed for ${changed.size} rows, " +
86+
s"e.g. ${changed.take(3).map(k => s"$k: ${before(k)} -> ${after.get(k)}").mkString(", ")}")
87+
}
88+
}
89+
90+
// === F4: deeply-nested data-skipping expression -> protobuf recursion =====
91+
//
92+
// Mirrors DataSkippingDeltaTests "remove redundant stats column references in
93+
// data skipping expression". A WHERE with ~101 AND'd conditions builds a very
94+
// deep boolean expression; serializing it to Comet's native proto exceeds
95+
// protobuf's default recursion limit (100), throwing
96+
// "Protocol message had too many levels of nesting" from BinaryExpr.mergeFrom.
97+
// Expected: the query runs.
98+
99+
ignore("F4: deeply-nested data-skipping filter does not overflow protobuf nesting") {
100+
assume(deltaSparkAvailable, "delta-spark not on the test classpath; skipping")
101+
val tbl = "f4_deep_filter"
102+
withTable(tbl) {
103+
val colNames = (0 to 100).map(i => s"col_$i")
104+
spark.sql(
105+
s"CREATE TABLE $tbl (${colNames.map(_ + " INT").mkString(", ")}) USING delta")
106+
spark.sql(
107+
s"INSERT INTO $tbl VALUES (${colNames.map(_ => "0").mkString(", ")})")
108+
val whereClause = colNames.map(c => s"$c != 1").mkString(" AND ")
109+
// Must not throw a protobuf recursion error.
110+
val rows = spark.sql(s"SELECT col_0 FROM $tbl WHERE $whereClause").collect()
111+
assert(rows.length == 1, s"expected the single all-zero row, got ${rows.length}")
112+
}
113+
}
114+
115+
// === F6: corrupted / empty file (SC-8810) =================================
116+
//
117+
// Mirrors DeltaSuite "SC-8810: skipping deleted file still throws on corrupted
118+
// file". With one data file truncated to 0 bytes, vanilla Spark+Delta throws a
119+
// `[FAILED_READ_FILE.NO_HINT]` SparkException. Comet's native reader instead
120+
// throws `CometNativeException: ... Requested range was invalid`. Expected:
121+
// the error is the Spark-compatible one (so user-facing error handling and the
122+
// Delta test pass). Repro asserts the message contains the Spark marker.
123+
124+
ignore("F6: reading a corrupted file surfaces a Spark-compatible error (SC-8810)") {
125+
assume(deltaSparkAvailable, "delta-spark not on the test classpath; skipping")
126+
withDeltaTable("f6_corrupt") { tablePath =>
127+
val ss = spark
128+
import ss.implicits._
129+
Seq(1).toDF().write.format("delta").mode("append").save(tablePath)
130+
Seq(2, 2).toDF().write.format("delta").mode("append").save(tablePath)
131+
Seq(4).toDF().write.format("delta").mode("append").save(tablePath)
132+
133+
// Truncate one data file to 0 bytes to simulate corruption.
134+
val dir = new java.io.File(tablePath)
135+
val parquet = dir.listFiles()
136+
.filter(f => !f.getName.startsWith("_") && f.getName.endsWith(".parquet"))
137+
.sortBy(_.getName)
138+
.head
139+
val ch = new java.io.FileOutputStream(parquet)
140+
try ch.getChannel.truncate(0) finally ch.close()
141+
142+
val ex = intercept[Exception] {
143+
spark.read.format("delta").load(tablePath).collect()
144+
}
145+
val msg = Option(ex.getMessage).getOrElse("") +
146+
Option(ex.getCause).map(c => Option(c.getMessage).getOrElse("")).getOrElse("")
147+
assert(
148+
msg.contains("FAILED_READ_FILE"),
149+
s"expected a Spark-compatible FAILED_READ_FILE error, got: $msg")
150+
}
151+
}
152+
}

0 commit comments

Comments
 (0)