Skip to content

Commit 14f0f59

Browse files
authored
test: report which writer ran each Iceberg write in the Iceberg CI jobs (apache#6155)
* test: report which writer ran each Iceberg write in the Iceberg CI jobs The Iceberg Spark test jobs run with the native Iceberg writer enabled, but a passing job cannot tell a native write from a silent fallback: the fallback warnings never reach the job log and no upstream test asserts which writer ran. Add a test-only config, spark.comet.testing.icebergWriteReport.dir, backed by the COMET_ICEBERG_WRITE_REPORT_DIR environment variable. When it is set, the driver plugin registers IcebergWriteReportListener, which appends one JSON line per Iceberg write: native, JVM under the split operator (with Comet's fallback reasons), or Spark's own V2 write. The core and extensions jobs set the variable, so the Iceberg diffs are unchanged, and dev/ci/summarize-iceberg-writes.py renders a per-job and all-shards table on the job summary page. dev/local-ci.sh prints the same summary. Closes apache#6148. * test: report streaming and Spark 3.4 CTAS/RTAS Iceberg writes, and count only each shard's latest attempt The write report missed two kinds of Iceberg write that Spark runs itself: - A streaming micro-batch runs as `WriteToDataSourceV2Exec` over a `MicroBatchWrite`, not as a `V2ExistingTableWriteExec`. - On Spark 3.4, CTAS and RTAS write the table from the create or replace exec through `TableWriteExecHelper.writeWithV2`. Spark 3.5+ runs that write as a nested append or overwrite, which the listener already sees, so `IcebergTableAsSelectShim` recognizes the execs on 3.4 only. Both went unreported, so the Spark count and the total left them out and the native share read high. The summarizer took each shard's latest attempt from its report files, so a retry that recorded no writes was hidden behind the earlier attempt's records. It now takes the latest attempt from the shard artifact directories and names a shard whose latest attempt recorded nothing. `dev/ci/test-summarize-iceberg-writes.py` covers that and runs in Preflight.
1 parent 81f2574 commit 14f0f59

14 files changed

Lines changed: 784 additions & 22 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,9 @@ jobs:
186186
- name: Check Iceberg shard inventory validation
187187
run: python3 dev/ci/test-iceberg-shards.py
188188

189+
- name: Check Iceberg write report summary
190+
run: python3 dev/ci/test-summarize-iceberg-writes.py
191+
189192
- name: Check CI config invariants
190193
run: python3 dev/ci/check-ci-config.py
191194

‎.github/workflows/iceberg_spark_test_reusable.yml‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -154,12 +154,20 @@ jobs:
154154
run: |
155155
cd apache-iceberg
156156
rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups
157-
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
157+
# COMET_ICEBERG_WRITE_REPORT_DIR records which writer ran each Iceberg
158+
# write; see dev/ci/summarize-iceberg-writes.py.
159+
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \
160+
./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
158161
:iceberg-spark:iceberg-spark-${{ inputs.spark-short }}_${{ inputs.scala }}:test \
159162
--init-script ../dev/ci/iceberg-test-shards.gradle \
160163
-PcometShardTask=:iceberg-spark:iceberg-spark-${{ inputs.spark-short }}_${{ inputs.scala }}:test \
161164
-PcometShardIndex=${{ matrix.shard }} -PcometShardCount=${{ needs.build-native.outputs.shard-count }} \
162165
-Pquick=true -x javadoc
166+
- name: Summarize Iceberg writes
167+
if: ${{ !cancelled() }}
168+
run: |
169+
python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark shard ${{ matrix.shard }}" \
170+
apache-iceberg/build/comet-iceberg-writes
163171
- name: Upload Iceberg shard inventory and test reports
164172
if: ${{ !cancelled() }}
165173
# iceberg-spark-shard-coverage downloads the inventory, so a flaky
@@ -170,6 +178,7 @@ jobs:
170178
path: |
171179
apache-iceberg/**/build/comet-shards/*.json
172180
apache-iceberg/**/build/test-results/test/*.xml
181+
apache-iceberg/build/comet-iceberg-writes/*.jsonl
173182
retention-days: 7
174183

175184
iceberg-spark-shard-coverage:
@@ -190,6 +199,11 @@ jobs:
190199
run: |
191200
python3 dev/ci/check-iceberg-shards.py --manifests iceberg-shard-reports \
192201
--task :iceberg-spark:iceberg-spark-${{ inputs.spark-short }}_${{ inputs.scala }}:test
202+
- name: Summarize Iceberg writes across shards
203+
if: ${{ !cancelled() }}
204+
run: |
205+
python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark, all shards" \
206+
iceberg-shard-reports
193207
194208
iceberg-spark-extensions:
195209
needs: build-native
@@ -222,9 +236,23 @@ jobs:
222236
run: |
223237
cd apache-iceberg
224238
rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups
225-
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
239+
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \
240+
./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
226241
:iceberg-spark:iceberg-spark-extensions-${{ inputs.spark-short }}_${{ inputs.scala }}:test \
227242
-Pquick=true -x javadoc
243+
- name: Summarize Iceberg writes
244+
if: ${{ !cancelled() }}
245+
run: |
246+
python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark-extensions" \
247+
apache-iceberg/build/comet-iceberg-writes
248+
- name: Upload Iceberg write report
249+
if: ${{ !cancelled() }}
250+
uses: ./.github/actions/upload-artifact-retry
251+
with:
252+
name: iceberg-spark-extensions-writes-${{ inputs.iceberg-full }}-spark-${{ inputs.spark-full }}-scala-${{ inputs.scala }}-jdk${{ inputs.java }}-attempt-${{ github.run_attempt }}
253+
path: apache-iceberg/build/comet-iceberg-writes/*.jsonl
254+
if-no-files-found: ignore
255+
retention-days: 7
228256

229257
iceberg-spark-runtime:
230258
needs: build-native

‎dev/ci/summarize-iceberg-writes.py‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
"""Summarize which writer ran the Iceberg writes of an Iceberg Spark test run.
19+
20+
The Iceberg Spark test jobs set COMET_ICEBERG_WRITE_REPORT_DIR, so Comet's
21+
IcebergWriteReportListener appends one JSON line per Iceberg write to a file
22+
in that directory. This prints how many writes ran on Comet's native writer,
23+
how many Comet's split operator left on Iceberg's JVM writer and why, and how
24+
many Spark planned without Comet's split operator:
25+
26+
python3 dev/ci/summarize-iceberg-writes.py --title "iceberg-spark shard 1" DIR...
27+
28+
Files under a directory named like a shard artifact (...-shard-N-attempt-M)
29+
are counted only for the latest attempt of each shard, so a rerun of failed
30+
jobs does not count a shard twice. The latest attempt is the newest such
31+
directory, whether or not it holds any report files, so a rerun that recorded
32+
no writes is reported as missing rather than replaced by an earlier attempt.
33+
The summary is also appended to $GITHUB_STEP_SUMMARY when that is set. It
34+
never fails the job.
35+
"""
36+
37+
import argparse
38+
from collections import Counter
39+
import json
40+
import os
41+
from pathlib import Path
42+
import re
43+
44+
45+
SHARD_ATTEMPT = re.compile(r"-shard-(\d+)-attempt-(\d+)$")
46+
WRITERS = [
47+
("native", "Comet native writer"),
48+
("jvm", "Iceberg JVM writer under Comet's split operator"),
49+
("spark", "Spark V2 write, not planned by Comet's split operator"),
50+
]
51+
TOP_REASONS = 20
52+
53+
54+
def shard_attempt(path):
55+
"""The (shard, attempt) of the shard artifact directory holding path, or None."""
56+
for part in path.parts:
57+
match = SHARD_ATTEMPT.search(part)
58+
if match:
59+
return int(match.group(1)), int(match.group(2))
60+
return None
61+
62+
63+
def report_files(roots):
64+
"""Every report file under roots, keeping only the latest attempt of each shard.
65+
66+
Returns the files and the (shard, attempt) pairs whose latest attempt holds no report file.
67+
The latest attempt comes from the artifact directories rather than the report files, because
68+
an attempt that recorded no writes still uploads its shard inventory and test reports.
69+
"""
70+
latest = {}
71+
files = []
72+
for root in map(Path, roots):
73+
for path in [root, *sorted(root.rglob("*"))]:
74+
key = shard_attempt(path)
75+
if key:
76+
latest[key[0]] = max(latest.get(key[0], 0), key[1])
77+
if path.suffix == ".jsonl" and path.is_file():
78+
files.append((path, key))
79+
kept = [(path, key) for path, key in files if key is None or latest[key[0]] == key[1]]
80+
reported = {key for _, key in kept}
81+
missing = sorted(key for key in latest.items() if key not in reported)
82+
return [path for path, _ in kept], missing
83+
84+
85+
def load(roots):
86+
files, missing = report_files(roots)
87+
writes = []
88+
for path in files:
89+
for line in path.read_text(encoding="utf-8").splitlines():
90+
if line.strip():
91+
writes.append(json.loads(line))
92+
return writes, missing
93+
94+
95+
def cell(text):
96+
return " ".join(text.split()).replace("|", "\\|")
97+
98+
99+
def summarize(title, writes, missing=()):
100+
lines = [f"### Iceberg writes: {title}", ""]
101+
for shard, attempt in missing:
102+
lines += [
103+
f"Shard {shard} recorded no Iceberg writes in its latest attempt ({attempt}), "
104+
"so none of its writes are counted below.",
105+
"",
106+
]
107+
if not writes:
108+
lines.append(
109+
"No Iceberg writes were recorded. Either the target ran none or "
110+
"COMET_ICEBERG_WRITE_REPORT_DIR did not reach the test JVMs."
111+
)
112+
return "\n".join(lines) + "\n"
113+
114+
total = len(writes)
115+
by_writer = Counter(w["writer"] for w in writes)
116+
lines += ["| Writer | Writes | Share |", "| --- | ---: | ---: |"]
117+
for key, label in WRITERS:
118+
count = by_writer.get(key, 0)
119+
lines.append(f"| {label} | {count} | {100.0 * count / total:.1f}% |")
120+
lines += [f"| Total | {total} | |", ""]
121+
failed = sum(1 for w in writes if w.get("failed"))
122+
if failed:
123+
lines += [f"{failed} of the {total} writes ran in queries that failed.", ""]
124+
125+
reasons = Counter()
126+
for w in writes:
127+
if w["writer"] == "jvm":
128+
for reason in w.get("reasons") or ["(no reason recorded)"]:
129+
reasons[reason] += 1
130+
if reasons:
131+
lines += [
132+
"#### Why the split operator kept the JVM writer",
133+
"",
134+
"A write can have several reasons, so the counts can add up to more than the "
135+
"JVM writes.",
136+
"",
137+
"| Writes | Reason |",
138+
"| ---: | --- |",
139+
]
140+
for reason, count in reasons.most_common(TOP_REASONS):
141+
lines.append(f"| {count} | {cell(reason)} |")
142+
if len(reasons) > TOP_REASONS:
143+
lines.append(f"| | and {len(reasons) - TOP_REASONS} more reasons |")
144+
lines.append("")
145+
146+
operators = Counter(w["node"] for w in writes if w["writer"] == "spark")
147+
if operators:
148+
lines += ["#### Spark V2 writes by operator", "", "| Writes | Operator |", "| ---: | --- |"]
149+
for node, count in operators.most_common():
150+
lines.append(f"| {count} | {cell(node)} |")
151+
lines.append("")
152+
153+
return "\n".join(lines) + "\n"
154+
155+
156+
def main():
157+
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
158+
parser.add_argument("--title", required=True, help="heading for the summary")
159+
parser.add_argument("roots", nargs="+", help="directories holding the report files")
160+
args = parser.parse_args()
161+
162+
summary = summarize(args.title, *load(args.roots))
163+
print(summary)
164+
step_summary = os.environ.get("GITHUB_STEP_SUMMARY")
165+
if step_summary:
166+
with open(step_summary, "a", encoding="utf-8") as out:
167+
out.write(summary + "\n")
168+
169+
170+
if __name__ == "__main__":
171+
main()
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#!/usr/bin/env python3
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
"""Fast regression tests for the Iceberg write report summary."""
21+
22+
import importlib.util
23+
import json
24+
from pathlib import Path
25+
import tempfile
26+
import unittest
27+
28+
29+
SPEC = importlib.util.spec_from_file_location(
30+
"summarize_iceberg_writes", Path(__file__).with_name("summarize-iceberg-writes.py"))
31+
SUMMARIZE = importlib.util.module_from_spec(SPEC)
32+
SPEC.loader.exec_module(SUMMARIZE)
33+
34+
ARTIFACT = "iceberg-spark-1.11.0-spark-4.1.3-scala-2.13-jdk17-shard-{}-attempt-{}"
35+
36+
37+
def records(*writers):
38+
return "".join(
39+
json.dumps({"writer": w, "node": "AppendData", "reasons": [], "failed": False}) + "\n"
40+
for w in writers)
41+
42+
43+
class SummarizeIcebergWritesTest(unittest.TestCase):
44+
def setUp(self):
45+
self.temp = tempfile.TemporaryDirectory(prefix="comet-iceberg-writes-test-")
46+
self.addCleanup(self.temp.cleanup)
47+
self.root = Path(self.temp.name)
48+
49+
def attempt(self, shard, attempt, *writers):
50+
"""A shard attempt's artifact as the coverage job downloads it.
51+
52+
Every attempt uploads its test reports. Only an attempt that recorded writes has a
53+
report file.
54+
"""
55+
artifact = self.root / ARTIFACT.format(shard, attempt)
56+
reports = artifact / "build/test-results/test"
57+
reports.mkdir(parents=True)
58+
(reports / "TEST-org.example.TestFixture.xml").write_text("<testsuite/>")
59+
if writers:
60+
writes = artifact / "build/comet-iceberg-writes"
61+
writes.mkdir(parents=True)
62+
(writes / "iceberg-writes-fixture.jsonl").write_text(records(*writers))
63+
64+
def load(self):
65+
writes, missing = SUMMARIZE.load([self.root])
66+
return sorted(w["writer"] for w in writes), missing
67+
68+
def test_every_shard_counts_once(self):
69+
self.attempt(1, 1, "native")
70+
self.attempt(2, 1, "jvm", "spark")
71+
self.assertEqual(self.load(), (["jvm", "native", "spark"], []))
72+
73+
def test_latest_attempt_replaces_an_earlier_one(self):
74+
self.attempt(1, 1, "native", "native")
75+
self.attempt(1, 2, "jvm")
76+
self.assertEqual(self.load(), (["jvm"], []))
77+
78+
def test_retry_without_writes_is_reported_missing_instead_of_stale(self):
79+
self.attempt(1, 1, "native")
80+
self.attempt(1, 2)
81+
self.attempt(2, 1, "spark")
82+
self.assertEqual(self.load(), (["spark"], [(1, 2)]))
83+
summary = SUMMARIZE.summarize("fixture", *SUMMARIZE.load([self.root]))
84+
self.assertIn("Shard 1 recorded no Iceberg writes in its latest attempt (2)", summary)
85+
self.assertIn("| Comet native writer | 0 | 0.0% |", summary)
86+
self.assertIn("| Spark V2 write, not planned by Comet's split operator | 1 | 100.0% |",
87+
summary)
88+
89+
def test_root_that_is_itself_a_shard_artifact(self):
90+
self.attempt(1, 3)
91+
writes, missing = SUMMARIZE.load([self.root / ARTIFACT.format(1, 3)])
92+
self.assertEqual((writes, missing), ([], [(1, 3)]))
93+
94+
def test_files_outside_shard_artifacts_always_count(self):
95+
# A shard job and dev/local-ci.sh summarize their own report directory directly.
96+
(self.root / "iceberg-writes-local.jsonl").write_text(records("native", "jvm"))
97+
self.assertEqual(self.load(), (["jvm", "native"], []))
98+
99+
def test_no_writes_at_all(self):
100+
summary = SUMMARIZE.summarize("fixture", *SUMMARIZE.load([self.root / "missing"]))
101+
self.assertIn("No Iceberg writes were recorded", summary)
102+
103+
104+
if __name__ == "__main__":
105+
unittest.main()

‎dev/local-ci.sh‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -389,16 +389,25 @@ run_iceberg() {
389389
gradlew ":iceberg-spark:iceberg-spark-runtime-${SPARK}_${SCALA}:integrationTest"
390390
;;
391391
esac
392+
case "$target" in
393+
shard-* | extensions)
394+
python3 "$REPO/dev/ci/summarize-iceberg-writes.py" --title "$target" \
395+
"$dest/build/comet-iceberg-writes/$target"
396+
;;
397+
esac
392398
ok "$target took $(hms $((SECONDS - started)))"
393399
done
394400
}
395401

396-
# Reads $dest, $spark and $SCALA from run_iceberg.
402+
# Reads $dest, $spark, $SCALA and $target from run_iceberg.
397403
gradlew() {
398404
(
399405
cd "$dest"
400406
# shellcheck disable=SC2031
401407
export SPARK_LOCAL_IP=localhost ENABLE_COMET=true ENABLE_COMET_ONHEAP=true
408+
# One directory per target, emptied first, so a rerun reports only its own writes.
409+
export COMET_ICEBERG_WRITE_REPORT_DIR="$dest/build/comet-iceberg-writes/$target"
410+
rm -rf "$COMET_ICEBERG_WRITE_REPORT_DIR"
402411
./gradlew "-DsparkVersions=$SPARK" "-DscalaVersion=$SCALA" \
403412
-DflinkVersions= -DkafkaVersions= "$@" -Pquick=true -x javadoc
404413
)

0 commit comments

Comments
 (0)