Repository navigation
Comet writer executes twice #2971
Description
Describe the bug
Comet writes executes twice with AQE
== Physical Plan ==
AdaptiveSparkPlan (8)
+- == Final Plan ==
CometNativeWrite (5)
+- CometNativeWrite (4)
+- ShuffleQueryStage (3), Statistics(sizeInBytes=9.5 KiB, rowCount=1.00E+3)
+- CometExchange (2)
+- CometScan [native_iceberg_compat] parquet (1)
+- == Initial Plan ==
CometNativeWrite (7)
+- CometExchange (6)
+- CometScan [native_iceberg_compat] parquet (1)
Steps to reproduce
Folder output3 doesn't exist
spark.read.parquet("/tmp/writertest/input").repartition(1).write.mode("overwrite").parquet("/tmp/writertest/output3")
Expected behavior
Should write only once
Additional context
No response
Activity