You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg scan does not report scan metrics or scan time in the Spark UI #6026
When Comet runs an Iceberg scan natively (CometIcebergNativeScanExec), the Spark SQL UI does not
show the scan metrics that plain Spark + Iceberg normally shows on the BatchScan node. All the
Iceberg planning numbers show up as 0:
number of data manifests, scanned/skipped manifests
number of result/skipped data files
total data file size
total planning duration
all the delete-file counts and sizes
The scan also does not report any "scan time", so there is no way to see how long the native read
took.
Vanilla Iceberg shows all of these (they come from Iceberg's Java planning), so users lose
visibility when Comet takes over the scan.
Expected: the native Iceberg scan node in the Spark UI shows the same planning numbers Iceberg-Java
reports, plus a scan time.
Activity