You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CometInMemoryTableScan prints its whole cached plan inline in EXPLAIN #6452
CometInMemoryTableScanExec prints every one of its constructor fields in its plan string, and one of them is the CachedRDDBuilder, whose toString holds the whole cached plan, physical and logical. So every plan that reads a relation cached in Comet's format carries that plan inline in the scan's line, with its raw newlines, which breaks the tree. It happens in EXPLAIN and in the Arguments: line of EXPLAIN FORMATTED. The line also prints the serializer's object identity (ArrowCachedBatchSerializer@62857555) and both attribute lists.
Other Comet operators override stringArgs to leave out their originalPlan. This one does not.
Steps to reproduce
On main at 8e4bded, with spark.comet.exec.inMemoryCache.enabled=true:
spark.range(0, 1000, 1, 2).selectExpr("id", "id % 10 AS k").createOrReplaceTempView("named_t")
spark.catalog.cacheTable("named_t")
spark.sql("SELECT k, count(*) FROM named_t GROUP BY k").explain()
The scan is described the way Spark's own cache scan is, for example CometInMemoryTableScan Scan In-memory table named_t [k#1L], followed by any pruning predicates.
Activity