Skip to content

CometInMemoryTableScan prints its whole cached plan inline in EXPLAIN #6452

Closed

Description

@andygrove

Describe the bug

CometInMemoryTableScanExec prints every one of its constructor fields in its plan string, and one of them is the CachedRDDBuilder, whose toString holds the whole cached plan, physical and logical. So every plan that reads a relation cached in Comet's format carries that plan inline in the scan's line, with its raw newlines, which breaks the tree. It happens in EXPLAIN and in the Arguments: line of EXPLAIN FORMATTED. The line also prints the serializer's object identity (ArrowCachedBatchSerializer@62857555) and both attribute lists.

Other Comet operators override stringArgs to leave out their originalPlan. This one does not.

Steps to reproduce

On main at 8e4bded, with spark.comet.exec.inMemoryCache.enabled=true:

spark.range(0, 1000, 1, 2).selectExpr("id", "id % 10 AS k").createOrReplaceTempView("named_t")
spark.catalog.cacheTable("named_t")
spark.sql("SELECT k, count(*) FROM named_t GROUP BY k").explain()
+- CometInMemoryTableScan Scan In-memory table named_t [k#1L], org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer@62857555, CachedRDDBuilder(org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer@62857555,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [id#0L, (id#0L % 10) AS k#1L]
+- *(1) Range (0, 1000, step=1, splits=2)
,Some(named_t),SubqueryAlias named_t
+- View (`named_t`, [id#0L, k#1L])
   +- Project [id#0L, (id#0L % cast(10 as bigint)) AS k#1L]
      +- Range (0, 1000, step=1, splits=Some(2))
), [id#0L, k#1L], [k#1L]

Expected behavior

The scan is described the way Spark's own cache scan is, for example CometInMemoryTableScan Scan In-memory table named_t [k#1L], followed by any pruning predicates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmeticrequires-triage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions