You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native CartesianProductExec: multi-partition inputs and spilling #7476
#6938 runs CartesianProductExec natively behind spark.comet.exec.cartesianProduct.enabled (default false), only without a join condition and when each input reports exactly one partition. Remaining work:
Multi-partition inputs. Spark crosses every partition pair through CartesianRDD. A non-bucketed Parquet scan reports UnknownPartitioning(0), so today the native path needs a COALESCE(1) or similar.
Output batching. One output batch per left row for each right batch, which explains the 3.1× large-left benchmark result against 18.6× for the square case.
Row order. Native output is right-batch-major, Spark's is left-row-major. Documented, since Spark reports no output ordering.
Activity