You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 33f4f88
Browse filesBrowse the repository at this point in the historyBrowse files
feat: run global aggregates with no grouping keys or functions natively (apache#6899)
* feat: run global aggregates with no grouping keys or functions natively
Spark plans `Aggregate(Nil, Nil, child)` when nothing reads a global
aggregate's output, as in `df.agg(sum($"a")).count()`. Comet fell back
to Spark for it with "No group by or aggregation".
DataFusion's AggregateStream can't build a batch with no columns
(apache/datafusion#25782), so the native planner plans the shape as a
count(1) aggregate under an empty projection. That needs no DataFusion
change.
Closesapache#6001.
* test: cover a Spark partial aggregate under a native final one
Also write the fixture's multi-file table with an explicit partition count, so it doesn't depend on the harness's parallelism.
* test: say why the fixture's five files become five partitions
* refactor: build the count with the planner's aggregate helper
Also drop test assertions that another assertion already implies, and shorten the serde comment that repeated the native planner's.
* docs: say newer DataFusion rejects an aggregate with no keys or functions
apache/datafusion#25786 makes AggregateExec::try_new reject that shape, so the count plan is how Comet runs it from now on, not a stopgap.
* perf: pass a NULL literal to the throwaway count
The aggregate expands its argument to each input batch's length.
Int64(1) filled an Int64Array per batch only for the projection to drop
the count, while NULL comes back as a cached NullArray with no buffers.
Also shorten the planner comment.
* test: cover HAVING without GROUP BY over an empty global aggregate
The parser plans it as a global aggregate that outputs a literal. Also
note that CollapseProject already gives the SELECT 1 queries a result
projection.
0 commit comments