You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d0a9842
Browse filesBrowse the repository at this point in the historyBrowse files
Add skewness metric to nodes that execute in partitioned mode (apache#25977) (#188)
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes#123` indicates that this PR will close issue #123.
-->
Related to apache#23237 and
[feat(metric): Add output skewness metric to detect skewed plans
easier](apache#21211)
- Closes #.
I think it would be useful to add the skewness metric for nodes that
execute in partitioned mdoe -- right now the only node that includes
that metric in the explain analyze is DataSourceExec for parquet.
This is specially useful to check how many partitions were idle during
Aggregations or HashJoins for example.
The PR adds a new metric `output_rows_skew` in the Explain Analyze
output for the following nodes:
- RepartitionExec
- HashJoinExec
- AggregateExec
- BoundedWindowAggExec
- WindowAggExec
Note that for a plan that has a Parquet DataSource the skewness might
not be the same as for the rest of the nodes, for example we might have
the following scenario:
```
DataSourceExec [100, 100, 100, 100] → 0% skewness, data is even
FilterExec [100, 0, 0, 0] → 100% (filter only matches in partition 0)
AggregateExec [ 3, 0, 0, 0] → 100% (Partial: groups, not rows)
RepartitionExec Hash [1, 1, 1, 0] → ~11% (3 groups re-spread by key)
```
I added a Explain Analyze test
Yes, a new metric will show in the explain analyze plan for the nodes
mentioned above.
(cherry picked from commit c2baac5)
0 commit comments