You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Benchmark nested types as native shuffle hash partitioning keys #5787
spark.comet.shuffle.native.partitioning.hash.nested.enabled (#5567) lets a struct, array or
map be a native shuffle hash partitioning key. Nothing in CometShuffleBenchmark measures those
shapes, so there is no baseline for whether native hashing of a nested key beats letting Spark do
the shuffle, and no way to see a regression in that path.
The existing cases do not cover it:
shuffleArrayBenchmark and shuffleStructBenchmark repartition on a nested column but have no
native-shuffle case, because the gate rejected nested keys when they were written.
shuffleDeeplyNestedBenchmark calls repartition(n) with no key, which is round robin rather
than hash partitioning, so it never exercises the hash kernel.
This is separate from #2904, which was about columnar/JVM shuffle throughput on nested schemas
with a keyless repartition(n).
Describe the potential solution
Add a shuffleNestedHashKeyBenchmark with Spark, Comet (Spark Shuffle), Comet (JVM Shuffle) and
Comet (Native Shuffle) cases, where the native case enables the nested config so it measures the
native hashing path rather than a silent fallback to Spark's shuffle.
Cover shapes that separate the two code paths in the native Murmur3 kernel: a list whose elements
are primitives is vectorized, while a list whose elements are themselves nested falls through to hash_list_array!. So struct<int, string>, array<int>, struct<array<int>, string>, array<struct<int, string>> and struct<map<string, int>, int>, at a small and a large partition
count.
Additional context
Measuring this is what surfaced #5777: array<struct<..>> as a native hash key runs at 0.3-0.4x
of leaving the shuffle to Spark, while every other shape measured is faster natively. The
benchmark should land independently of any kernel change so the before/after comparison has a
reference point that is not moving at the same time.
Activity