Repository navigation
Commit 9f8fade
committed
test: benchmark the codegen-dispatch path against the operator fallback it replaces
Every expression this PR newly routes through the JVM codegen dispatcher used
to fall the whole projection back to Spark, so the comparison that matters is
dispatch against `spark.comet.exec.scalaUDF.codegen.enabled=false`, not against
Spark. `CometCodegenDispatchBenchmark` runs both, plus a Comet-off reference,
over `lpad`/`rpad` on binary, `encode`, the utf-8 form of `to_binary`,
`to_time(fmt)`, a projection that mixes a dispatched call with native ones, and
a grouped aggregate that only stays native if the projection under it does.
Sub-batch (1024) and multi-batch (1M) row counts. Each table repeats its
baseline so the machine's noise floor is visible next to the arms, and each
case is checked to be fully native with dispatch on, not fully native with it
off, and to have actually reached the dispatcher -- the first two shapes I
tried failed that check because `length()` on binary has no native path, which
would have made those tables measure nothing.
Compilation is a one-time cost that the steady-state tables warm away, so a
separate table times a cold kernel against the same query run twice.
All three arms are run and their rows compared before anything is timed.1 parent 517c466 commit 9f8fade
1 file changed
Lines changed: 407 additions & 0 deletions
0 commit comments