You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 16fb61d
Browse filesBrowse the repository at this point in the historyBrowse files
feat: replace the DataFusion conf allowlist with a spill codec config
Replace `spark.comet.exec.allowedDataFusionConfigs` with
`spark.comet.exec.spill.compression.codec`, which sets the codec for
the files that native sorts, aggregations, sort-merge joins, and nested
loop joins spill through DataFusion. It accepts `lz4`, `zstd`, and
`none`, and defaults to `lz4`, the codec Spark uses for its own spill
files by default, so native spills are now compressed by default.
The JVM sends the resolved value, and the native side maps it to
DataFusion's `SpillCompression` before the `spark.comet.datafusion.*`
testing pass-through, which still wins.
Add a test that a spilling native sort writes fewer bytes with the
default codec than with `none`, and list nested loop joins in the spill
compression docs.
0 commit comments