Skip to content

Commit 16fb61d

Browse files
committed
feat: replace the DataFusion conf allowlist with a spill codec config
Replace `spark.comet.exec.allowedDataFusionConfigs` with `spark.comet.exec.spill.compression.codec`, which sets the codec for the files that native sorts, aggregations, sort-merge joins, and nested loop joins spill through DataFusion. It accepts `lz4`, `zstd`, and `none`, and defaults to `lz4`, the codec Spark uses for its own spill files by default, so native spills are now compressed by default. The JVM sends the resolved value, and the native side maps it to DataFusion's `SpillCompression` before the `spark.comet.datafusion.*` testing pass-through, which still wins. Add a test that a spilling native sort writes fewer bytes with the default codec than with `none`, and list nested loop joins in the spill compression docs.
1 parent d56c0b5 commit 16fb61d

7 files changed

Lines changed: 90 additions & 45 deletions

File tree

‎docs/source/user-guide/latest/tuning.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,8 @@ guide is split into the following pages:
4141
Rows with many columns, long strings, or nested columns such as arrays of structs make every batch
4242
larger, so queries over such data spill more. For these workloads:
4343

44-
- Compress spill files, which Comet writes uncompressed by default. See
44+
- Choose the codec for spill files from native sorts, aggregations, and joins, which Comet
45+
compresses with `lz4` by default. See
4546
[Compressing Spill Files](tuning/memory.md#compressing-spill-files).
4647
- Lower the batch size if wide batches cause spilling or out-of-memory errors. See
4748
[Batch Size](tuning/memory.md#batch-size).

‎docs/source/user-guide/latest/tuning/memory.md‎

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -285,17 +285,10 @@ multiple of this value.
285285

286286
## Compressing Spill Files
287287

288-
Native sorts, aggregations, and sort-merge joins spill through DataFusion, which writes spill files
289-
uncompressed by default, while Spark compresses its own with `lz4` by default. To compress them,
290-
which reduces spill I/O and how much of `spark.comet.maxTempDirectorySize` they use, set:
291-
292-
```
293-
spark.comet.datafusion.execution.spill_compression=zstd
294-
```
295-
296-
The accepted values are `uncompressed` (the default), `lz4_frame`, and `zstd`. `lz4_frame`
297-
compresses and decompresses faster than `zstd` but writes larger files. `lz4` is not a valid value,
298-
and an invalid value fails every query that runs a Comet native plan. This setting does not need
299-
`spark.comet.exec.respectDataFusionConfigs`, because `spark.comet.exec.allowedDataFusionConfigs`
300-
lists it by default. Comet's shuffle writers do not use it. See
288+
`spark.comet.exec.spill.compression.codec` sets the codec for the files that native sorts,
289+
aggregations, sort-merge joins, and nested loop joins spill to disk. The default is `lz4`, which
290+
Spark also uses for its own spill files by default. `zstd` writes smaller files, which reduces
291+
spill I/O and how much of `spark.comet.maxTempDirectorySize` they use, but takes more CPU to
292+
compress and decompress them. `none` writes them uncompressed. Comet's shuffle writers use
293+
`spark.comet.shuffle.compression.codec` instead. See
301294
[Shuffle Compression](shuffle.md#shuffle-compression).

‎native/core/src/execution/jni_api.rs‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ use std::collections::HashSet;
4040
use arrow::array::{Array, RecordBatch};
4141
use arrow::datatypes::DataType as ArrowDataType;
4242
use datafusion::common::{DataFusionError, Result as DataFusionResult};
43+
use datafusion::config::SpillCompression;
4344
use datafusion::execution::disk_manager::DiskManagerMode;
4445
use datafusion::execution::memory_pool::MemoryPool;
4546
use datafusion::execution::runtime_env::RuntimeEnvBuilder;
@@ -129,9 +130,9 @@ use crate::execution::tracing::{
129130
use crate::execution::memory_pools::logging_pool::LoggingMemoryPool;
130131
use crate::execution::spark_config::{
131132
SparkConfig, COMET_DEBUG_ENABLED, COMET_DEBUG_MEMORY,
132-
COMET_EXEC_AGGREGATE_SKIP_PARTIAL_ENABLED, COMET_EXPLAIN_NATIVE_ENABLED,
133-
COMET_MAX_TEMP_DIRECTORY_SIZE, COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED,
134-
COMET_TRACING_ENABLED, SPARK_EXECUTOR_CORES,
133+
COMET_EXEC_AGGREGATE_SKIP_PARTIAL_ENABLED, COMET_EXEC_SPILL_COMPRESSION_CODEC,
134+
COMET_EXPLAIN_NATIVE_ENABLED, COMET_MAX_TEMP_DIRECTORY_SIZE,
135+
COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED, COMET_TRACING_ENABLED, SPARK_EXECUTOR_CORES,
135136
};
136137
use crate::parquet::encryption_support::{CometEncryptionFactory, ENCRYPTION_FACTORY_ID};
137138
use crate::parquet::parquet_support::CometObjectStoreRegistry;
@@ -892,15 +893,34 @@ fn prepare_datafusion_session_context(
892893
// DataFusion session options. `pushdown_filters` enables the parquet reader's
893894
// RowFilter evaluation during decode (late materialization); `reorder_filters`
894895
// is only meaningful when pushdown_filters is on, so they move together. Set
895-
// before the `spark.comet.datafusion.*` pass-through below, so an explicit
896-
// override of either key wins instead of being silently forced back to `true`.
896+
// before the `spark.comet.datafusion.*` testing escape hatch pass-through below,
897+
// so an explicit override of either key wins instead of being silently forced
898+
// back to `true`.
897899
if spark_config.get_bool(COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED) {
898900
session_config =
899901
session_config.set_str("datafusion.execution.parquet.pushdown_filters", "true");
900902
session_config =
901903
session_config.set_str("datafusion.execution.parquet.reorder_filters", "true");
902904
}
903905

906+
// Translate Comet's spill codec into DataFusion's spill compression, which DataFusion
907+
// otherwise leaves uncompressed. Set before the pass-through below for the same reason,
908+
// so an explicit `spark.comet.datafusion.execution.spill_compression` still wins.
909+
let spill_compression = match spark_config
910+
.get(COMET_EXEC_SPILL_COMPRESSION_CODEC)
911+
.map(String::as_str)
912+
{
913+
None | Some("lz4") => SpillCompression::Lz4Frame,
914+
Some("zstd") => SpillCompression::Zstd,
915+
Some("none") => SpillCompression::Uncompressed,
916+
Some(codec) => {
917+
return Err(CometError::Config(format!(
918+
"Unsupported spill compression codec: {codec}"
919+
)))
920+
}
921+
};
922+
session_config = session_config.with_spill_compression(spill_compression);
923+
904924
// Pass through DataFusion configs from Spark.
905925
// e.g: spark-shell --conf spark.comet.datafusion.sql_parser.parse_float_as_decimal=true
906926
// becomes datafusion.sql_parser.parse_float_as_decimal=true

‎native/core/src/execution/spark_config.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ pub(crate) const COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED: &str =
2626
"spark.comet.parquet.rowFilterPushdown.enabled";
2727
pub(crate) const COMET_EXEC_AGGREGATE_SKIP_PARTIAL_ENABLED: &str =
2828
"spark.comet.exec.aggregate.skipPartial.enabled";
29+
pub(crate) const COMET_EXEC_SPILL_COMPRESSION_CODEC: &str =
30+
"spark.comet.exec.spill.compression.codec";
2931
pub(crate) const SPARK_EXECUTOR_CORES: &str = "spark.executor.cores";
3032

3133
/// Comet configs read through this trait must be resolved by the JVM first:

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1253,29 +1253,29 @@ object CometConf extends ShimCometConf {
12531253
.bytesConf(ByteUnit.BYTE)
12541254
.createWithDefault(100L * 1024 * 1024 * 1024) // 100 GB
12551255

1256+
// Used on native side. Check spark_config.rs how the config is used
1257+
val COMET_EXEC_SPILL_COMPRESSION_CODEC: ConfigEntry[String] =
1258+
conf(s"$COMET_EXEC_CONFIG_PREFIX.spill.compression.codec")
1259+
.category(CATEGORY_TUNING)
1260+
.doc(
1261+
"The codec used to compress the files that native sorts, aggregations, and joins " +
1262+
"spill to disk. lz4 and zstd are supported, and none disables compression. zstd " +
1263+
"writes smaller files than lz4 but uses more CPU. Comet's shuffle writers use " +
1264+
s"${COMET_SHUFFLE_COMPRESSION_CODEC.key} instead. $TUNING_GUIDE.")
1265+
.stringConf
1266+
.checkValues(Set("lz4", "zstd", "none"))
1267+
.createWithDefault("lz4")
1268+
12561269
val COMET_RESPECT_DATAFUSION_CONFIGS: ConfigEntry[Boolean] =
12571270
conf(s"$COMET_EXEC_CONFIG_PREFIX.respectDataFusionConfigs")
12581271
.category(CATEGORY_TESTING)
12591272
.doc(
12601273
"Development and testing configuration option to allow DataFusion configs set in " +
12611274
"Spark configuration settings starting with `spark.comet.datafusion.` to be passed " +
1262-
"into native execution. Configs listed in " +
1263-
"`spark.comet.exec.allowedDataFusionConfigs` are passed even when this is false.")
1275+
"into native execution.")
12641276
.booleanConf
12651277
.createWithDefault(false)
12661278

1267-
val COMET_ALLOWED_DATAFUSION_CONFIGS: ConfigEntry[Seq[String]] =
1268-
conf(s"$COMET_EXEC_CONFIG_PREFIX.allowedDataFusionConfigs")
1269-
.category(CATEGORY_TUNING)
1270-
.doc(
1271-
"A comma-separated list of `spark.comet.datafusion.` configs, each named by its full " +
1272-
"key, that are passed into native execution even when " +
1273-
s"`${COMET_RESPECT_DATAFUSION_CONFIGS.key}` is false. Setting this replaces the " +
1274-
s"default list. $TUNING_GUIDE.")
1275-
.stringConf
1276-
.toSequence
1277-
.createWithDefault(Seq("spark.comet.datafusion.execution.spill_compression"))
1278-
12791279
val COMET_STRICT_TESTING: ConfigEntry[Boolean] = conf(s"$COMET_PREFIX.testing.strict")
12801280
.category(CATEGORY_TESTING)
12811281
.doc("Experimental option to enable strict testing, which will fail tests that could be " +

‎spark/src/main/scala/org/apache/comet/CometExecIterator.scala‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -830,11 +830,9 @@ object CometExecIterator extends Logging {
830830

831831
def serializeCometSQLConfs(): Array[Byte] = {
832832
val builder = ConfigMap.newBuilder()
833-
val respectDataFusionConfigs = CometConf.COMET_RESPECT_DATAFUSION_CONFIGS.get(SQLConf.get)
834-
val allowedDataFusionConfigs = CometConf.COMET_ALLOWED_DATAFUSION_CONFIGS.get(SQLConf.get)
835833
cometSqlConfs.foreach { case (k, v) =>
836834
if (k.startsWith(s"${CometConf.COMET_PREFIX}.datafusion.")) {
837-
if (respectDataFusionConfigs || allowedDataFusionConfigs.contains(k)) {
835+
if (CometConf.COMET_RESPECT_DATAFUSION_CONFIGS.get(SQLConf.get)) {
838836
builder.putEntries(k, v)
839837
}
840838
} else {
@@ -855,6 +853,7 @@ object CometExecIterator extends Logging {
855853
CometConf.COMET_DEBUG_ENABLED,
856854
CometConf.COMET_DEBUG_MEMORY_ENABLED,
857855
CometConf.COMET_EXEC_AGGREGATE_SKIP_PARTIAL_ENABLED,
856+
CometConf.COMET_EXEC_SPILL_COMPRESSION_CODEC,
858857
CometConf.COMET_EXPLAIN_NATIVE_ENABLED,
859858
CometConf.COMET_MAX_TEMP_DIRECTORY_SIZE,
860859
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED,

‎spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala‎

Lines changed: 39 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -110,17 +110,47 @@ class CometExecSuite extends CometTestBase {
110110
}
111111
}
112112

113-
test("SQLConf serde passes allowed DataFusion configs without respectDataFusionConfigs") {
114-
def entries = ConfigMap.parseFrom(CometExecIterator.serializeCometSQLConfs()).getEntriesMap
115-
val spillCompression = "spark.comet.datafusion.execution.spill_compression"
116-
val spillReservation = "spark.comet.datafusion.execution.sort_spill_reservation_bytes"
113+
test("native sort spill files are compressed with the default spill codec") {
114+
val numRows = 20000
115+
val compressibleValue = "native-sort-spill-compression-" * 8
116+
withTempPath { path =>
117+
// A single input file keeps all rows in one task so its sort outgrows the tiny memory
118+
// pool below and must spill.
119+
spark
120+
.createDataFrame((0 until numRows).map(i => (i, compressibleValue)))
121+
.coalesce(1)
122+
.write
123+
.parquet(path.getAbsolutePath)
117124

118-
withSQLConf(spillCompression -> "zstd", spillReservation -> "65536") {
119-
assert(entries.get(spillCompression) == "zstd")
120-
assert(!entries.containsKey(spillReservation))
125+
withParquetTable(path.getAbsolutePath, "tbl") {
126+
def sortSpilledBytes(codecConf: (String, String)*): Long = {
127+
var spilledBytes = 0L
128+
withSQLConf(
129+
Seq(
130+
CometConf.COMET_BATCH_SIZE.key -> "1024",
131+
CometConf.COMET_OFFHEAP_MEMORY_POOL_FRACTION.key -> "0.002",
132+
CometConf.COMET_RESPECT_DATAFUSION_CONFIGS.key -> "true",
133+
"spark.comet.datafusion.execution.sort_spill_reservation_bytes" -> "65536") ++
134+
codecConf: _*) {
135+
val sorted = sql("SELECT * FROM tbl").sortWithinPartitions($"_1".desc)
136+
assert(sorted.collect().length == numRows)
137+
val plan = sorted.queryExecution.executedPlan
138+
val sorts = collect(plan) { case sort: CometSortExec => sort }
139+
assert(sorts.nonEmpty, s"Expected a native sort:\n$plan")
140+
spilledBytes = sorts.map(_.metrics("spilled_bytes").value).sum
141+
assert(spilledBytes > 0L, "Native sort did not spill")
142+
}
143+
spilledBytes
144+
}
121145

122-
withSQLConf(CometConf.COMET_ALLOWED_DATAFUSION_CONFIGS.key -> spillReservation) {
123-
assert(entries.get(spillReservation) == "65536")
146+
// `spilled_bytes` counts what DataFusion wrote to disk, so this shows the default codec
147+
// reaching DataFusion's spill writer, not just crossing JNI.
148+
val uncompressed =
149+
sortSpilledBytes(CometConf.COMET_EXEC_SPILL_COMPRESSION_CODEC.key -> "none")
150+
val compressed = sortSpilledBytes()
151+
assert(
152+
compressed < uncompressed,
153+
s"Spilled $compressed bytes with the default codec and $uncompressed with none")
124154
}
125155
}
126156
}

0 commit comments

Comments
 (0)