Skip to content

Commit 5514cce

Browse files
committed
perf: create a native plan's spill directories on its first spill
Replace the executor-wide shared DiskManager with a per-plan one whose Directories mode DiskManager is built through DataFusion's TempFileFactory hook on the plan's first spill. A plan that never spills no longer creates or removes a datafusion-* directory on every local disk, and spark.comet.maxTempDirectorySize stays a per-plan limit, so the config and tuning docs go back to their previous text. Update the CometNativeSuite temp dir test: a plan that does not spill creates no temp dir, a sort that spills creates one in a Spark local dir, and both are gone once the plans are released.
1 parent 0843ac8 commit 5514cce

6 files changed

Lines changed: 198 additions & 88 deletions

File tree

‎docs/source/user-guide/latest/tuning/memory.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -276,10 +276,13 @@ flushes sorted spill files. A value larger than `spark.comet.batchSize` is cappe
276276
## Limiting Spill Disk Usage
277277

278278
Native operators that spill to disk (aggregate, sort, shuffle) are bounded by
279-
`spark.comet.maxTempDirectorySize` (default 100 GB). All Comet native plans in an executor share
280-
the limit, so it bounds the spill files of the whole executor. If the limit is reached, further
281-
spills fail and the query errors out. Raise this on workloads with large sort/aggregate/shuffle
282-
spills, or lower it to protect executors on shared disks.
279+
`spark.comet.maxTempDirectorySize` (default 100 GB). The operators of one Comet native plan share
280+
the limit. A Spark task can run more than one native plan at a time, for example the native
281+
operators on either side of a union or a coalesce, so an executor running `N` concurrent tasks may
282+
use more than `N` times this value on shared local disks. If the limit is reached, further spills
283+
fail and the query errors out. Raise this on workloads with large sort/aggregate/shuffle spills, or
284+
lower it to protect executors on shared disks, remembering that the total across an executor is a
285+
multiple of this value.
283286

284287
## Compressing Spill Files
285288

‎native/core/src/execution/jni_api.rs‎

Lines changed: 5 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -44,12 +44,10 @@ use arrow::array::{Array, RecordBatch};
4444
use arrow::datatypes::DataType as ArrowDataType;
4545
use datafusion::common::{DataFusionError, Result as DataFusionResult};
4646
use datafusion::config::SpillCompression;
47-
use datafusion::execution::disk_manager::{DiskManager, DiskManagerMode};
4847
use datafusion::execution::memory_pool::MemoryPool;
4948
use datafusion::execution::runtime_env::RuntimeEnvBuilder;
5049
use datafusion::logical_expr::ScalarUDF;
5150
use datafusion::{
52-
execution::disk_manager::DiskManagerBuilder,
5351
physical_plan::{display::DisplayableExecutionPlan, SendableRecordBatchStream},
5452
prelude::{SessionConfig, SessionContext},
5553
};
@@ -125,6 +123,7 @@ use crate::execution::shuffle::{
125123
decode_remote_shuffle_batch, read_ipc_compressed, CompressionCodec, ShuffleWriterExec,
126124
};
127125
use crate::execution::spark_plan::SparkPlan;
126+
use crate::execution::spill_dirs::plan_disk_manager;
128127

129128
use crate::execution::tracing::{
130129
get_thread_id, log_memory_usage, trace_begin, trace_end, with_trace, POOL_TOTAL_METRIC,
@@ -990,44 +989,6 @@ fn configure_skip_partial_aggregation(config: &mut SessionConfig, plan: &Operato
990989
}
991990
}
992991

993-
/// The key of a shared [`DiskManager`]: the local directories and the temp directory size limit.
994-
type DiskManagerKey = (Vec<String>, u64);
995-
996-
static DISK_MANAGERS: OnceLock<Mutex<HashMap<DiskManagerKey, Arc<DiskManager>>>> = OnceLock::new();
997-
998-
/// Returns the [`DiskManager`] shared by every native plan in this process that uses the same
999-
/// local directories and size limit.
1000-
///
1001-
/// A `DiskManager` in `Directories` mode creates a `datafusion-*` directory in every local
1002-
/// directory when it is built, and removes it recursively when it is dropped. Building one per
1003-
/// plan costs a `stat` and a `mkdir` per local disk when the plan is created, and a recursive
1004-
/// delete when it is released, whether or not the plan ever spills (#7060). Sharing one makes
1005-
/// that a once-per-executor cost. Spill files are still deleted when their last reference is
1006-
/// dropped, so a long-lived `DiskManager` does not accumulate them.
1007-
///
1008-
/// The size limit then covers all plans sharing the manager, not each plan separately.
1009-
fn shared_disk_manager(
1010-
local_dirs: Vec<String>,
1011-
max_temp_directory_size: u64,
1012-
) -> CometResult<Arc<DiskManager>> {
1013-
let mut managers = DISK_MANAGERS
1014-
.get_or_init(|| Mutex::new(HashMap::new()))
1015-
.lock();
1016-
let key = (local_dirs, max_temp_directory_size);
1017-
if let Some(disk_manager) = managers.get(&key) {
1018-
return Ok(Arc::clone(disk_manager));
1019-
}
1020-
let paths = key.0.iter().map(PathBuf::from).collect();
1021-
let disk_manager = Arc::new(
1022-
DiskManagerBuilder::default()
1023-
.with_mode(DiskManagerMode::Directories(paths))
1024-
.with_max_temp_directory_size(max_temp_directory_size)
1025-
.build()?,
1026-
);
1027-
managers.insert(key, Arc::clone(&disk_manager));
1028-
Ok(disk_manager)
1029-
}
1030-
1031992
/// Configure DataFusion session context.
1032993
fn prepare_datafusion_session_context(
1033994
batch_size: usize,
@@ -1038,9 +999,10 @@ fn prepare_datafusion_session_context(
1038999
spark_config: &HashMap<String, String>,
10391000
spark_plan: &Operator,
10401001
) -> CometResult<SessionContext> {
1041-
let mut rt_config = RuntimeEnvBuilder::new();
1042-
rt_config.disk_manager = Some(shared_disk_manager(local_dirs, max_temp_directory_size)?);
1043-
rt_config = rt_config.with_object_store_registry(Arc::new(CometObjectStoreRegistry::default()));
1002+
let paths = local_dirs.into_iter().map(PathBuf::from).collect();
1003+
let mut rt_config = RuntimeEnvBuilder::new()
1004+
.with_disk_manager_builder(plan_disk_manager(paths, max_temp_directory_size))
1005+
.with_object_store_registry(Arc::new(CometObjectStoreRegistry::default()));
10441006
rt_config = rt_config.with_memory_pool(memory_pool);
10451007

10461008
let mut session_config = SessionConfig::new()
@@ -3695,20 +3657,6 @@ mod tests {
36953657
})
36963658
}
36973659

3698-
#[test]
3699-
fn plans_share_a_disk_manager_per_set_of_local_dirs() {
3700-
let dir = tempfile::tempdir().unwrap();
3701-
let dirs = vec![dir.path().to_str().unwrap().to_string()];
3702-
let first = shared_disk_manager(dirs.clone(), 1 << 20).unwrap();
3703-
let second = shared_disk_manager(dirs.clone(), 1 << 20).unwrap();
3704-
assert!(Arc::ptr_eq(&first, &second));
3705-
// The temp directory is created once, not per plan.
3706-
assert_eq!(std::fs::read_dir(dir.path()).unwrap().count(), 1);
3707-
3708-
let other_limit = shared_disk_manager(dirs, 1 << 21).unwrap();
3709-
assert!(!Arc::ptr_eq(&first, &other_limit));
3710-
}
3711-
37123660
#[test]
37133661
fn execute_plan_drains_a_plan() {
37143662
let _guard = serial();

‎native/core/src/execution/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ pub(crate) mod sort;
3333
pub(crate) mod spark_aggregate_state;
3434
pub(crate) mod spark_config;
3535
pub(crate) mod spark_plan;
36+
pub(crate) mod spill_dirs;
3637
pub(crate) mod tracing;
3738
pub(crate) mod utils;
3839

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! The spill directories of a native plan.
19+
//!
20+
//! A [`DiskManager`] in `Directories` mode creates a `datafusion-*` directory in every local
21+
//! directory as soon as it is built, and removes it recursively when it is dropped. Each native
22+
//! plan has its own `DiskManager`, so building one in that mode costs a `stat` and a `mkdir` per
23+
//! local directory when the plan is created, and a recursive delete when it is released, even if
24+
//! the plan never spills (#7060). The plan's disk manager built here creates that `Directories`
25+
//! mode `DiskManager` on the plan's first spill instead, so a plan that never spills touches no
26+
//! directories, and a plan that spills gets the same directories, size limit and cleanup as
27+
//! before.
28+
29+
use datafusion::common::Result;
30+
use datafusion::execution::disk_manager::{DiskManager, DiskManagerBuilder, DiskManagerMode};
31+
use datafusion::execution::{SpillFile, TempFileFactory};
32+
use parking_lot::Mutex;
33+
use std::path::PathBuf;
34+
use std::sync::Arc;
35+
36+
/// Returns the builder of a native plan's [`DiskManager`], which spills into a directory in each
37+
/// of `local_dirs` and limits the plan's spill files to `max_temp_directory_size` bytes in total.
38+
pub(crate) fn plan_disk_manager(
39+
local_dirs: Vec<PathBuf>,
40+
max_temp_directory_size: u64,
41+
) -> DiskManagerBuilder {
42+
let builder = DiskManagerBuilder::default()
43+
.with_mode(DiskManagerMode::Directories(local_dirs))
44+
.with_max_temp_directory_size(max_temp_directory_size);
45+
DiskManagerBuilder::default()
46+
.with_temp_file_factory(Arc::new(LazyDiskManager {
47+
builder,
48+
disk_manager: Mutex::new(None),
49+
}))
50+
.with_max_temp_directory_size(max_temp_directory_size)
51+
}
52+
53+
/// Creates spill files through a [`DiskManager`] that it builds for the first one. Every file
54+
/// counts against that manager's size limit, and keeps the manager and its directories alive.
55+
struct LazyDiskManager {
56+
builder: DiskManagerBuilder,
57+
disk_manager: Mutex<Option<Arc<DiskManager>>>,
58+
}
59+
60+
impl TempFileFactory for LazyDiskManager {
61+
fn create_temp_file(&self, description: &str) -> Result<Arc<dyn SpillFile>> {
62+
let disk_manager = {
63+
let mut disk_manager = self.disk_manager.lock();
64+
match disk_manager.as_ref() {
65+
Some(disk_manager) => Arc::clone(disk_manager),
66+
None => Arc::clone(disk_manager.insert(Arc::new(self.builder.clone().build()?))),
67+
}
68+
};
69+
disk_manager.create_tmp_file(description)
70+
}
71+
}
72+
73+
#[cfg(test)]
74+
mod tests {
75+
use super::*;
76+
use std::io::Write;
77+
use std::path::Path;
78+
79+
fn entries(dir: &Path) -> Vec<PathBuf> {
80+
std::fs::read_dir(dir)
81+
.unwrap()
82+
.map(|entry| entry.unwrap().path())
83+
.collect()
84+
}
85+
86+
#[test]
87+
fn spill_directories_are_created_on_the_first_spill() -> Result<()> {
88+
let local_dirs = [tempfile::tempdir()?, tempfile::tempdir()?];
89+
let paths = local_dirs.iter().map(|dir| dir.path().to_path_buf());
90+
let disk_manager = Arc::new(plan_disk_manager(paths.collect(), 1 << 20).build()?);
91+
let spill_dirs = || -> Vec<PathBuf> {
92+
local_dirs
93+
.iter()
94+
.flat_map(|dir| entries(dir.path()))
95+
.collect()
96+
};
97+
assert!(disk_manager.tmp_files_enabled());
98+
assert!(spill_dirs().is_empty());
99+
100+
// The first spill file creates a directory in each local directory, and later ones
101+
// reuse them.
102+
let first = disk_manager.create_tmp_file("first")?;
103+
let second = disk_manager.create_tmp_file("second")?;
104+
let dirs = spill_dirs();
105+
assert_eq!(dirs.len(), 2);
106+
for file in [&first, &second] {
107+
let parent = file.path().unwrap().parent().unwrap();
108+
assert!(dirs.iter().any(|dir| dir == parent));
109+
}
110+
111+
drop((disk_manager, first, second));
112+
assert!(spill_dirs().is_empty());
113+
Ok(())
114+
}
115+
116+
#[test]
117+
fn each_plan_has_its_own_spill_directories_and_size_limit() -> Result<()> {
118+
let local_dir = tempfile::tempdir()?;
119+
let plan = || plan_disk_manager(vec![local_dir.path().to_path_buf()], 10).build();
120+
let (first, second) = (Arc::new(plan()?), Arc::new(plan()?));
121+
122+
let first_file = first.create_tmp_file("first")?;
123+
let mut first_writer = first_file.open_writer()?;
124+
first_writer.write_all(&[0; 8])?;
125+
// A plan's spill files count against its own limit ...
126+
assert!(first_writer.write_all(&[0; 8]).is_err());
127+
// ... and not against another plan's.
128+
let second_file = second.create_tmp_file("second")?;
129+
second_file.open_writer()?.write_all(&[0; 8])?;
130+
assert_eq!(entries(local_dir.path()).len(), 2);
131+
Ok(())
132+
}
133+
}

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1363,9 +1363,11 @@ object CometConf extends ShimCometConf {
13631363
.category(CATEGORY_TUNING)
13641364
.doc(
13651365
"The maximum amount of data (in bytes) stored inside the temporary directories " +
1366-
"used by native operators when spilling. Shared by all Comet native plans running " +
1367-
"in an executor, so it bounds the spill files of the whole executor. Once the " +
1368-
"limit is reached, further spills will fail and the query will error out.")
1366+
"used by native operators when spilling. Applied to each Comet native plan " +
1367+
"separately, and a Spark task can run more than one native plan at a time, so an " +
1368+
"executor running N concurrent tasks may use more than N times this value on shared " +
1369+
"local disks. Once the limit is reached, further spills will fail and the query will " +
1370+
"error out.")
13691371
.bytesConf(ByteUnit.BYTE)
13701372
.createWithDefault(100L * 1024 * 1024 * 1024) // 100 GB
13711373

‎spark/src/test/scala/org/apache/comet/CometNativeSuite.scala‎

Lines changed: 47 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,14 @@
1919

2020
package org.apache.comet
2121

22+
import scala.concurrent.duration.DurationInt
23+
2224
import org.apache.spark.{SparkEnv, SparkException}
23-
import org.apache.spark.sql.CometTestBase
25+
import org.apache.spark.sql.{CometTestBase, DataFrame}
2426
import org.apache.spark.sql.catalyst.expressions.PrettyAttribute
2527
import org.apache.spark.sql.comet.{CometBroadcastExchangeExec, CometExec, CometExecUtils}
2628
import org.apache.spark.sql.comet.execution.arrow.CometArrowStream
29+
import org.apache.spark.sql.functions.col
2730
import org.apache.spark.sql.types.{LongType, StructField, StructType}
2831
import org.apache.spark.sql.vectorized.ColumnarBatch
2932

@@ -57,34 +60,54 @@ class CometNativeSuite extends CometTestBase {
5760
}
5861

5962
test("Comet native should use spark local dir as temp dir") {
60-
withParquetTable((0 until 100000).map(i => (i, i + 1)), "table") {
61-
val dirs = SparkEnv.get.blockManager.getLocalDiskDirs
62-
dirs.foreach { dir =>
63+
val dirs = SparkEnv.get.blockManager.getLocalDiskDirs
64+
// The number of DataFusion temp dirs in the Spark local dirs.
65+
val tempDirs = () =>
66+
dirs.map { dir =>
6367
val files = new java.io.File(dir).listFiles()
64-
assert(!files.exists(f => f.isDirectory && f.getName.startsWith("datafusion-")))
65-
}
66-
67-
// Check if the DataFusion temporary dir exists in the Spark local dirs when a spark job involving
68-
// Comet native operator is running.
69-
val observedDataFusionDir = spark
70-
.table("table")
71-
.selectExpr("_1 + _2 as value")
72-
.rdd
73-
.mapPartitions { _ =>
74-
dirs.map { dir =>
75-
val files = new java.io.File(dir).listFiles()
76-
files.count(f => f.isDirectory && f.getName.startsWith("datafusion-"))
77-
}.iterator
68+
files.count(f => f.isDirectory && f.getName.startsWith("datafusion-"))
69+
}.sum
70+
// Runs `df` and returns how many temp dirs its tasks see once their first row is ready, while
71+
// the native plan that produced it is still open.
72+
val tempDirsWhileRunning = (df: DataFrame) =>
73+
df.rdd
74+
.mapPartitions { rows =>
75+
rows.hasNext
76+
Iterator(tempDirs())
7877
}
79-
.sum()
80-
assert(observedDataFusionDir > 0)
78+
.collect()
79+
.sum
80+
assert(tempDirs() == 0)
8181

82-
// DataFusion temporary dir should be cleaned up after the job is done.
83-
dirs.foreach { dir =>
84-
val files = new java.io.File(dir).listFiles()
85-
assert(!files.exists(f => f.isDirectory && f.getName.startsWith("datafusion-")))
82+
// A native plan creates its temp dirs when it first spills, so one that never spills creates
83+
// none.
84+
withParquetTable((0 until 100000).map(i => (i, i + 1)), "table") {
85+
assert(tempDirsWhileRunning(spark.table("table").selectExpr("_1 + _2 as value")) == 0)
86+
}
87+
88+
withTempPath { path =>
89+
// A single input file keeps all rows in one task so its sort outgrows the tiny memory pool
90+
// below and must spill.
91+
spark
92+
.createDataFrame((0 until 20000).map(i => (i, "native-sort-spill-" * 16)))
93+
.coalesce(1)
94+
.write
95+
.parquet(path.getAbsolutePath)
96+
withSQLConf(
97+
CometConf.COMET_BATCH_SIZE.key -> "1024",
98+
CometConf.COMET_OFFHEAP_MEMORY_POOL_FRACTION.key -> "0.002",
99+
CometConf.COMET_RESPECT_DATAFUSION_CONFIGS.key -> "true",
100+
"spark.comet.datafusion.execution.sort_spill_reservation_bytes" -> "65536") {
101+
val sorted = spark.read.parquet(path.getAbsolutePath).sortWithinPartitions(col("_1").desc)
102+
assert(tempDirsWhileRunning(sorted) > 0, "The native sort did not spill to a local dir")
86103
}
87104
}
105+
106+
// The temp dirs are removed once the plans are released. The sort reads its spill files on
107+
// Tokio tasks that can finish just after the release, so allow them a moment.
108+
eventually(timeout(10.seconds), interval(100.milliseconds)) {
109+
assert(tempDirs() == 0)
110+
}
88111
}
89112

90113
test("test maxBroadcastTableSize") {

0 commit comments

Comments
 (0)