Skip to content

Commit 646ff18

Browse files
andygrovemixermtclaude
authored
fix: park the native scan loop instead of busy-polling while waiting on native I/O (#6219)
* fix: park the native scan loop instead of busy-polling while waiting on native I/O executePlan's ScanExec path re-polled the plan's stream in a tight loop whenever it returned Pending, relying on pull_input_batches to block on the JVM iterators in between. Once every JVM-fed scan holds a batch or has reached EOF that pull is a no-op, so a stream pending on native I/O (a Parquet or Iceberg scan reading from S3 or HDFS) spun the executor thread at 100% CPU for the whole read. A broadcast hash join over a native scan hits this on every probe-side read. pull_input_batches and the two scan operators now report whether a buffer was refilled. When the stream is Pending and nothing was pulled, the loop parks the block_on task until a waker registered by that poll fires, bounded by a short safety timeout, then re-enters the loop so JVM-fed scans still get refilled. Awaiting the stream directly is not safe: ScanExec returns Pending without a waker when an operator drains and re-polls it within one poll. The metrics interval is checked every iteration now that iterations are no longer spins. Closes #6091 Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix: wake the scan streams on refill and park the loop without a timeout ScanStream and ShuffleScanStream now register the poll's waker when their buffer is empty and get_next_batch wakes it after refilling, so every Pending from the plan carries a waker. The loop in executePlan moves into next_batch(stream, on_pending): poll, refill and check metrics on Pending, then park until a waker fires. The 100 ms timeout, the bool from get_next_batch and the tokio time feature are gone. EOF stays buffered so a re-poll of an exhausted scan returns Ready(None) again instead of another JNI round trip. The tracing memory sample sits behind the metrics interval, and development.md describes the loop. Tests drive next_batch with a stream pending on a sleep (with the park removed it pulls 135,311 times in 50 ms) and with a ScanExec refilled by the pull closure under a timeout, plus a ShuffleScanStream waker test. Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix: bound the sleep-based loop test and correct the threading notes The JVM data source path polls operators on the Spark executor thread inside block_on; only tasks they spawn run on tokio workers. The heading now names ShuffleScanExec as well, since pull_input_batches feeds both streams and both register a waker. The native-wait test gets the same ten second bound as the refill test, so a lost wake fails instead of hanging the suite. Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix: skip the park after a pull that made a JNI call A JNI pull can run another Comet native plan on the executor thread, for example when a native writer's input is itself native and is fed in through CometArrowStream. That plan's block_on shares the thread's tokio parker, which holds a single wake-up token. If the outer plan's native I/O completes while the nested plan is parked, the nested park consumes the wake-up, and the outer loop, which parked after every pull, then waits for a wake that has already fired. get_next_batch on both scan operators reports again whether it made the JNI call, and next_batch parks only when no scan did. A no-op pull still parks, so the busy-poll in #6091 stays fixed. The wakers stay, so every Pending still carries one and the park needs no timeout. The new nested block_on test runs a block_on inside the pull closure that outlasts the stream's sleep, and asserts on elapsed time, because the test's ten-second timeout would otherwise rescue the lost wake and let it pass. * fix: notice a wake-up with a flag of the loop's own, not the pull's result next_batch skipped the park after a pull that made a JNI call, because the call can run another Comet plan on this thread, and that plan's block_on shares the thread's parker and can take the wake-up meant for this loop. That relied on every such call reporting itself, and the metrics update in the same closure calls into the JVM without doing so. As suggested in review, next_batch now polls the stream with a waker that sets a flag of its own before forwarding to block_on. A nested park can take the parker's wake-up but not the flag, so the loop parks only when the flag is still clear after the pull, and otherwise wakes block_on itself to poll again. get_next_batch and pull_input_batches return () again, as in the earlier commits. It polls again by yielding to block_on rather than looping inside one poll, so each poll gets a fresh coop budget. A stream that has spent its budget wakes itself and returns Pending, and a loop would only get past that because block_in_place leaves the budget unconstrained on a thread that is not a tokio worker. The loop tests now assert on elapsed time in the shared helper, because the timeout's timer polls the future again when it fires, and that can finish it. Without the check, a WakeFlag that did not forward to block_on still passed the native I/O test, in 10 s. * docs: say why next_batch runs on_pending inside block_in_place The comment came over from the old loop and said block_in_place lets tokio move other tasks off this worker. next_batch runs inside block_on on the Spark task thread, which is not a tokio worker, so there is no core to hand off. There block_in_place stops the coop budget and exits the runtime context, and exiting is what lets a nested Comet plan's block_on run inside on_pending instead of panicking with "Cannot start a runtime from within a runtime". on_pending also runs the metrics update now, not only the JNI pull. Suggested in review. --------- Co-authored-by: Michael Taranov <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent b01c168 commit 646ff18

4 files changed

Lines changed: 336 additions & 108 deletions

File tree

‎docs/source/contributor-guide/development.md‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,12 +43,19 @@ onto a tokio worker thread and batches are delivered to the executor thread via
4343
The executor thread parks in `blocking_recv()` until the next batch is ready. This avoids
4444
busy-polling on I/O-bound workloads.
4545

46-
**JVM data source path (ScanExec present):** The executor thread calls `block_on()` and polls the
47-
DataFusion stream directly, interleaving `pull_input_batches()` calls on `Poll::Pending` to feed
48-
data from the JVM into ScanExec operators.
49-
50-
In both cases, DataFusion operators execute on **tokio worker threads**, not on the Spark executor
51-
task thread. All Spark tasks on an executor share one tokio runtime.
46+
**JVM data source path (ScanExec or ShuffleScanExec present):** The executor thread calls
47+
`block_on()` and polls the DataFusion stream directly. On `Poll::Pending` it calls
48+
`pull_input_batches()` to feed data from the JVM into the ScanExec and ShuffleScanExec operators,
49+
whose streams register the poll's waker and are woken by the refill. The stream is polled with a
50+
waker that also sets a flag. If the flag is still clear when the pull returns, the stream is
51+
waiting on native I/O, and the thread parks until a waker fires instead of busy-polling. If it is
52+
set, the loop polls again. It checks the flag rather than trusting the thread's parker because the
53+
pull can run another Comet plan on the same thread, and that plan's `block_on()` shares the parker
54+
and can consume the wake-up meant for the outer loop.
55+
56+
On the async I/O path, DataFusion operators execute on **tokio worker threads**. On the JVM data
57+
source path, `block_on()` polls them on the Spark executor task thread, and any tasks they spawn
58+
run on the shared runtime. All Spark tasks on an executor share one tokio runtime.
5259

5360
### Rules for native code
5461

‎native/core/src/execution/jni_api.rs‎

Lines changed: 223 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -77,8 +77,7 @@ use datafusion_spark::function::string::substring::SparkSubstring;
7777
use datafusion_spark::function::url::try_url_decode::TryUrlDecode as SparkTryUrlDecode;
7878
use datafusion_spark::function::url::url_decode::UrlDecode as SparkUrlDecode;
7979
use datafusion_spark::function::url::url_encode::UrlEncode as SparkUrlEncode;
80-
use futures::poll;
81-
use futures::stream::StreamExt;
80+
use futures::stream::{Stream, StreamExt};
8281
use futures::FutureExt;
8382
use jni::objects::JByteBuffer;
8483
use jni::sys::{jlongArray, JNI_FALSE};
@@ -95,8 +94,13 @@ use parking_lot::Mutex;
9594
use prost::Message;
9695
use std::collections::HashMap;
9796
use std::path::PathBuf;
97+
use std::sync::atomic::{AtomicBool, Ordering};
9898
use std::time::{Duration, Instant};
99-
use std::{sync::Arc, task::Poll};
99+
use std::{
100+
future::poll_fn,
101+
sync::Arc,
102+
task::{Context, Poll, Wake, Waker},
103+
};
100104
use tokio::runtime::{Handle, Runtime};
101105
use tokio::sync::mpsc;
102106

@@ -508,8 +512,6 @@ struct ExecutionContext {
508512
pub metrics_update_interval: Option<Duration>,
509513
// The last update time of metrics
510514
pub metrics_last_update_time: Instant,
511-
/// Counter to avoid checking time on every poll iteration (reduces syscalls)
512-
pub poll_count_since_metrics_check: u32,
513515
/// The time it took to create the native plan and configure the context
514516
pub plan_creation_time: Duration,
515517
/// DataFusion SessionContext
@@ -723,7 +725,6 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_createPlan(
723725
metrics,
724726
metrics_update_interval,
725727
metrics_last_update_time: Instant::now(),
726-
poll_count_since_metrics_check: 0,
727728
plan_creation_time,
728729
session_ctx: session,
729730
debug_native,
@@ -1019,6 +1020,67 @@ fn pull_input_batches(exec_context: &mut ExecutionContext) -> Result<(), CometEr
10191020
})
10201021
}
10211022

1023+
/// Forwards a wake-up to the `block_on` task and records that it happened, so `next_batch` can
1024+
/// tell whether the stream was woken even if something else took the wake-up from the thread's
1025+
/// parker.
1026+
struct WakeFlag {
1027+
woken: AtomicBool,
1028+
parent: Waker,
1029+
}
1030+
1031+
impl Wake for WakeFlag {
1032+
fn wake(self: Arc<Self>) {
1033+
self.wake_by_ref()
1034+
}
1035+
1036+
fn wake_by_ref(self: &Arc<Self>) {
1037+
self.woken.store(true, Ordering::Release);
1038+
self.parent.wake_by_ref();
1039+
}
1040+
}
1041+
1042+
/// Drives `stream` to its next item. JVM-fed scans return `Pending` until `on_pending` refills
1043+
/// them, so every pending poll runs it, and the refill wakes the stream.
1044+
///
1045+
/// Each poll gives the stream a `WakeFlag` waker. If the flag is still clear when `on_pending`
1046+
/// returns, the stream is waiting on native I/O and `block_on` parks until it completes.
1047+
/// Otherwise `next_batch` wakes `block_on` itself so that it polls again at once. It can't rely on
1048+
/// the wake-up that set the flag, because `on_pending` can run another Comet plan on this thread,
1049+
/// as when a native writer's input is itself native. That plan's `block_on` shares this thread's
1050+
/// parker, which holds a single wake-up, and its park can take this one.
1051+
///
1052+
/// It polls again by yielding to `block_on` rather than looping here, so that each poll starts
1053+
/// with a fresh coop budget. A stream that has spent its budget wakes itself and returns
1054+
/// `Pending`, and a loop here would only get past that because `block_in_place` happens to leave
1055+
/// this thread's budget unconstrained.
1056+
async fn next_batch<S>(
1057+
stream: &mut S,
1058+
mut on_pending: impl FnMut() -> Result<(), CometError>,
1059+
) -> Result<Option<RecordBatch>, CometError>
1060+
where
1061+
S: Stream<Item = DataFusionResult<RecordBatch>> + Unpin,
1062+
{
1063+
poll_fn(|cx| {
1064+
let flag = Arc::new(WakeFlag {
1065+
woken: AtomicBool::new(false),
1066+
parent: cx.waker().clone(),
1067+
});
1068+
let waker = Waker::from(Arc::clone(&flag));
1069+
if let Poll::Ready(item) = stream.poll_next_unpin(&mut Context::from_waker(&waker)) {
1070+
return Poll::Ready(Ok(item.transpose()?));
1071+
}
1072+
// `on_pending` calls into the JVM, which can run another Comet plan on this thread.
1073+
// `block_in_place` exits the runtime context so that plan's `block_on` doesn't panic.
1074+
tokio::task::block_in_place(&mut on_pending)?;
1075+
if flag.woken.load(Ordering::Acquire) {
1076+
// Poll again at once: a nested `block_on` may have taken the wake-up.
1077+
cx.waker().wake_by_ref();
1078+
}
1079+
Poll::Pending
1080+
})
1081+
.await
1082+
}
1083+
10221084
/// Accept serialized query plan and the addresses of Arrow Arrays from Spark,
10231085
/// then execute the query. Return addresses of arrow vector.
10241086
/// # Safety
@@ -1156,54 +1218,31 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
11561218
}
11571219
}
11581220

1159-
// ScanExec path: busy-poll to interleave JVM batch pulls with stream polling
1160-
get_runtime().block_on(async {
1161-
loop {
1162-
let next_item = exec_context.stream.as_mut().unwrap().next();
1163-
let poll_output = poll!(next_item);
1164-
1165-
// Only check time/tracing every 100 polls to reduce overhead
1166-
exec_context.poll_count_since_metrics_check += 1;
1167-
if exec_context.poll_count_since_metrics_check >= 100 {
1168-
exec_context.poll_count_since_metrics_check = 0;
1169-
if let Some(interval) = exec_context.metrics_update_interval {
1170-
let now = Instant::now();
1171-
if now - exec_context.metrics_last_update_time >= interval {
1172-
update_metrics(env, exec_context)?;
1173-
exec_context.metrics_last_update_time = now;
1174-
}
1175-
}
1176-
if exec_context.tracing_enabled {
1177-
log_memory_usage(
1178-
&exec_context.tracing_memory_metric_name,
1179-
total_reserved_for_thread(exec_context.rust_thread_id) as u64,
1180-
);
1181-
}
1182-
}
1183-
1184-
match poll_output {
1185-
Poll::Ready(Some(output)) => {
1186-
return prepare_output(
1187-
env,
1188-
array_addrs,
1189-
schema_addrs,
1190-
output?,
1191-
exec_context.debug_native,
1192-
);
1193-
}
1194-
Poll::Ready(None) => {
1195-
log_plan_metrics(exec_context, stage_id, partition);
1196-
return Ok(-1);
1197-
}
1198-
Poll::Pending => {
1199-
// JNI call to pull batches from JVM into ScanExec operators.
1200-
// block_in_place lets tokio move other tasks off this worker
1201-
// while we wait for JVM data.
1202-
tokio::task::block_in_place(|| pull_input_batches(exec_context))?;
1203-
}
1204-
}
1221+
// ScanExec path: JVM-fed scans return `Pending` until `pull_input_batches` refills
1222+
// them and wakes the stream. A poll that is still pending, with nothing having woken
1223+
// the stream by the end of the pull, waits on native I/O, and `next_batch` parks
1224+
// until it completes.
1225+
let mut stream = exec_context.stream.take().unwrap();
1226+
let next = get_runtime().block_on(next_batch(&mut stream, || {
1227+
pull_input_batches(exec_context)?;
1228+
update_metrics_on_interval(env, exec_context)
1229+
}));
1230+
exec_context.stream = Some(stream);
1231+
let next = next?;
1232+
update_metrics_on_interval(env, exec_context)?;
1233+
match next {
1234+
Some(batch) => prepare_output(
1235+
env,
1236+
array_addrs,
1237+
schema_addrs,
1238+
batch,
1239+
exec_context.debug_native,
1240+
),
1241+
None => {
1242+
log_plan_metrics(exec_context, stage_id, partition);
1243+
Ok(-1)
12051244
}
1206-
})
1245+
}
12071246
});
12081247

12091248
if exec_context.tracing_enabled {
@@ -1265,6 +1304,30 @@ pub extern "system" fn Java_org_apache_comet_Native_releasePlan(
12651304
})
12661305
}
12671306

1307+
/// Runs `update_metrics` once the configured interval has passed and, with tracing on, samples
1308+
/// this thread's pool reservation at the same cadence.
1309+
fn update_metrics_on_interval(
1310+
env: &mut Env,
1311+
exec_context: &mut ExecutionContext,
1312+
) -> CometResult<()> {
1313+
let Some(interval) = exec_context.metrics_update_interval else {
1314+
return Ok(());
1315+
};
1316+
let now = Instant::now();
1317+
if now - exec_context.metrics_last_update_time < interval {
1318+
return Ok(());
1319+
}
1320+
update_metrics(env, exec_context)?;
1321+
exec_context.metrics_last_update_time = now;
1322+
if exec_context.tracing_enabled {
1323+
log_memory_usage(
1324+
&exec_context.tracing_memory_metric_name,
1325+
total_reserved_for_thread(exec_context.rust_thread_id) as u64,
1326+
);
1327+
}
1328+
Ok(())
1329+
}
1330+
12681331
fn update_metrics(env: &mut Env, exec_context: &mut ExecutionContext) -> CometResult<()> {
12691332
if let Some(native_query) = &exec_context.root_op {
12701333
let metrics = exec_context.metrics.as_obj();
@@ -1827,15 +1890,22 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_columnarToRowClose(
18271890
#[cfg(test)]
18281891
mod tests {
18291892
use super::*;
1893+
use crate::execution::operators::InputBatch;
1894+
use crate::execution::planner::TEST_EXEC_CONTEXT_ID;
1895+
use arrow::array::{ArrayRef, Int32Array};
18301896
use arrow::datatypes::{DataType, Field, Schema};
18311897
use datafusion::execution::memory_pool::{
18321898
MemoryConsumer, MemoryReservation, UnboundedMemoryPool,
18331899
};
18341900
use datafusion::execution::FunctionRegistry;
1901+
use datafusion::execution::TaskContext;
18351902
use datafusion::logical_expr::ReturnFieldArgs;
1903+
use datafusion::physical_plan::ExecutionPlan;
18361904
use datafusion_comet_proto::spark_expression;
18371905
use datafusion_comet_proto::spark_expression::{AggExpr, Count, Expr, Sum};
18381906
use datafusion_comet_proto::spark_operator::{HashAggregate, ShuffleWriter};
1907+
use std::cell::Cell;
1908+
use std::future::Future;
18391909

18401910
#[test]
18411911
fn skip_partial_eligibility_is_fail_closed() {
@@ -2312,4 +2382,104 @@ mod tests {
23122382
assert_eq!(ret.data_type(), &DataType::Int32, "length({input})");
23132383
}
23142384
}
2385+
2386+
fn single_worker_runtime() -> Runtime {
2387+
tokio::runtime::Builder::new_multi_thread()
2388+
.worker_threads(1)
2389+
.enable_all()
2390+
.build()
2391+
.unwrap()
2392+
}
2393+
2394+
/// Fails when a wake is lost. The timeout keeps a lost wake from hanging the suite, but when
2395+
/// its timer fires it polls `future` again, which can finish it, so this also fails when
2396+
/// `future` took more than five seconds.
2397+
async fn without_a_lost_wake<F: Future>(future: F) -> F::Output {
2398+
let start = Instant::now();
2399+
let output = tokio::time::timeout(Duration::from_secs(10), future)
2400+
.await
2401+
.expect("timed out: a wake was lost");
2402+
let elapsed = start.elapsed();
2403+
assert!(
2404+
elapsed < Duration::from_secs(5),
2405+
"took {elapsed:?}: a wake was lost, and only the timeout's timer woke the task"
2406+
);
2407+
output
2408+
}
2409+
2410+
#[test]
2411+
fn next_batch_parks_while_the_stream_waits_on_native_io() {
2412+
let batch = RecordBatch::new_empty(Arc::new(Schema::empty()));
2413+
let mut stream = futures::stream::once(async move {
2414+
tokio::time::sleep(Duration::from_millis(50)).await;
2415+
Ok::<_, DataFusionError>(batch)
2416+
})
2417+
.boxed();
2418+
let mut pulls = 0;
2419+
let next = single_worker_runtime()
2420+
.block_on(without_a_lost_wake(next_batch(&mut stream, || {
2421+
// Every JVM-fed scan already holds a batch, so the pull wakes nothing.
2422+
pulls += 1;
2423+
Ok(())
2424+
})))
2425+
.unwrap();
2426+
assert!(next.is_some());
2427+
assert!(
2428+
pulls < 5,
2429+
"the loop pulled {pulls} times during one 50 ms wait"
2430+
);
2431+
}
2432+
2433+
#[test]
2434+
fn next_batch_resumes_on_a_refill_and_stops_pulling_after_eof() {
2435+
let mut scan =
2436+
ScanExec::new(TEST_EXEC_CONTEXT_ID, None, "", vec![DataType::Int32]).unwrap();
2437+
let mut stream = scan.execute(0, Arc::new(TaskContext::default())).unwrap();
2438+
let column: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3]));
2439+
let mut inputs = vec![InputBatch::new(vec![column], Some(3)), InputBatch::EOF].into_iter();
2440+
let pulls = Cell::new(0);
2441+
let mut pull = || {
2442+
pulls.set(pulls.get() + 1);
2443+
if let Some(input) = inputs.next() {
2444+
scan.set_input_batch(input);
2445+
}
2446+
Ok::<(), CometError>(())
2447+
};
2448+
// Only the refill's wake gets the stream polled again.
2449+
single_worker_runtime().block_on(without_a_lost_wake(async {
2450+
let first = next_batch(&mut stream, &mut pull).await.unwrap();
2451+
assert_eq!(first.unwrap().num_rows(), 3);
2452+
assert_eq!(pulls.get(), 1);
2453+
assert!(next_batch(&mut stream, &mut pull).await.unwrap().is_none());
2454+
assert_eq!(pulls.get(), 2);
2455+
assert!(next_batch(&mut stream, &mut pull).await.unwrap().is_none());
2456+
assert_eq!(pulls.get(), 2);
2457+
}));
2458+
}
2459+
2460+
/// A pull that runs another Comet plan on this thread, whose `block_on` parks until after the
2461+
/// stream's native I/O has completed. The nested park takes the I/O's wake-up from the
2462+
/// thread's parker, so `next_batch` has to have seen the wake some other way, or it parks
2463+
/// until `without_a_lost_wake`'s timer wakes it.
2464+
#[test]
2465+
fn next_batch_polls_again_when_a_nested_block_on_took_the_wake_up() {
2466+
let runtime = single_worker_runtime();
2467+
let handle = runtime.handle().clone();
2468+
let batch = RecordBatch::new_empty(Arc::new(Schema::empty()));
2469+
let mut stream = futures::stream::once(async move {
2470+
tokio::time::sleep(Duration::from_millis(20)).await;
2471+
Ok::<_, DataFusionError>(batch)
2472+
})
2473+
.boxed();
2474+
let mut pulls = 0;
2475+
let next = runtime
2476+
.block_on(without_a_lost_wake(next_batch(&mut stream, || {
2477+
pulls += 1;
2478+
handle.block_on(tokio::time::sleep(Duration::from_millis(100)));
2479+
Ok(())
2480+
})))
2481+
.unwrap();
2482+
assert!(next.is_some());
2483+
assert_eq!(pulls, 1);
2484+
}
23152485
}

0 commit comments

Comments
 (0)