Skip to content

Commit 5a82a25

Browse files
adivaishCopilot
andcommitted
perf(delta): split-parallel native Delta scan across Spark cores
The v1 native Delta scan read the whole table on one thread, so it lost to Spark's parallel multi-file read despite a fast per-thread kernel. This makes the native scan read across N Spark partitions, each reading a disjoint subset of the table's files, so the read (and the downstream native compute) parallelizes across cores. Native: - snapshot_info(uri) returns (version, numFiles); open_snapshot_at pins a version. - scan_to_batches_split(uri, cols, predicate, version, partition_index, num_partitions) opens the pinned snapshot, enumerates scan files, sorts them by path (so every partition derives the identical order independently), and processes only files where idx % N == partition_index. It replicates the kernel execute() per-file loop with public APIs (read_parquet_files + transform_to_logical for column mapping / partition values, and a deletion-vector keep-mask via arrow filter_record_batch). DeltaScanExec uses it when num_partitions > 1, else the kernel's all-in-one read. New JNI Native.deltaSnapshotInfo. - Native unit tests: split read covers every row exactly once vs the full read; split + projection. Spark: - DeltaScan proto carries version. CometScanRule calls deltaSnapshotInfo on the driver to pin the version and pick numPartitions = min(defaultParallelism, numFiles); CometDeltaNativeScanExec emits that many partitions (outputPartitioning + CometExecRDD). The partition index flows to native via createPlan/executePlan; the count via partitionCount. - Fall back to Spark for deletion-vector (merge-on-read) tables: Delta reads hidden __delta_internal_* columns + a Filter there, a plan shape incompatible with a scan that applies DVs internally. - e2e: multi-file split scan (5 partitions) matches Spark for a full read and a groupBy; DV merge-on-read table falls back and still matches Spark. Benchmark (WSL 5 cores, Spark 4.1 + Delta 4.1.0, median of 7): full-table aggregate flips from a loss to a win — 1M rows 0.64x -> 1.67x vs Spark (2.10x vs comet-convert); 4M rows 0.61x -> 1.90x vs Spark (2.53x vs comet-convert). Write is unchanged (1.7x-5x). Part of apache#174. Co-authored-by: Copilot <[email protected]>
1 parent 068e4c9 commit 5a82a25

9 files changed

Lines changed: 396 additions & 19 deletions

File tree

‎native/core/src/delta/mod.rs‎

Lines changed: 196 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,19 +30,20 @@
3030
use std::collections::HashMap;
3131
use std::sync::Arc;
3232

33-
use delta_kernel::arrow::array::RecordBatch;
33+
use delta_kernel::arrow::array::{BooleanArray, RecordBatch};
34+
use delta_kernel::arrow::compute::filter_record_batch;
3435
use delta_kernel::arrow::datatypes::SchemaRef as ArrowSchemaRef;
3536
use delta_kernel::committer::FileSystemCommitter;
3637
use delta_kernel::engine::arrow_conversion::{TryFromArrow, TryIntoArrow};
3738
use delta_kernel::engine::arrow_data::{ArrowEngineData, EngineDataArrowExt};
3839
use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor;
3940
use delta_kernel::engine::default::storage::store_from_url_opts;
4041
use delta_kernel::engine::default::{DefaultEngine, DefaultEngineBuilder};
41-
use delta_kernel::scan::state::ScanFile;
42+
use delta_kernel::scan::state::{transform_to_logical, ScanFile};
4243
use delta_kernel::schema::{SchemaRef, StructType};
4344
use delta_kernel::transaction::create_table::create_table as kernel_create_table;
4445
use delta_kernel::transaction::{CommitResult, RetryableTransaction};
45-
use delta_kernel::{DeltaResult, Error, Snapshot, SnapshotRef};
46+
use delta_kernel::{DeltaResult, Engine, Error, FileMeta, Snapshot, SnapshotRef};
4647
use itertools::Itertools;
4748
use url::Url;
4849

@@ -79,6 +80,35 @@ fn open_snapshot(table_uri: &str) -> DeltaResult<(SnapshotRef, KernelEngine)> {
7980
Ok((snapshot, engine))
8081
}
8182

83+
/// Open a snapshot of the Delta table at `table_uri`, pinned to `version` when `Some` (else
84+
/// latest). Used so every split-read partition reads the exact same snapshot.
85+
fn open_snapshot_at(
86+
table_uri: &str,
87+
version: Option<u64>,
88+
) -> DeltaResult<(SnapshotRef, KernelEngine)> {
89+
let (url, engine) = build_engine(table_uri)?;
90+
let mut builder = Snapshot::builder_for(url);
91+
if let Some(v) = version {
92+
builder = builder.at_version(v);
93+
}
94+
let snapshot = builder.build(&engine)?;
95+
Ok((snapshot, engine))
96+
}
97+
98+
/// Return the latest snapshot's version and live data-file count for the table at `table_uri`. The
99+
/// driver uses this to pin the version and choose a partition count for split-parallel reads.
100+
pub fn snapshot_info(table_uri: &str) -> DeltaResult<(u64, u64)> {
101+
let (snapshot, engine) = open_snapshot(table_uri)?;
102+
let version = snapshot.version();
103+
let scan = snapshot.scan_builder().build()?;
104+
let mut count: u64 = 0;
105+
for batch in scan.scan_metadata(&engine)? {
106+
let files = batch?.visit_scan_files(Vec::<()>::new(), |acc, _f| acc.push(()))?;
107+
count += files.len() as u64;
108+
}
109+
Ok((version, count))
110+
}
111+
82112
/// Open the latest snapshot of the Delta table at `table_uri` and return a short human-readable
83113
/// summary (version + logical schema).
84114
pub fn snapshot_summary(table_uri: &str) -> DeltaResult<String> {
@@ -156,6 +186,98 @@ pub fn scan_to_batches(
156186
Ok(batches)
157187
}
158188

189+
/// Split-parallel read: read only the data files assigned to partition `partition_index` of
190+
/// `num_partitions`, from the snapshot pinned at `version`. Files are enumerated, sorted by path
191+
/// for a deterministic cross-partition assignment, then partitioned round-robin by index. This
192+
/// replicates the kernel's per-file read loop (deletion vectors via a keep-mask filter, plus the
193+
/// logical transform for column mapping / partition values) using only public kernel APIs, so the
194+
/// output is identical to `scan_to_batches` for the assigned subset. Used by `DeltaScanExec` when
195+
/// the JVM schedules more than one partition, so the read parallelizes across Spark cores.
196+
pub fn scan_to_batches_split(
197+
table_uri: &str,
198+
columns: Option<&[String]>,
199+
predicate: Option<delta_kernel::expressions::PredicateRef>,
200+
version: Option<u64>,
201+
partition_index: usize,
202+
num_partitions: usize,
203+
) -> DeltaResult<Vec<RecordBatch>> {
204+
let num_partitions = num_partitions.max(1);
205+
let (snapshot, engine) = open_snapshot_at(table_uri, version)?;
206+
let read_schema = projected_read_schema(&snapshot, columns)?;
207+
let scan = snapshot
208+
.scan_builder()
209+
.with_schema_opt(read_schema)
210+
.with_predicate(predicate)
211+
.build()?;
212+
213+
// Enumerate all live scan files, then sort by path so every partition derives the identical
214+
// ordering independently (the round-robin assignment below depends on a stable order).
215+
let mut scan_files: Vec<ScanFile> = Vec::new();
216+
for batch in scan.scan_metadata(&engine)? {
217+
scan_files = batch?.visit_scan_files(scan_files, |acc, f| acc.push(f))?;
218+
}
219+
scan_files.sort_by(|a, b| a.path.cmp(&b.path));
220+
221+
let table_root = scan.snapshot().table_root().clone();
222+
let physical_schema = Arc::clone(scan.physical_schema());
223+
let logical_schema = Arc::clone(scan.logical_schema());
224+
225+
let mut batches: Vec<RecordBatch> = Vec::new();
226+
for scan_file in scan_files
227+
.into_iter()
228+
.enumerate()
229+
.filter(|(i, _)| i % num_partitions == partition_index)
230+
.map(|(_, f)| f)
231+
{
232+
let file_path = table_root.join(&scan_file.path)?;
233+
// Deletion-vector keep-mask for the whole file (true = keep). None when the file has no DV.
234+
let mut keep_mask: Option<Vec<bool>> = scan_file
235+
.dv_info
236+
.get_selection_vector(&engine, &table_root)?;
237+
let meta = FileMeta {
238+
last_modified: 0,
239+
size: scan_file.size as u64,
240+
location: file_path,
241+
};
242+
let read_iter = engine.parquet_handler().read_parquet_files(
243+
&[meta],
244+
Arc::clone(&physical_schema),
245+
None,
246+
)?;
247+
for read_result in read_iter {
248+
let logical = transform_to_logical(
249+
&engine,
250+
read_result?,
251+
&physical_schema,
252+
&logical_schema,
253+
scan_file.transform.clone(),
254+
)?;
255+
let batch = logical.try_into_record_batch()?;
256+
let n = batch.num_rows();
257+
// Consume the leading `n` entries of the keep mask for this batch; the remainder
258+
// applies to subsequent batches of the same file. Rows past the mask are kept.
259+
let masked = match keep_mask.as_mut() {
260+
Some(mask) => {
261+
let rest = if n < mask.len() {
262+
Some(mask.split_off(n))
263+
} else {
264+
None
265+
};
266+
let mut this = std::mem::take(mask);
267+
keep_mask = rest;
268+
if this.len() < n {
269+
this.resize(n, true);
270+
}
271+
Some(filter_record_batch(&batch, &BooleanArray::from(this))?)
272+
}
273+
None => None,
274+
};
275+
batches.push(masked.unwrap_or(batch));
276+
}
277+
}
278+
Ok(batches)
279+
}
280+
159281
/// Read the latest snapshot's logical schema as an Arrow schema, projected to `columns` when
160282
/// `Some`. Used by `DeltaScanExec` to report its output schema without reading any data.
161283
pub fn snapshot_arrow_schema(
@@ -335,6 +457,77 @@ mod tests {
335457
batches.iter().map(|b| b.num_rows()).sum()
336458
}
337459

460+
fn sorted_ids(batches: &[RecordBatch]) -> Vec<i64> {
461+
let mut ids: Vec<i64> = batches
462+
.iter()
463+
.flat_map(|b| {
464+
b.column(0)
465+
.as_any()
466+
.downcast_ref::<Int64Array>()
467+
.unwrap()
468+
.values()
469+
.to_vec()
470+
})
471+
.collect();
472+
ids.sort_unstable();
473+
ids
474+
}
475+
476+
#[test]
477+
fn split_read_covers_all_rows_exactly_once() {
478+
let dir = TempDir::new().unwrap();
479+
let uri = table_uri(&dir);
480+
let schema = test_schema();
481+
create_table(&uri, Arc::clone(&schema)).unwrap();
482+
// five commits => five data files, so a 3-way split has uneven file counts per partition.
483+
for i in 0..5 {
484+
rt().block_on(append(&uri, make_batch(&schema, i * 10, 10)))
485+
.unwrap();
486+
}
487+
488+
let (version, num_files) = snapshot_info(&uri).unwrap();
489+
assert_eq!(num_files, 5);
490+
491+
let full = sorted_ids(&scan_to_batches(&uri, None, None).unwrap());
492+
assert_eq!(full.len(), 50);
493+
494+
let num_partitions = 3;
495+
let mut combined: Vec<i64> = Vec::new();
496+
for p in 0..num_partitions {
497+
let part =
498+
scan_to_batches_split(&uri, None, None, Some(version), p, num_partitions).unwrap();
499+
combined.extend(sorted_ids(&part));
500+
}
501+
combined.sort_unstable();
502+
// Every row is read exactly once across the partitions, matching the full read.
503+
assert_eq!(combined, full);
504+
}
505+
506+
#[test]
507+
fn split_read_with_projection_matches_full() {
508+
let dir = TempDir::new().unwrap();
509+
let uri = table_uri(&dir);
510+
let schema = test_schema();
511+
create_table(&uri, Arc::clone(&schema)).unwrap();
512+
for i in 0..4 {
513+
rt().block_on(append(&uri, make_batch(&schema, i * 8, 8)))
514+
.unwrap();
515+
}
516+
let (version, _) = snapshot_info(&uri).unwrap();
517+
let cols = vec!["id".to_string()];
518+
519+
let mut total = 0usize;
520+
for p in 0..2 {
521+
let part = scan_to_batches_split(&uri, Some(&cols), None, Some(version), p, 2).unwrap();
522+
for b in &part {
523+
assert_eq!(b.num_columns(), 1);
524+
assert_eq!(b.schema().field(0).name(), "id");
525+
}
526+
total += total_rows(&part);
527+
}
528+
assert_eq!(total, 32);
529+
}
530+
338531
#[test]
339532
fn opening_a_missing_table_returns_err() {
340533
assert!(snapshot_summary("file:///definitely/not/a/delta/table").is_err());

‎native/core/src/delta/scan_exec.rs‎

Lines changed: 36 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -36,19 +36,23 @@ use delta_kernel::expressions::PredicateRef;
3636
use futures::TryStreamExt;
3737

3838
use super::predicate::translate_filters;
39-
use super::{scan_to_batches, snapshot_arrow_schema};
39+
use super::{scan_to_batches, scan_to_batches_split, snapshot_arrow_schema};
4040

4141
/// Leaf operator that reads a Delta table through the kernel default engine and emits the
4242
/// resulting Arrow batches. The kernel applies deletion vectors, column mapping, and partition
4343
/// values, so the output is Spark-correct. When `projection` is `Some`, only those columns are
4444
/// read, in the given order (column pruning). Translatable data filters are pushed into the kernel
45-
/// for best-effort file skipping. This v1 reads the table as a single partition; per-file split
46-
/// parallelism is a follow-up.
45+
/// for best-effort file skipping. When `num_partitions > 1`, this instance reads only the file
46+
/// subset assigned to `partition_index` (from the pinned `version`), so the read parallelizes
47+
/// across Spark cores; with a single partition it uses the kernel's all-in-one read.
4748
#[derive(Debug)]
4849
pub struct DeltaScanExec {
4950
table_uri: String,
5051
projection: Option<Vec<String>>,
5152
predicate: Option<PredicateRef>,
53+
version: Option<u64>,
54+
partition_index: usize,
55+
num_partitions: usize,
5256
output_schema: SchemaRef,
5357
plan_properties: Arc<PlanProperties>,
5458
metrics: ExecutionPlanMetricsSet,
@@ -59,10 +63,15 @@ impl DeltaScanExec {
5963
/// is `Some`, the output schema is pruned to those columns, in order. `data_filters` are the
6064
/// scan's Spark filters; the supported subset is translated into a kernel predicate for data
6165
/// skipping (the rest are dropped — correctness is enforced by a Filter above the scan).
66+
/// `version`/`partition_index`/`num_partitions` drive split-parallel reads (one Spark task per
67+
/// partition, each reading a disjoint file subset of the pinned snapshot).
6268
pub fn try_new(
6369
table_uri: String,
6470
projection: Option<Vec<String>>,
6571
data_filters: &[spark_expression::Expr],
72+
version: Option<u64>,
73+
partition_index: usize,
74+
num_partitions: usize,
6675
) -> DFResult<Self> {
6776
let output_schema =
6877
snapshot_arrow_schema(&table_uri, projection.as_deref()).map_err(|e| {
@@ -79,6 +88,9 @@ impl DeltaScanExec {
7988
table_uri,
8089
projection,
8190
predicate,
91+
version,
92+
partition_index,
93+
num_partitions: num_partitions.max(1),
8294
output_schema,
8395
plan_properties,
8496
metrics: ExecutionPlanMetricsSet::new(),
@@ -128,13 +140,28 @@ impl ExecutionPlan for DeltaScanExec {
128140
let table_uri = self.table_uri.clone();
129141
let projection = self.projection.clone();
130142
let predicate = self.predicate.clone();
143+
let version = self.version;
144+
let partition_index = self.partition_index;
145+
let num_partitions = self.num_partitions;
131146
let schema = Arc::clone(&self.output_schema);
132147

133148
// The kernel read is synchronous and blocking; run it on a blocking worker so it does not
134-
// stall the tokio reactor, then stream the resulting batches.
149+
// stall the tokio reactor, then stream the resulting batches. With a single partition use
150+
// the kernel's all-in-one read; otherwise read only this partition's file subset.
135151
let fut = async move {
136152
let batches = tokio::task::spawn_blocking(move || {
137-
scan_to_batches(&table_uri, projection.as_deref(), predicate)
153+
if num_partitions > 1 {
154+
scan_to_batches_split(
155+
&table_uri,
156+
projection.as_deref(),
157+
predicate,
158+
version,
159+
partition_index,
160+
num_partitions,
161+
)
162+
} else {
163+
scan_to_batches(&table_uri, projection.as_deref(), predicate)
164+
}
138165
})
139166
.await
140167
.map_err(|e| DataFusionError::Execution(format!("delta: scan task failed: {e}")))?
@@ -191,7 +218,7 @@ mod tests {
191218
rt.block_on(super::super::append(&uri, batch)).unwrap();
192219

193220
let exec: Arc<dyn ExecutionPlan> =
194-
Arc::new(DeltaScanExec::try_new(uri, None, &[]).unwrap());
221+
Arc::new(DeltaScanExec::try_new(uri, None, &[], None, 0, 1).unwrap());
195222
assert_eq!(exec.schema().fields().len(), 2);
196223

197224
let ctx = SessionContext::new();
@@ -229,8 +256,9 @@ mod tests {
229256
let rt = tokio::runtime::Runtime::new().unwrap();
230257
rt.block_on(super::super::append(&uri, batch)).unwrap();
231258

232-
let exec: Arc<dyn ExecutionPlan> =
233-
Arc::new(DeltaScanExec::try_new(uri, Some(vec!["score".to_string()]), &[]).unwrap());
259+
let exec: Arc<dyn ExecutionPlan> = Arc::new(
260+
DeltaScanExec::try_new(uri, Some(vec!["score".to_string()]), &[], None, 0, 1).unwrap(),
261+
);
234262
assert_eq!(exec.schema().fields().len(), 1);
235263
assert_eq!(exec.schema().field(0).name(), "score");
236264

‎native/core/src/execution/jni_api.rs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1337,3 +1337,24 @@ pub extern "system" fn Java_org_apache_comet_Native_writeDeltaTable(
13371337
Ok(version as jlong)
13381338
})
13391339
}
1340+
1341+
/// Return [version, numFiles] for the latest snapshot of the Delta table at `table_uri`. The
1342+
/// driver pins `version` into the scan plan and uses `numFiles` to choose a partition count for
1343+
/// split-parallel reads. Compiled only with the `delta` feature.
1344+
#[cfg(feature = "delta")]
1345+
#[no_mangle]
1346+
pub extern "system" fn Java_org_apache_comet_Native_deltaSnapshotInfo(
1347+
e: EnvUnowned,
1348+
_class: JClass,
1349+
table_uri: JString,
1350+
) -> jlongArray {
1351+
try_unwrap_or_throw(&e, |env| {
1352+
let uri: String = table_uri.try_to_string(env)?;
1353+
let (version, num_files) = crate::delta::snapshot_info(&uri).map_err(|err| {
1354+
CometError::Internal(format!("native delta snapshot_info failed: {err}"))
1355+
})?;
1356+
let long_array = env.new_long_array(2)?;
1357+
long_array.set_region(env, 0, &[version as jlong, num_files as jlong])?;
1358+
Ok(long_array.into_raw())
1359+
})
1360+
}

‎native/core/src/execution/planner.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1616,6 +1616,13 @@ impl PhysicalPlanner {
16161616
scan.table_uri.clone(),
16171617
projection,
16181618
&scan.data_filters,
1619+
if scan.version < 0 {
1620+
None
1621+
} else {
1622+
Some(scan.version as u64)
1623+
},
1624+
self.partition as usize,
1625+
partition_count,
16191626
)
16201627
.map_err(|e| GeneralError(format!("Failed to create DeltaScanExec: {e}")))?;
16211628
Ok((

‎native/proto/src/proto/operator.proto‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,8 @@ message DeltaScan {
107107
// file-level data skipping. Best-effort: a Filter above the scan still enforces correctness, so
108108
// only the supported subset is translated; the rest are dropped.
109109
repeated spark.spark_expression.Expr data_filters = 3;
110+
// Snapshot version to pin, so every split-read partition reads the identical snapshot.
111+
int64 version = 4;
110112
}
111113

112114
// Common data shared by all partitions in split mode (sent once at planning)

0 commit comments

Comments
 (0)