Skip to content

Commit e063de9

Browse files
committed
Merge remote-tracking branch 'upstream/main' into feat_native_cartesian_single_partition
# Conflicts: # spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala
2 parents 986b4ec + 383ec56 commit e063de9

23 files changed

Lines changed: 1827 additions & 37 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -570,6 +570,7 @@ jobs:
570570
org.apache.comet.exec.CometJoinSuite
571571
org.apache.comet.exec.CometTypedDatasetSuite
572572
org.apache.spark.sql.comet.CometMapInBatchSuite
573+
org.apache.spark.sql.comet.CometCollectMetricsSuite
573574
org.apache.spark.sql.comet.CometArrowPythonUdfSuite
574575
org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite
575576
org.apache.comet.CometNativeSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -236,6 +236,7 @@ jobs:
236236
org.apache.comet.exec.CometJoinSuite
237237
org.apache.comet.exec.CometTypedDatasetSuite
238238
org.apache.spark.sql.comet.CometMapInBatchSuite
239+
org.apache.spark.sql.comet.CometCollectMetricsSuite
239240
org.apache.spark.sql.comet.CometArrowPythonUdfSuite
240241
org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite
241242
org.apache.comet.CometNativeSuite

‎docs/source/contributor-guide/ffi.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ implementation depends on the source of the data:
9797

9898
- `RowArrowReader`: a Spark `Iterator[InternalRow]` (row input)
9999
- `SparkColumnarArrowReader`: a non-Arrow Spark `ColumnarBatch`
100-
- `ColumnarBatchArrowReader`: an Arrow-backed `ColumnarBatch` (transfers `VectorSchemaRoot` ownership)
100+
- `ColumnarBatchArrowReader`: an Arrow-backed `ColumnarBatch` (retains its buffers before closing the input batch)
101101

102102
The exported `ArrowArrayStream`s are boxed into the `Array[Object]` that `CometExecIterator` / `CometExecRDD` pass
103103
to native `createPlan`, one slot per scan input.
@@ -166,6 +166,12 @@ Off-heap Memory: │
166166

167167
### Ownership and Lifecycle
168168

169+
For JVM columnar consumers, `CometNativeArrowSource` uses `CometArrowStream.readerBatchIter` to return batches
170+
with independent vectors that retain the reader's buffers without copying the data. Closing a batch or its
171+
columns leaves the source reader's reusable vector structure intact for its next batch. The iterator releases
172+
the batch vectors when it advances, and closes the final batch, reader and allocator at task completion. A later
173+
native consumer's `ColumnarBatchArrowReader` retains the buffers before closing its input batch.
174+
169175
The Arrow C Stream Interface transfers ownership by reference count: native takes ownership of each imported batch,
170176
so it is safe to buffer batches in operators such as `SortExec` or `ShuffleWriterExec` without a deep copy.
171177

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,28 @@ under the License.
1919

2020
# Operator Compatibility
2121

22+
## Observations
23+
24+
`CollectMetricsExec`, used by `DataFrame.observe`, preserves Spark's observation counters while
25+
passing the original Arrow batches between native operators. Metric expressions still execute
26+
in Spark over row views of each batch; the data is not copied into a row-based pipeline.
27+
Spark retains ownership of accumulator publication, observation listeners, resets and snapshots.
28+
29+
This path requires native input and is enabled by `spark.comet.exec.collectMetrics.enabled`.
30+
Observations retain Spark execution when a consumer can stop reading mid-batch. This includes
31+
limits, joins, `Dataset.mapPartitions`, replacement sampling at a nonpositive rate, and external
32+
actions such as `foreachPartition` or `rdd.take`, which do not introduce a physical-plan node.
33+
The final row iterator is therefore
34+
treated as potentially partial, even when the eventual action reads all rows. A local sort or
35+
aggregate does not remove this restriction: an external consumer may never request its output,
36+
so Spark may leave its input unread while native stream setup fetches a batch.
37+
38+
A materializing shuffle or broadcast exchange allows columnar observations in its producer
39+
subtree, unless another short-circuiting consumer intervenes. This preserves native observation
40+
in aggregation and shuffle pipelines while keeping counters aligned with Spark's consumption.
41+
Direct observations and write pipelines without such an exchange retain Spark collection;
42+
writer nodes alone do not enable columnar observations.
43+
2244
## Empty Relations
2345

2446
On Spark 4.0 and later, Comet supports `EmptyRelationExec` as a native input. It is enabled by

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -69,11 +69,12 @@ omitted from the tables below and may be reconsidered based on demand:
6969

7070
## Projection and filtering
7171

72-
| Operator | Status | Notes |
73-
| ------------- | ------ | -------------------------------------------------------------------------------------------- |
74-
| `ProjectExec` | ✅ | |
75-
| `FilterExec` | ✅ | |
76-
| `SampleExec` | ⚠️ | Sampling without replacement only. See [Operator Compatibility](compatibility/operators.md). |
72+
| Operator | Status | Notes |
73+
| -------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
74+
| `ProjectExec` | ✅ | |
75+
| `FilterExec` | ✅ | |
76+
| `SampleExec` | ⚠️ | Sampling without replacement only. See [Operator Compatibility](compatibility/operators.md). |
77+
| `CollectMetricsExec` | ✅ | Spark evaluates observation counters while Comet forwards Arrow batches to native consumers. See [Observations](compatibility/operators.md#observations). |
7778

7879
## Sorting and limiting
7980

‎native/core/src/execution/planner.rs‎

Lines changed: 39 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ use arrow::array::{
106106
NullArray, StringBuilder, TimestampMicrosecondArray,
107107
};
108108
use arrow::buffer::{BooleanBuffer, NullBuffer, OffsetBuffer};
109-
use datafusion::physical_plan::filter::FilterExec;
109+
use datafusion::physical_plan::filter::FilterExecBuilder;
110110
use datafusion::physical_plan::limit::GlobalLimitExec;
111111
use datafusion_comet_proto::spark_expression::ListLiteral;
112112
use datafusion_comet_proto::spark_operator::SparkFilePartition;
@@ -1595,10 +1595,13 @@ impl PhysicalPlanner {
15951595
let predicate =
15961596
self.create_expr(filter.predicate.as_ref().unwrap(), child.schema())?;
15971597

1598-
let filter: Arc<dyn ExecutionPlan> = Arc::new(FilterExec::try_new(
1599-
predicate,
1600-
Arc::clone(&child.native_plan),
1601-
)?);
1598+
// FilterExec merges its output up to its own batch size, 8192 rows by default.
1599+
// Use the session batch size so it doesn't build batches bigger than that.
1600+
let filter: Arc<dyn ExecutionPlan> = Arc::new(
1601+
FilterExecBuilder::new(predicate, Arc::clone(&child.native_plan))
1602+
.with_batch_size(self.session_ctx.copied_config().batch_size())
1603+
.build()?,
1604+
);
16021605

16031606
Ok((
16041607
scans,
@@ -3812,6 +3815,34 @@ mod tests {
38123815
assert_eq!(0, filter_exec.additional_native_plans.len());
38133816
}
38143817

3818+
fn batch_size_planner(batch_size: usize) -> PhysicalPlanner {
3819+
let config = SessionConfig::new().with_batch_size(batch_size);
3820+
PhysicalPlanner::new(Arc::new(SessionContext::new_with_config(config)), 0)
3821+
}
3822+
3823+
/// The filter merges its output up to the session batch size and no further. Without it, the
3824+
/// filter merges up to 8192 rows, which can exceed Arrow's 2 GiB limit for large string values.
3825+
#[tokio::test]
3826+
async fn filter_output_respects_session_batch_size() {
3827+
let planner = batch_size_planner(2);
3828+
let op = create_filter(create_scan(), 7);
3829+
let (mut scans, _, planned) = planner.create_plan(&op, &mut vec![], 1).unwrap();
3830+
3831+
// Five single-row batches that all pass the filter.
3832+
let mut inputs = [(0..5)
3833+
.map(|_| InputBatch::Batch(vec![Arc::new(Int32Array::from(vec![7])) as ArrayRef], 1))
3834+
.chain(std::iter::once(InputBatch::EOF))
3835+
.collect::<Vec<_>>()
3836+
.into_iter()];
3837+
let stream = planned
3838+
.native_plan
3839+
.execute(0, SessionContext::new().task_ctx())
3840+
.unwrap();
3841+
let batches = collect_with_inputs(stream, &mut scans, &mut inputs).await;
3842+
let sizes: Vec<_> = batches.iter().map(|batch| batch.num_rows()).collect();
3843+
assert_eq!(sizes, vec![2, 2, 1]);
3844+
}
3845+
38153846
/// Comparisons follow Spark's float ordering without normalizing their operands, except in the
38163847
/// data filters that a scan pushes into the Parquet reader, where pruning has to see a plain
38173848
/// comparison of the column.
@@ -4900,12 +4931,14 @@ mod tests {
49004931
})),
49014932
..Default::default()
49024933
};
4903-
let planner = PhysicalPlanner::default();
4934+
let planner = batch_size_planner(5);
49044935
let (mut scans, _, planned) =
49054936
planner.create_plan(&project, &mut vec![], 1).unwrap();
49064937
let filter_plan = &planned.native_plan.children()[0];
49074938
let filter = filter_plan.downcast_ref::<FilterExec>().unwrap();
49084939
assert_eq!(filter.projection().as_deref(), output.as_deref());
4940+
// The filter keeps the session batch size, whether or not pruning rebuilt it.
4941+
assert_eq!(filter.batch_size(), 5);
49094942
assert_eq!(filter.input().schema().fields().len(), 4);
49104943
for (index, field) in planned.schema().fields().iter().enumerate() {
49114944
assert_eq!(field.name(), &format!("col_{index}"));

‎native/operators/src/filter.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ mod tests {
7676
use datafusion::physical_plan::execution_plan::{
7777
ChildrenPropertiesMode, ReplaceChildrenOptions,
7878
};
79+
use datafusion::physical_plan::filter::FilterExecBuilder;
7980
use datafusion::physical_plan::statistics::{StatisticsArgs, StatisticsContext};
8081

8182
fn filter() -> Result<FilterExec> {
@@ -92,7 +93,9 @@ mod tests {
9293
.collect::<std::result::Result<Vec<_>, _>>()?;
9394
let input = MemorySourceConfig::try_new_exec(&partitions, Arc::clone(&schema), None)?;
9495
let predicate = binary(col("key", &schema)?, Operator::Gt, lit(5i32), &schema)?;
95-
FilterExec::try_new(predicate, input)
96+
FilterExecBuilder::new(predicate, input)
97+
.with_batch_size(5)
98+
.build()
9699
}
97100

98101
#[test]
@@ -113,6 +116,7 @@ mod tests {
113116
let rebuilt = rebuilt.downcast_ref::<CometFilterExec>().unwrap();
114117
assert_eq!(rebuilt.allows_runtime_filter_pushdown(), allowed);
115118
assert!(Arc::ptr_eq(rebuilt.predicate(), filter.predicate()));
119+
assert_eq!(rebuilt.inner().batch_size(), 5);
116120
}
117121
}
118122
Ok(())

‎native/shuffle/benches/shuffle_writer.rs‎

Lines changed: 93 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,10 @@
1616
// under the License.
1717

1818
use arrow::array::builder::{Date32Builder, Decimal128Builder, Int32Builder};
19-
use arrow::array::{builder::StringBuilder, Array, Int32Array, RecordBatch};
19+
use arrow::array::{
20+
builder::StringBuilder, Array, ArrayRef, BooleanArray, Decimal128Array, Int32Array, Int64Array,
21+
RecordBatch,
22+
};
2023
use arrow::datatypes::{DataType, Field, Schema};
2124
use arrow::row::{RowConverter, SortField};
2225
use criterion::{criterion_group, criterion_main, BatchSize, Bencher, Criterion};
@@ -188,6 +191,86 @@ fn criterion_benchmark(c: &mut Criterion) {
188191
high_partition_group.finish();
189192
}
190193

194+
// Include decimal SUM's non-null is_empty state, nullable Boolean-heavy rows,
195+
// and an Int64-only control. Both low and high partition counts matter because
196+
// short runs are more common with fewer hash partitions.
197+
fn boolean_state_benchmark(c: &mut Criterion) {
198+
let mut group = c.benchmark_group("shuffle_writer_boolean_state");
199+
group.sample_size(10);
200+
for shape in ["decimal_sum", "nullable_booleans", "int64_control"] {
201+
for partitions in [4, 200] {
202+
for key_run in [1, 64] {
203+
// Build distinct batches so the gather reads independent buffers and the
204+
// writer does not encode and compress the same rows 100 times.
205+
let batches = (0..100)
206+
.map(|batch_index| boolean_state_batch(8192, shape, key_run, batch_index))
207+
.collect::<Vec<_>>();
208+
group.bench_function(
209+
format!("{shape}/partitions={partitions}/key_run={key_run}"),
210+
|b| {
211+
bench_end_to_end(b, || {
212+
create_shuffle_writer_from_batches(
213+
CompressionCodec::Lz4Frame,
214+
CometPartitioning::Hash(
215+
vec![Arc::new(Column::new("key", 0))],
216+
partitions,
217+
),
218+
batches.clone(),
219+
)
220+
})
221+
},
222+
);
223+
}
224+
}
225+
}
226+
group.finish();
227+
}
228+
229+
fn boolean_state_batch(
230+
rows: usize,
231+
shape: &str,
232+
key_run: usize,
233+
batch_index: usize,
234+
) -> RecordBatch {
235+
let first_row = batch_index * rows;
236+
let row_range = first_row..first_row + rows;
237+
let mut fields = vec![Field::new("key", DataType::Int64, false)];
238+
let mut columns: Vec<ArrayRef> = vec![Arc::new(Int64Array::from_iter_values(
239+
row_range.clone().map(|row| (row / key_run) as i64),
240+
))];
241+
if shape == "decimal_sum" {
242+
fields.push(Field::new("sum", DataType::Decimal128(38, 2), true));
243+
fields.push(Field::new("is_empty", DataType::Boolean, false));
244+
columns.push(Arc::new(
245+
Decimal128Array::from_iter(
246+
row_range
247+
.clone()
248+
.map(|row| (row % 7 != 0).then_some(row as i128)),
249+
)
250+
.with_precision_and_scale(38, 2)
251+
.unwrap(),
252+
));
253+
columns.push(Arc::new(BooleanArray::from_iter(
254+
row_range.clone().map(|row| Some(row % 7 == 0)),
255+
)));
256+
} else {
257+
for column in 0..8 {
258+
if shape == "nullable_booleans" {
259+
fields.push(Field::new(format!("b{column}"), DataType::Boolean, true));
260+
columns.push(Arc::new(BooleanArray::from_iter(row_range.clone().map(
261+
|row| ((row + column) % 5 != 0).then_some((row + column) % 3 == 0),
262+
))));
263+
} else {
264+
fields.push(Field::new(format!("i{column}"), DataType::Int64, false));
265+
columns.push(Arc::new(Int64Array::from_iter_values(
266+
row_range.clone().map(|row| (row + column) as i64),
267+
)));
268+
}
269+
}
270+
}
271+
RecordBatch::try_new(Arc::new(Schema::new(fields)), columns).unwrap()
272+
}
273+
191274
/// Times one execution of a freshly built writer per iteration. A `ShuffleWriterExec`
192275
/// publishes its partition offsets once, so it cannot be re-executed; building it is
193276
/// setup and stays outside the measurement.
@@ -211,6 +294,14 @@ fn create_shuffle_writer_exec(
211294
num_batches: usize,
212295
) -> ShuffleWriterExec {
213296
let batches = create_batches(rows_per_batch, num_batches);
297+
create_shuffle_writer_from_batches(compression_codec, partitioning, batches)
298+
}
299+
300+
fn create_shuffle_writer_from_batches(
301+
compression_codec: CompressionCodec,
302+
partitioning: CometPartitioning,
303+
batches: Vec<RecordBatch>,
304+
) -> ShuffleWriterExec {
214305
let schema = batches[0].schema();
215306
let partitions = &[batches];
216307
ShuffleWriterExec::try_new(
@@ -593,6 +684,6 @@ fn config() -> Criterion {
593684
criterion_group! {
594685
name = benches;
595686
config = config();
596-
targets = criterion_benchmark, partitioning_benchmark, schema_encoding_benchmark, ipc_context_reuse_benchmark
687+
targets = criterion_benchmark, boolean_state_benchmark, partitioning_benchmark, schema_encoding_benchmark, ipc_context_reuse_benchmark
597688
}
598689
criterion_main!(benches);

0 commit comments

Comments
 (0)