Skip to content

Commit 2c28107

Browse files
committed
docs: point contributors at OperatorBuilder and refresh planner pointers
Update the 'Update the Planner' step of the new-operator guide to describe adding an OperatorBuilder under planner/ and registering it, instead of adding an arm to create_plan. Refresh the window debugging example to match WindowBuilder::build, and point the make_all_fields_nullable comments at planner/aggregate.rs.
1 parent 2808120 commit 2c28107

4 files changed

Lines changed: 105 additions & 46 deletions

File tree

‎docs/source/contributor-guide/adding_a_new_operator.md‎

Lines changed: 71 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -468,22 +468,75 @@ Run `make` to update the user guide. The new configuration option will be added
468468

469469
#### Update the Planner
470470

471-
In `native/core/src/execution/planner.rs`, add a match case in the operator deserialization logic to handle your new protobuf message:
472-
473-
```rust
474-
use datafusion_comet_proto::spark_operator::operator::OpStruct;
475-
476-
// In the create_plan or similar method:
477-
match op.op_struct.as_ref() {
478-
Some(OpStruct::Scan(scan)) => {
479-
// ... existing cases ...
480-
}
481-
Some(OpStruct::YourNewOperator(your_op)) => {
482-
create_your_operator_exec(your_op, children, session_ctx)
483-
}
484-
// ... other cases ...
485-
}
486-
```
471+
Plan the operator in an `OperatorBuilder`, not in a match arm in `PhysicalPlanner::create_plan`.
472+
`create_plan` dispatches to `OperatorRegistry` first and only falls through to its own `match` for
473+
operators that have not moved yet, so new operators should not add code there.
474+
475+
1. Create a module under `native/core/src/execution/planner/`, for example `your_operator.rs`, and
476+
declare it in `native/core/src/execution/planner.rs`. Implement `OperatorBuilder` for a unit
477+
struct. The builder plans the child operators through `planner.create_plan` and wraps the result
478+
in a `SparkPlan`:
479+
480+
```rust
481+
use std::sync::Arc;
482+
483+
use datafusion::physical_plan::ExecutionPlan;
484+
use datafusion_comet_proto::spark_operator::Operator;
485+
use jni::objects::{Global, JObject};
486+
487+
use super::{operator_registry::OperatorBuilder, PhysicalPlanner, PlanCreationResult};
488+
use crate::{execution::spark_plan::SparkPlan, extract_op};
489+
490+
/// Builder for `YourNewOperator`.
491+
pub struct YourNewOperatorBuilder;
492+
493+
impl OperatorBuilder for YourNewOperatorBuilder {
494+
fn build(
495+
&self,
496+
spark_plan: &Operator,
497+
inputs: &mut Vec<Arc<Global<JObject<'static>>>>,
498+
partition_count: usize,
499+
planner: &PhysicalPlanner,
500+
) -> PlanCreationResult {
501+
let your_op = extract_op!(spark_plan, YourNewOperator);
502+
let (scans, shuffle_scans, child) =
503+
planner.create_plan(&spark_plan.children[0], inputs, partition_count)?;
504+
505+
// Deserialize expressions and configuration, then create the execution plan.
506+
let exec = create_your_operator_exec(your_op, Arc::clone(&child.native_plan))?;
507+
508+
Ok((
509+
scans,
510+
shuffle_scans,
511+
Arc::new(SparkPlan::new(spark_plan.plan_id, exec, vec![child])),
512+
))
513+
}
514+
}
515+
```
516+
517+
See `explode.rs` for a small builder and `window.rs` for one that rewrites its output.
518+
519+
2. In `native/core/src/execution/planner/operator_registry.rs`, add a variant to `OperatorType`,
520+
map the `OpStruct` variant to it in `get_operator_type`, and add a `register_*_operators` method
521+
that inserts the builder, called from `register_all_operators`:
522+
523+
```rust
524+
OpStruct::YourNewOperator(_) => Some(OperatorType::YourNewOperator),
525+
```
526+
527+
```rust
528+
fn register_your_operator_operators(&mut self) {
529+
use super::your_operator::YourNewOperatorBuilder;
530+
531+
self.builders.insert(
532+
OperatorType::YourNewOperator,
533+
Box::new(YourNewOperatorBuilder),
534+
);
535+
}
536+
```
537+
538+
Keep the helpers that only your operator uses in the same module, as `aggregate.rs` and
539+
`join.rs` do.
487540

488541
`ProjectionBuilder` prunes a DataFusion filter's output for column-only projections, including
489542
empty projections such as the input to `count(*)`. Each required output column is filtered once,
@@ -493,16 +546,15 @@ plan ID. Computed projections and projections that use every input column are un
493546

494547
#### Implement the Operator
495548

496-
Create the operator implementation, either in an existing file or a new file in `native/core/src/execution/operators/`:
549+
Create the operator implementation that the builder calls, either in an existing file or a new file in `native/core/src/execution/operators/`:
497550

498551
```rust
499552
use datafusion::physical_plan::{ExecutionPlan, ...};
500553
use datafusion_comet_proto::spark_operator::YourNewOperator;
501554

502555
pub fn create_your_operator_exec(
503556
op: &YourNewOperator,
504-
children: Vec<Arc<dyn ExecutionPlan>>,
505-
session_ctx: &SessionContext,
557+
child: Arc<dyn ExecutionPlan>,
506558
) -> Result<Arc<dyn ExecutionPlan>, ExecutionError> {
507559
// Deserialize expressions and configuration
508560
// Create and return the execution plan

‎docs/source/contributor-guide/debugging.md‎

Lines changed: 26 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -255,35 +255,42 @@ so `spark.comet.debug.enabled` and `COMET_LOG_LEVEL` do not affect it.
255255
#### Example: dumping window operator output
256256

257257
In `WindowBuilder::build` in `native/core/src/execution/planner/window.rs`,
258-
the window operator builds a `BoundedWindowAggExec` and hands it to
259-
`SparkPlan::new`. Wrap the node between those two steps:
258+
the window operator builds `window_agg`, either a `BoundedWindowAggExec` or a
259+
`WindowAggExec` depending on whether every window expression runs with bounded
260+
memory. It then optionally wraps `window_agg` in a casting `ProjectionExec` as
261+
`final_plan` and hands `final_plan` to `SparkPlan::new`. To see the raw window
262+
output, wrap `window_agg` right after it is built, before the cast projection:
260263

261264
```rust
262-
let window_agg = Arc::new(BoundedWindowAggExec::try_new(
263-
window_expr?,
264-
Arc::clone(&child.native_plan),
265-
InputOrderMode::Sorted,
266-
!partition_exprs.is_empty(),
267-
)?);
265+
let window_agg: Arc<dyn ExecutionPlan> = if all_bounded {
266+
Arc::new(BoundedWindowAggExec::try_new(
267+
window_expr,
268+
Arc::clone(&child.native_plan),
269+
InputOrderMode::Sorted,
270+
!partition_exprs.is_empty(),
271+
)?)
272+
} else {
273+
Arc::new(WindowAggExec::try_new(
274+
window_expr,
275+
Arc::clone(&child.native_plan),
276+
!partition_exprs.is_empty(),
277+
)?)
278+
};
268279

269280
#[cfg(debug_assertions)]
270-
let window_agg: Arc<dyn datafusion::physical_plan::ExecutionPlan> = {
281+
let window_agg: Arc<dyn ExecutionPlan> = {
271282
use crate::debug::DebugExecutionDataStream;
272283
Arc::new(DebugExecutionDataStream::new("window-output", window_agg))
273284
};
274285

275-
Ok((
276-
scans,
277-
shuffle_scans,
278-
Arc::new(SparkPlan::new(spark_plan.plan_id, window_agg, vec![child])),
279-
))
286+
// ... `final_plan` is built from `window_agg`, and
287+
// `SparkPlan::new(spark_plan.plan_id, final_plan, vec![child])` wraps it.
280288
```
281289

282-
The `let` rebinding keeps the original `window_agg` typed concretely on release
283-
builds (where the `cfg` block is compiled out) and re-binds it to
284-
`Arc<dyn ExecutionPlan>` when the debug module is active. No annotation is
285-
needed on the first `let` because release builds use the original binding
286-
directly.
290+
Wrapping `window_agg` leaves the rest of the builder unchanged. To see the
291+
output after the cast, wrap `final_plan` the same way just before the
292+
`SparkPlan::new` call. The `let` rebinding is only compiled in when
293+
`debug_assertions` is on, so release builds use the original binding.
287294

288295
Rebuild with `make core` and run a window test, for example:
289296

‎native/common/src/schema.rs‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -100,8 +100,8 @@ fn cast_error(
100100
/// Describes where `expected` and `actual` first diverge, as a dotted path rooted at `path`.
101101
/// Returns `None` when the two types are equal.
102102
///
103-
/// The nested arms cover the shapes Comet actually builds — see `make_all_fields_nullable` in the
104-
/// planner and `to_arrow_datatype` in the serde layer, which walk the same set.
103+
/// The nested arms cover the shapes Comet actually builds — see `make_all_fields_nullable` in
104+
/// `planner/aggregate.rs` and `to_arrow_datatype` in the serde layer, which walk the same set.
105105
fn describe_type_mismatch(path: &str, expected: &DataType, actual: &DataType) -> Option<String> {
106106
if expected == actual {
107107
return None;
@@ -208,9 +208,9 @@ pub fn describe_timestamp_label_mismatch(
208208
/// counts, different list flavours, ...) are left as `base`; those are real type differences and
209209
/// are handled by the cast in [`cast_and_stamp_schema`].
210210
///
211-
/// Related but not interchangeable: `make_all_fields_nullable` in the planner widens one type
212-
/// unconditionally, and arrow's `Field::try_merge` unions two but errors on a leaf type difference
213-
/// instead of tolerating it and does not recurse into maps or fixed-size lists.
211+
/// Related but not interchangeable: `make_all_fields_nullable` in `planner/aggregate.rs` widens
212+
/// one type unconditionally, and arrow's `Field::try_merge` unions two but errors on a leaf type
213+
/// difference instead of tolerating it and does not recurse into maps or fixed-size lists.
214214
pub fn widen_nested_nullability(base: &DataType, other: &DataType) -> DataType {
215215
match (base, other) {
216216
(DataType::List(b), DataType::List(o)) => DataType::List(widen_field(b, o)),

‎native/core/src/execution/planner.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -131,9 +131,9 @@ pub type PlanCreationResult =
131131
/// the key field non-nullable and every nested key/value type byte-for-byte unchanged. Any non-`Map`
132132
/// type is returned unchanged.
133133
///
134-
/// This is the shallow counterpart of `make_all_fields_nullable`, used for `map_entries`.
135-
/// `map_entries` reuses the input map's entries array as its output list values but declares that
136-
/// element's `value` field nullable (`Struct(key non-null, value nullable)`), deriving both nested
134+
/// This is the shallow counterpart of `make_all_fields_nullable` (in `planner/aggregate.rs`), used
135+
/// for `map_entries`. `map_entries` reuses the input map's entries array as its output list values
136+
/// but declares that element's `value` field nullable (`Struct(key non-null, value nullable)`), deriving both nested
137137
/// types verbatim from the input. Arrow's `GenericListArray::try_new` compares the declared element
138138
/// field's type against the reused values array's type in full, so the ONLY field that can mismatch
139139
/// is the entries `value` field's own `nullable` flag; widening just that field is sufficient.

0 commit comments

Comments
 (0)