Skip to content

Commit ac19b2d

Browse files
adriangbclaude
andauthored
docs: clarify what a canceled partition means in the hash join dynamic filter (apache#27087)
## Which issue does this PR close? - This PR does not close an issue. It is related to apache#21625 and follows up on apache#21666. ## Rationale for this change The partitioned hash join dynamic filter coordinator marks a partition as "canceled" when its stream is dropped before it reports its build data. A reader can assume that this means the whole query was canceled. That assumption is wrong. A parent operator can legally finish early and drop one hash join partition stream while sibling partitions keep running. In apache#21625 (the TPC-H Q18 hang), a `RightSemi` join with an empty build side for some partitions does exactly this. An editor who relies on the wrong reading could reintroduce that hang. ## What changes are included in this PR? Comment-only changes. The docs define a canceled partition by what a reader can observe: a parent operator drops one partition's stream before that stream reads its probe side. They avoid internal terms such as report delivery. - `BuildReportHandle` explains what cancellation means for the join. The other partitions and their shared probe input keep running, a `RepartitionExec` discards the rows routed to the canceled partition, and the canceled stream produces no output. The other partitions stop waiting for the canceled one. - `PartitionStatus::CanceledUnknown` explains only what it means for the dynamic filter. The partition's build keys are unknown, and the filter may keep or reject the rows routed to it, because no stream joins those rows. - `BuildReportState::Canceled` and `SharedBuildAccumulator::report_canceled_partition` point to those two docs. One em-dash in an existing comment is replaced. The last commit rewrites the earlier wording in plain terms, so reviewing the full diff is easier than reviewing commit by commit. ## What is the testing strategy for this PR? The changes are comment-only, so they add no tests. `cargo fmt` and `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings` pass. ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent e7c51c7 commit ac19b2d

2 files changed

Lines changed: 43 additions & 2 deletions

File tree

‎datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,17 @@ struct AccumulatorState {
371371
enum PartitionStatus {
372372
Pending,
373373
Reported(PartitionData),
374+
/// A parent operator dropped this partition's stream before the stream
375+
/// reported its build side. The build keys of this partition are unknown.
376+
///
377+
/// The filter does not wait for this partition. Without this status, the
378+
/// other partitions wait forever for a report that never comes.
379+
///
380+
/// The filter may keep or reject the probe rows routed to this partition.
381+
/// Both choices are correct, because the dropped stream never joins those
382+
/// rows and never produces output.
383+
///
384+
/// `BuildReportHandle` explains when a parent operator drops a stream.
374385
CanceledUnknown,
375386
}
376387

@@ -505,6 +516,11 @@ impl SharedBuildAccumulator {
505516
self.wait_for_completion().await
506517
}
507518

519+
/// Tells the filter that `partition_id` never reports its build side, so the
520+
/// other partitions stop waiting for it. See
521+
/// [`PartitionStatus::CanceledUnknown`].
522+
///
523+
/// The call does nothing if the partition has already reported.
508524
pub(crate) fn report_canceled_partition(&self, partition_id: usize) {
509525
let finalize_input = {
510526
let mut guard = self.inner.lock();

‎datafusion/physical-plan/src/joins/hash_join/stream.rs‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -245,18 +245,43 @@ pub(super) struct EmitUnmatchedBuildRowsState {
245245
/// `Scheduled` means the reporting `OnceFut` has been constructed but is lazy:
246246
/// the coordinator has not necessarily observed the report. Only `Delivered`
247247
/// guarantees the coordinator saw it, so `Drop` must still cancel a `Scheduled`
248-
/// partition — otherwise sibling partitions can wait forever for a report that
248+
/// partition. Otherwise sibling partitions can wait forever for a report that
249249
/// never runs.
250250
#[derive(Debug, PartialEq, Eq)]
251251
enum BuildReportState {
252252
NotReported,
253253
Scheduled,
254254
Delivered,
255+
/// The stream was dropped before it started to probe. See
256+
/// [`BuildReportHandle`].
255257
Canceled,
256258
Finalized,
257259
}
258260

259-
/// Owns the stream-side lifecycle for one partition's build-data report.
261+
/// Tracks whether this stream has reported its build side to the shared
262+
/// dynamic filter. If the stream is dropped first, the handle cancels the
263+
/// partition.
264+
///
265+
/// # Canceled partitions
266+
///
267+
/// In `Partitioned` mode, each partition of the join runs as its own stream.
268+
/// A parent operator can drop one of these streams early, because it needs no
269+
/// output from that partition. For example, a parent join whose own build
270+
/// side is empty for a partition drops this join's stream for that partition
271+
/// without polling it (<https://github.com/apache/datafusion/issues/21625>).
272+
///
273+
/// A partition is canceled when its stream is dropped before the stream
274+
/// starts to read its probe side. Cancellation affects that one partition:
275+
///
276+
/// - The other partitions of this join keep running. The query is not
277+
/// canceled.
278+
/// - The probe input keeps running, because the other partitions read from
279+
/// it. A `RepartitionExec` in that input discards the rows that it routes
280+
/// to the canceled partition.
281+
/// - The canceled stream has produced no output and never will.
282+
///
283+
/// The other partitions wait for the build side of every partition before
284+
/// they probe. The handle tells them not to wait for the canceled partition.
260285
struct BuildReportHandle {
261286
partition: usize,
262287
mode: PartitionMode,

0 commit comments

Comments
 (0)