You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8d54633
Browse filesBrowse the repository at this point in the historyBrowse files
fix: drain PiecewiseMergeJoin output before state transitions (apache#24298)
## Which issue does this PR close?
- Closesapache#24297
## Rationale for this change
`ClassicPWMJStream` transitions phases while its output `BatchCoalescer`
still holds completed batches. The Left/Full unmatched pass falls back
into its producer when the drained queue is an exact multiple of
`batch_size` — emitting duplicate rows forever — and queued unmatched
Right/Full batches are dropped at terminal transitions. A zero-row
placeholder batch also escapes as stream output on empty results.
## What changes are included in this PR?
- Add `BatchProcessState::next_drained_batch()` (`finish_buffered_batch`
+ `next_completed_batch`); `None` guarantees the coalescer is empty
- `process_stream_batch` and `process_unmatched_buffered_batch` drain
one batch per poll and transition only after a confirmed-empty pop, so a
finished producer can never re-run and no queued batch is lost
- Scan completion returns `StatefulStreamResult::Continue` instead of
the empty placeholder batch; empty joins now yield no batches
## Are these changes tested?
Yes. The existing unmatched Left/Right tests now run with `batch_size =
1` (the Left one bounded with `take(6)` so the old duplicate loop fails
as a snapshot mismatch instead of hanging) — both fail on main and pass
here. The empty-join tests assert no batches are emitted.
## Are there any user-facing changes?
No (PWMJ is experimental and off by default).
0 commit comments