Repository navigation
Commit 329b267
fix: avoid buffering unbounded repartition output indefinitely (apache#24193)
## Which issue does this PR close?
- Closes apache#24044.
## Rationale for this change
`RepartitionExec` coalesces small batches on the producer side until the
configured batch size is reached or all input senders finish. An
unbounded input may never finish, so an available partial batch can be
withheld indefinitely. This prevents an incremental query from emitting
rows that are ready for downstream processing.
## What changes are included in this PR?
- Skip producer-side batch coalescing when the input is unbounded.
- Preserve the existing coalescing behavior for bounded inputs and
preserve-order execution.
- Add a regression test with an unbounded source that emits one partial
batch and then remains open.
## Are these changes tested?
Yes.
- `cargo fmt --all -- --check`
- `cargo test -p datafusion-physical-plan`
- `cargo clippy --all-targets --all-features -- -D warnings`
- The extended workspace test command required by `AGENTS.md`, including
all 503 sqllogictest files
The regression test fails on the previous implementation because the
partial batch is never emitted, and passes with this change. The
existing bounded-input coalescing test also continues to pass.
## Are there any user-facing changes?
Yes. Repartitioning an unbounded input can now emit partial batches
promptly instead of waiting indefinitely to fill the configured batch
size. This may expose smaller batches to downstream operators. There are
no public API changes.
Signed-off-by: goutamadwant <[email protected]>1 parent e8407a2 commit 329b267
1 file changed
Lines changed: 60 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
456 | 456 | | |
457 | 457 | | |
458 | 458 | | |
| 459 | + | |
459 | 460 | | |
460 | 461 | | |
461 | 462 | | |
| |||
524 | 525 | | |
525 | 526 | | |
526 | 527 | | |
527 | | - | |
528 | | - | |
529 | | - | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
530 | 532 | | |
531 | 533 | | |
532 | 534 | | |
| |||
1127 | 1129 | | |
1128 | 1130 | | |
1129 | 1131 | | |
1130 | | - | |
| 1132 | + | |
| 1133 | + | |
1131 | 1134 | | |
1132 | 1135 | | |
1133 | 1136 | | |
| |||
2239 | 2242 | | |
2240 | 2243 | | |
2241 | 2244 | | |
| 2245 | + | |
2242 | 2246 | | |
2243 | 2247 | | |
2244 | 2248 | | |
| |||
2263 | 2267 | | |
2264 | 2268 | | |
2265 | 2269 | | |
| 2270 | + | |
| 2271 | + | |
| 2272 | + | |
| 2273 | + | |
| 2274 | + | |
| 2275 | + | |
| 2276 | + | |
| 2277 | + | |
| 2278 | + | |
| 2279 | + | |
| 2280 | + | |
| 2281 | + | |
| 2282 | + | |
| 2283 | + | |
| 2284 | + | |
| 2285 | + | |
| 2286 | + | |
| 2287 | + | |
| 2288 | + | |
| 2289 | + | |
| 2290 | + | |
2266 | 2291 | | |
2267 | 2292 | | |
2268 | 2293 | | |
| |||
3007 | 3032 | | |
3008 | 3033 | | |
3009 | 3034 | | |
| 3035 | + | |
| 3036 | + | |
| 3037 | + | |
| 3038 | + | |
| 3039 | + | |
| 3040 | + | |
| 3041 | + | |
| 3042 | + | |
| 3043 | + | |
| 3044 | + | |
| 3045 | + | |
| 3046 | + | |
| 3047 | + | |
| 3048 | + | |
| 3049 | + | |
| 3050 | + | |
| 3051 | + | |
| 3052 | + | |
| 3053 | + | |
| 3054 | + | |
| 3055 | + | |
| 3056 | + | |
| 3057 | + | |
| 3058 | + | |
| 3059 | + | |
| 3060 | + | |
| 3061 | + | |
| 3062 | + | |
| 3063 | + | |
| 3064 | + | |
| 3065 | + | |
3010 | 3066 | | |
3011 | 3067 | | |
3012 | 3068 | | |
| |||
0 commit comments