Repository navigation
Commit f7a7acb
committed
fix(physical-plan): honor distinct soft limits in SingleHashAggregateStream
## Which issue does this PR close?
- Closes [apache#24980](apache#24980)
## Rationale for this change
`SingleHashAggregateStream` ignores the distinct soft limit pushed into
`AggregateExec`. As a result, single-stage `SELECT DISTINCT ... LIMIT n`
queries consume all input even after enough distinct groups have been
collected.
This change stops input consumption once the in-memory hash table
contains at least the requested number of distinct groups. Any existing
spills are merged before producing output, and the downstream limit
operator enforces the exact output row count.
## What changes are included in this PR?
* Add a distinct soft-limit check to `SingleHashAggregateStream` after
processing each input batch.
* Reuse the input-exhausted transition when the soft limit is reached,
preserving spill merging and output preparation.
* Add unit tests covering early termination with and without spilling.
- Update the `AggregateExec::limit_options` documentation to list
`SingleHash` as supporting distinct soft limits.
## Are these changes tested?
Added unit tests covering:
* Reaching the soft limit without spilling.
* Reaching the soft limit after spilling, preserving spilled groups and
deduplicating overlapping groups.
* Rejecting further input consumption after the soft limit is reached in
the spill case.
Following test commands have been executed and passed
- `cargo test --profile=ci --test sqllogictests`
- `cargo test -p datafusion`
- `cargo test -p datafusion-cli`
## Are there any user-facing changes?
Single-stage `DISTINCT` queries with a limit can stop consuming input
earlier, reducing unnecessary work. Query semantics are unchanged.1 parent 5747e87 commit f7a7acb
2 files changed
Lines changed: 267 additions & 25 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
868 | 868 | | |
869 | 869 | | |
870 | 870 | | |
871 | | - | |
872 | | - | |
| 871 | + | |
| 872 | + | |
873 | 873 | | |
874 | 874 | | |
875 | 875 | | |
| |||
4424 | 4424 | | |
4425 | 4425 | | |
4426 | 4426 | | |
| 4427 | + | |
| 4428 | + | |
| 4429 | + | |
| 4430 | + | |
| 4431 | + | |
| 4432 | + | |
| 4433 | + | |
| 4434 | + | |
| 4435 | + | |
| 4436 | + | |
| 4437 | + | |
| 4438 | + | |
| 4439 | + | |
| 4440 | + | |
| 4441 | + | |
| 4442 | + | |
| 4443 | + | |
| 4444 | + | |
| 4445 | + | |
| 4446 | + | |
| 4447 | + | |
| 4448 | + | |
| 4449 | + | |
| 4450 | + | |
| 4451 | + | |
| 4452 | + | |
| 4453 | + | |
| 4454 | + | |
| 4455 | + | |
| 4456 | + | |
| 4457 | + | |
| 4458 | + | |
| 4459 | + | |
| 4460 | + | |
| 4461 | + | |
| 4462 | + | |
| 4463 | + | |
| 4464 | + | |
| 4465 | + | |
| 4466 | + | |
| 4467 | + | |
| 4468 | + | |
| 4469 | + | |
| 4470 | + | |
| 4471 | + | |
| 4472 | + | |
| 4473 | + | |
| 4474 | + | |
| 4475 | + | |
| 4476 | + | |
| 4477 | + | |
| 4478 | + | |
| 4479 | + | |
| 4480 | + | |
| 4481 | + | |
| 4482 | + | |
| 4483 | + | |
| 4484 | + | |
| 4485 | + | |
| 4486 | + | |
| 4487 | + | |
| 4488 | + | |
| 4489 | + | |
| 4490 | + | |
| 4491 | + | |
| 4492 | + | |
| 4493 | + | |
| 4494 | + | |
| 4495 | + | |
| 4496 | + | |
| 4497 | + | |
| 4498 | + | |
| 4499 | + | |
| 4500 | + | |
| 4501 | + | |
| 4502 | + | |
| 4503 | + | |
| 4504 | + | |
| 4505 | + | |
| 4506 | + | |
| 4507 | + | |
| 4508 | + | |
| 4509 | + | |
| 4510 | + | |
| 4511 | + | |
| 4512 | + | |
| 4513 | + | |
| 4514 | + | |
| 4515 | + | |
| 4516 | + | |
| 4517 | + | |
| 4518 | + | |
| 4519 | + | |
| 4520 | + | |
| 4521 | + | |
| 4522 | + | |
| 4523 | + | |
| 4524 | + | |
| 4525 | + | |
| 4526 | + | |
| 4527 | + | |
| 4528 | + | |
| 4529 | + | |
| 4530 | + | |
| 4531 | + | |
| 4532 | + | |
| 4533 | + | |
| 4534 | + | |
| 4535 | + | |
| 4536 | + | |
| 4537 | + | |
| 4538 | + | |
| 4539 | + | |
| 4540 | + | |
| 4541 | + | |
| 4542 | + | |
| 4543 | + | |
| 4544 | + | |
| 4545 | + | |
| 4546 | + | |
| 4547 | + | |
| 4548 | + | |
| 4549 | + | |
| 4550 | + | |
| 4551 | + | |
| 4552 | + | |
| 4553 | + | |
| 4554 | + | |
| 4555 | + | |
| 4556 | + | |
| 4557 | + | |
| 4558 | + | |
| 4559 | + | |
| 4560 | + | |
| 4561 | + | |
| 4562 | + | |
| 4563 | + | |
| 4564 | + | |
| 4565 | + | |
| 4566 | + | |
| 4567 | + | |
| 4568 | + | |
| 4569 | + | |
| 4570 | + | |
| 4571 | + | |
| 4572 | + | |
| 4573 | + | |
| 4574 | + | |
| 4575 | + | |
| 4576 | + | |
| 4577 | + | |
| 4578 | + | |
| 4579 | + | |
| 4580 | + | |
| 4581 | + | |
| 4582 | + | |
| 4583 | + | |
| 4584 | + | |
| 4585 | + | |
| 4586 | + | |
| 4587 | + | |
| 4588 | + | |
| 4589 | + | |
| 4590 | + | |
| 4591 | + | |
| 4592 | + | |
| 4593 | + | |
| 4594 | + | |
| 4595 | + | |
| 4596 | + | |
| 4597 | + | |
| 4598 | + | |
| 4599 | + | |
| 4600 | + | |
| 4601 | + | |
| 4602 | + | |
| 4603 | + | |
| 4604 | + | |
| 4605 | + | |
| 4606 | + | |
| 4607 | + | |
| 4608 | + | |
| 4609 | + | |
| 4610 | + | |
| 4611 | + | |
| 4612 | + | |
| 4613 | + | |
| 4614 | + | |
| 4615 | + | |
| 4616 | + | |
| 4617 | + | |
| 4618 | + | |
| 4619 | + | |
4427 | 4620 | | |
4428 | 4621 | | |
4429 | 4622 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
107 | 111 | | |
108 | 112 | | |
109 | 113 | | |
| |||
374 | 378 | | |
375 | 379 | | |
376 | 380 | | |
| 381 | + | |
377 | 382 | | |
378 | 383 | | |
379 | 384 | | |
| |||
449 | 454 | | |
450 | 455 | | |
451 | 456 | | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
452 | 478 | | |
453 | 479 | | |
454 | 480 | | |
| |||
490 | 516 | | |
491 | 517 | | |
492 | 518 | | |
493 | | - | |
494 | | - | |
495 | | - | |
496 | | - | |
497 | | - | |
498 | | - | |
499 | | - | |
500 | | - | |
501 | | - | |
502 | | - | |
503 | | - | |
504 | | - | |
505 | | - | |
506 | | - | |
507 | | - | |
508 | | - | |
509 | | - | |
510 | | - | |
511 | | - | |
512 | | - | |
513 | | - | |
514 | | - | |
515 | | - | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
516 | 564 | | |
| 565 | + | |
517 | 566 | | |
518 | 567 | | |
519 | 568 | | |
| |||
0 commit comments