Repository navigation
Commit 6700d02
authored
test: bound merge fan-in in the final hash aggregate spill test (apache#25424)
## Which issue does this PR close?
- Closes apache#25423.
## Rationale for this change
Case G of `aggregate_memory_spill.slt` runs four final partitions in a 1
MiB greedy pool. A partition that replays its spills cannot spill again.
Its replay table fails to grow when its own merge buffers and the state
of the other partitions fill the pool. CI hit this on two merge-queue
runs (see the issue).
## What changes are included in this PR?
- `aggregate_memory_spill.slt`: set
`datafusion.runtime.max_spill_merge_fan_in = 2` for Case G and reset it
at the end, as apache#25252 did for `ordered_aggregate_spill.slt`. The memory
limit stays at 1 MiB. Cases A to F are unchanged.
- `FinalHashAggregateStream::new_with_input` builds the stream over a
given input, like `OrderedFinalAggregateStream::new_with_input`. `new`
calls it.
- Tests in `hash_stream.rs` spill and replay partition 0 while
partitions 1 to 3 keep their state in a shared 1 MiB greedy pool.
Channel inputs return `Pending` after each batch, so the interleaving
does not depend on task scheduling. Input sizes follow the observed
reservations: the held partitions read batches until they reserve 512
KiB, and partition 0 reads batches until it has spilled the requested
number of times. So the cases do not depend on the memory accounting of
a platform or feature set, such as `force_hash_collisions`. The tests
check the grouped results against a reference, the spill metrics, and
that the pool returns to the held size after completion, after a drop
during replay, and after an input error, and to zero after all streams
drop. These tests cover the spill and replay lifecycle; the SLT case
covers the fan-in regression.
## What is the testing strategy for this PR?
Local runs on this branch:
- The SLT file 480 times as 16 concurrent copies, each in its own
working directory. Before the change: 12 failures, all in Case G with
the `ResourcesExhausted` error from the issue. After the change: 0
failures.
- The new tests 50 times with default features and 20 times with
`force_hash_collisions` enabled in `datafusion-common`, once with
`--profile ci` and that feature as the extended CI job runs them, and
all `datafusion-physical-plan` unit tests.
- `cargo test -p datafusion-sqllogictest --test sqllogictests` for all
files.
- `cargo fmt --all -- --check` and `cargo clippy --all-targets
--all-features -- -D warnings`.
## Are there any user-facing changes?
No.1 parent 1cfc355 commit 6700d02
2 files changed
Lines changed: 306 additions & 3 deletions
File tree
- datafusion
- physical-plan/src/aggregates
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
736 | 736 | | |
737 | 737 | | |
738 | 738 | | |
739 | | - | |
740 | 739 | | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
741 | 751 | | |
742 | 752 | | |
743 | 753 | | |
| |||
1022 | 1032 | | |
1023 | 1033 | | |
1024 | 1034 | | |
| 1035 | + | |
1025 | 1036 | | |
1026 | 1037 | | |
1027 | 1038 | | |
1028 | 1039 | | |
1029 | | - | |
| 1040 | + | |
1030 | 1041 | | |
1031 | 1042 | | |
| 1043 | + | |
| 1044 | + | |
1032 | 1045 | | |
1033 | 1046 | | |
| 1047 | + | |
1034 | 1048 | | |
1035 | 1049 | | |
1036 | | - | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
1037 | 1053 | | |
1038 | 1054 | | |
1039 | 1055 | | |
| |||
1607 | 1623 | | |
1608 | 1624 | | |
1609 | 1625 | | |
| 1626 | + | |
| 1627 | + | |
| 1628 | + | |
| 1629 | + | |
| 1630 | + | |
| 1631 | + | |
| 1632 | + | |
| 1633 | + | |
| 1634 | + | |
| 1635 | + | |
| 1636 | + | |
| 1637 | + | |
| 1638 | + | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
| 1691 | + | |
| 1692 | + | |
| 1693 | + | |
| 1694 | + | |
| 1695 | + | |
| 1696 | + | |
| 1697 | + | |
| 1698 | + | |
| 1699 | + | |
| 1700 | + | |
| 1701 | + | |
| 1702 | + | |
| 1703 | + | |
| 1704 | + | |
| 1705 | + | |
| 1706 | + | |
| 1707 | + | |
| 1708 | + | |
| 1709 | + | |
| 1710 | + | |
| 1711 | + | |
| 1712 | + | |
| 1713 | + | |
| 1714 | + | |
| 1715 | + | |
| 1716 | + | |
| 1717 | + | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
| 1721 | + | |
| 1722 | + | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
| 1726 | + | |
| 1727 | + | |
| 1728 | + | |
| 1729 | + | |
| 1730 | + | |
| 1731 | + | |
| 1732 | + | |
| 1733 | + | |
| 1734 | + | |
| 1735 | + | |
| 1736 | + | |
| 1737 | + | |
| 1738 | + | |
| 1739 | + | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
| 1746 | + | |
| 1747 | + | |
| 1748 | + | |
| 1749 | + | |
| 1750 | + | |
| 1751 | + | |
| 1752 | + | |
| 1753 | + | |
| 1754 | + | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
| 1759 | + | |
| 1760 | + | |
| 1761 | + | |
| 1762 | + | |
| 1763 | + | |
| 1764 | + | |
| 1765 | + | |
| 1766 | + | |
| 1767 | + | |
| 1768 | + | |
| 1769 | + | |
| 1770 | + | |
| 1771 | + | |
| 1772 | + | |
| 1773 | + | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
| 1779 | + | |
| 1780 | + | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
| 1799 | + | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
| 1814 | + | |
| 1815 | + | |
| 1816 | + | |
| 1817 | + | |
| 1818 | + | |
| 1819 | + | |
| 1820 | + | |
| 1821 | + | |
| 1822 | + | |
| 1823 | + | |
| 1824 | + | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
| 1829 | + | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
| 1877 | + | |
| 1878 | + | |
| 1879 | + | |
| 1880 | + | |
| 1881 | + | |
| 1882 | + | |
| 1883 | + | |
| 1884 | + | |
| 1885 | + | |
| 1886 | + | |
| 1887 | + | |
| 1888 | + | |
| 1889 | + | |
| 1890 | + | |
| 1891 | + | |
| 1892 | + | |
| 1893 | + | |
| 1894 | + | |
| 1895 | + | |
| 1896 | + | |
| 1897 | + | |
| 1898 | + | |
| 1899 | + | |
| 1900 | + | |
| 1901 | + | |
| 1902 | + | |
1610 | 1903 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
201 | 208 | | |
202 | 209 | | |
203 | 210 | | |
| |||
230 | 237 | | |
231 | 238 | | |
232 | 239 | | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
233 | 243 | | |
234 | 244 | | |
235 | 245 | | |
| |||
0 commit comments