Repository navigation
Commit 18b1e35
fix: grouped first_value/last_value FILTER excludes NULL predicate rows (#23707)
## Which issue does this PR close?
- Closes #22666
## Rationale for this change
Under SQL aggregate `FILTER` semantics, a row passes only when the
predicate evaluates to `true`; rows where the predicate is `null` must
be excluded. Grouped `first_value` / `last_value` checked only
`BooleanArray::value(idx)`, without checking validity, so a NULL
predicate row whose underlying value bit is set (as produced by
comparison kernels, e.g. `null::int < 1`) was treated as passing:
```sql
SELECT g, first_value(a ORDER BY a) FILTER (WHERE b < 1) AS fv
FROM (VALUES (0, 10, CAST(NULL AS INT)), (0, 20, 2)) AS t(g, a, b)
GROUP BY g;
-- returned fv = 10, must return fv = NULL
-- (row 1: b < 1 is NULL; row 2: b < 1 is FALSE — no row satisfies `true`)
```
## What changes are included in this PR?
`datafusion/functions-aggregate/src/first_last.rs`, in
`FirstLastGroupsAccumulator::get_filtered_extreme_of_each_group` (shared
by `first_value` and `last_value`, and by both the `update_batch` and
`merge_batch` paths):
- `passed_filter` now requires `is_valid(idx) && value(idx)` (the
`Some(true)` semantics), matching the convention already used by
`variance.rs` / `correlation.rs`.
- The `is_set_arr` read gets the same validity check. This is *not* only
an internal bitmap: `convert_to_state` stores the user FILTER clause
(including its nulls) in the last state column, so on the merge path
(e.g. skip-partial-aggregation) NULL predicate rows were likewise
treated as set. Verified with a forced skip-partial run (100k unique
groups, all-NULL predicates): 83,616 groups were incorrectly assigned
non-NULL values before the fix, 0 after. For genuine internal bitmaps
(no nulls) the added check is trivially true, so behavior there is
unchanged.
Regression coverage:
- sqllogictest (`aggregate.slt`): the issue reproducer, the `last_value`
counterpart, mixed TRUE/FALSE/NULL predicates, all-TRUE and no-FILTER
controls, the (already correct) non-grouped path, and a window-function
no-regression case.
- Unit tests: `test_group_acc_filter_null_predicate` (update path) and
`test_group_acc_merge_null_is_set` (merge path via `convert_to_state` →
`merge_batch`), both constructing `BooleanArray`s whose null slots carry
a set value bit.
## Are these changes tested?
Yes — see above. Verified `./dev/rust_lint.sh`, `cargo test -p
datafusion-functions-aggregate --lib`, the `aggregate`/`window`
sqllogictest files, and `datafusion-cli` end-to-end (grouped
first/last_value now return NULL for the issue reproducer;
mixed-predicate and non-grouped results unchanged).
Also audited the rest of `functions-aggregate` for the same
validity-blind pattern: shared helpers (`nulls.rs::filter_to_validity`,
`accumulate.rs`, `prim_op.rs`, `count.rs`, `array_agg.rs`,
`variance.rs`, `correlation.rs`) already handle validity correctly, and
non-grouped paths pre-filter with arrow's `filter` kernel (which drops
NULL predicate rows), so no other aggregate needs changes.
## Performance
`functions-aggregate/benches/first_last.rs` was run against `main`. The
added checks are one validity-bit test per row on the grouped path;
measured deltas were within the machine's noise floor (±5% on unchanged
`filter=false` cases). A variant hoisting the null check out of the row
loop showed no measurable benefit beyond noise, so the simple idiomatic
form is kept.
## Are there any user-facing changes?
Only the bug fix: grouped `first_value`/`last_value` with a nullable
`FILTER` predicate now correctly exclude NULL-predicate rows, matching
SQL semantics and the behavior of other aggregates. No API or
configuration changes.
---------
Co-authored-by: Claude <[email protected]>1 parent f40d99a commit 18b1e35
2 files changed
Lines changed: 185 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
555 | 555 | | |
556 | 556 | | |
557 | 557 | | |
558 | | - | |
559 | | - | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
560 | 567 | | |
561 | 568 | | |
562 | 569 | | |
| |||
1415 | 1422 | | |
1416 | 1423 | | |
1417 | 1424 | | |
| 1425 | + | |
1418 | 1426 | | |
1419 | 1427 | | |
1420 | 1428 | | |
| |||
1773 | 1781 | | |
1774 | 1782 | | |
1775 | 1783 | | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
| 1799 | + | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
| 1814 | + | |
| 1815 | + | |
| 1816 | + | |
| 1817 | + | |
| 1818 | + | |
| 1819 | + | |
| 1820 | + | |
| 1821 | + | |
| 1822 | + | |
| 1823 | + | |
| 1824 | + | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
| 1829 | + | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
| 1877 | + | |
| 1878 | + | |
| 1879 | + | |
| 1880 | + | |
| 1881 | + | |
| 1882 | + | |
| 1883 | + | |
| 1884 | + | |
| 1885 | + | |
| 1886 | + | |
| 1887 | + | |
| 1888 | + | |
| 1889 | + | |
| 1890 | + | |
| 1891 | + | |
| 1892 | + | |
| 1893 | + | |
| 1894 | + | |
| 1895 | + | |
1776 | 1896 | | |
1777 | 1897 | | |
1778 | 1898 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6689 | 6689 | | |
6690 | 6690 | | |
6691 | 6691 | | |
| 6692 | + | |
| 6693 | + | |
| 6694 | + | |
| 6695 | + | |
| 6696 | + | |
| 6697 | + | |
| 6698 | + | |
| 6699 | + | |
| 6700 | + | |
| 6701 | + | |
| 6702 | + | |
| 6703 | + | |
| 6704 | + | |
| 6705 | + | |
| 6706 | + | |
| 6707 | + | |
| 6708 | + | |
| 6709 | + | |
| 6710 | + | |
| 6711 | + | |
| 6712 | + | |
| 6713 | + | |
| 6714 | + | |
| 6715 | + | |
| 6716 | + | |
| 6717 | + | |
| 6718 | + | |
| 6719 | + | |
| 6720 | + | |
| 6721 | + | |
| 6722 | + | |
| 6723 | + | |
| 6724 | + | |
| 6725 | + | |
| 6726 | + | |
| 6727 | + | |
| 6728 | + | |
| 6729 | + | |
| 6730 | + | |
| 6731 | + | |
| 6732 | + | |
| 6733 | + | |
| 6734 | + | |
| 6735 | + | |
| 6736 | + | |
| 6737 | + | |
| 6738 | + | |
| 6739 | + | |
| 6740 | + | |
| 6741 | + | |
| 6742 | + | |
| 6743 | + | |
| 6744 | + | |
| 6745 | + | |
| 6746 | + | |
| 6747 | + | |
| 6748 | + | |
| 6749 | + | |
| 6750 | + | |
| 6751 | + | |
| 6752 | + | |
| 6753 | + | |
| 6754 | + | |
6692 | 6755 | | |
6693 | 6756 | | |
6694 | 6757 | | |
| |||
0 commit comments