Repository navigation
Commit 70bdc95
## Which issue does this PR close?
- Backport of #22815 to `branch-54` (for 54.1.0, tracked in #22547).
- Fixes wrong results #22796 (regression introduced by the inline-u128
hashing fast path in #21064, which is present on `branch-54` as commit
1cb4de4).
## Rationale for this change
`approx_distinct` over-counts `Utf8View` values when the same short (<=
12 byte) string appears across batches with different layouts: an
all-inline batch hashes it as a raw `u128` view, while a mixed batch
(with a data buffer) hashed it as a `&str`, so the same value is counted
twice.
## What changes are included in this PR?
This is an **adapted** backport, not a clean cherry-pick. `branch-54`
only has the scalar `StringViewHLLAccumulator` path; the group
`HllGroupsAccumulator` / `Utf8ViewHasher` code the upstream fix also
touched comes from #22768, which is **not** on `branch-54`.
- Applied verbatim from #22815: the scalar
`StringViewHLLAccumulator::update_batch` mixed-batch fix (per-row length
check) and its regression test
`utf8view_acc_split_batches_match_single_mixed_batch`.
- Omitted as not applicable: the `Utf8ViewHasher` group-accumulator hunk
and its group test.
- Added: a `#[cfg(test)] mod tests` scaffold and imports, since
`branch-54`'s `approx_distinct.rs` had no test module.
## Are these changes tested?
Yes. The scalar regression test passes with the fix and fails without it
(over-count), confirming it exercises the regression.
## Are there any user-facing changes?
`approx_distinct` on `Utf8View` / `VARCHAR VIEW` columns now returns
correct (lower) counts. Results may differ from the previously incorrect
values. No API changes.
Co-authored-by: Huaijin <[email protected]>
Co-authored-by: Yongting You <[email protected]>
1 parent 6c770b5 commit 70bdc95
1 file changed
Lines changed: 61 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
245 | 245 | | |
246 | 246 | | |
247 | 247 | | |
248 | | - | |
249 | | - | |
250 | 248 | | |
| 249 | + | |
| 250 | + | |
251 | 251 | | |
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
256 | 256 | | |
257 | | - | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
258 | 271 | | |
259 | 272 | | |
260 | 273 | | |
| |||
464 | 477 | | |
465 | 478 | | |
466 | 479 | | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
0 commit comments