Repository navigation
Commit 43f567a
authored
fix: account for hash-join row-index storage (apache#25508)
## Which issue does this PR close?
Addresses the shared accounting bug identified while reviewing
[apache#25491](apache#25491 (comment)).
Split out so it can be reviewed and backported independently of
prepared-build reuse.
## Rationale for this change
Hash joins reserve memory for their hash buckets, but omit the row-index
vector used to link build rows with the same key. As a result, a join
can allocate beyond its admitted budget and underreport its build
memory. For example, 100 million build rows need an additional 400 MB
for this vector that the memory pool currently does not see.
## What changes are included in this PR?
The shared hash-map constructor now reserves the bucket estimate and
row-index storage together before allocating either, and includes both
in the build-memory metric. This covers ordinary CollectLeft and
partitioned joins, using four bytes per row for the u32 representation
and eight for u64. Perfect-hash array maps keep their existing
accounting.
## What is the testing strategy for this PR?
A regression gives the constructor one byte less than the complete
budget and checks that it refuses the allocation without charging the
pool or metric. It also verifies success at the exact limit and release
on drop. The test fails against the original constructor and passes with
this fix.
All 533 hash-join tests, full workspace Clippy with all targets/features
and warnings denied, formatting, and whitespace checks pass on this
branch. The combined prepared-build branch also passed the extended
workspace run with this constructor fix (11,995 Rust tests and 521 SQL
test files; eight tests ignored).
Local checks used Rust 1.98.1 and temporary source overrides for the
requested public releases of Arrow 60.0.0, object_store 0.14.2,
sqlparser 0.63.0, and compression-codecs 0.4.42 because the registry
mirror lacked them. No dependency or lockfile changes are included.
Ordinary-path benchmark comparisons for the combined branch, including
the accounting fix, are reported in apache#25491; they are not an isolated
performance claim for this fix.
## Are there any user-facing changes?
Hash joins using these maps account for their row-index storage in
memory limits and build-memory metrics. A budget that previously
succeeded only because this allocation was omitted can now be rejected
before allocation.1 parent 0e3c398 commit 43f567a
1 file changed
Lines changed: 48 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2797 | 2797 | | |
2798 | 2798 | | |
2799 | 2799 | | |
2800 | | - | |
| 2800 | + | |
| 2801 | + | |
| 2802 | + | |
| 2803 | + | |
| 2804 | + | |
| 2805 | + | |
| 2806 | + | |
| 2807 | + | |
2801 | 2808 | | |
2802 | 2809 | | |
2803 | 2810 | | |
2804 | 2811 | | |
2805 | 2812 | | |
2806 | | - | |
| 2813 | + | |
| 2814 | + | |
| 2815 | + | |
| 2816 | + | |
| 2817 | + | |
| 2818 | + | |
| 2819 | + | |
| 2820 | + | |
2807 | 2821 | | |
2808 | 2822 | | |
2809 | 2823 | | |
| |||
3218 | 3232 | | |
3219 | 3233 | | |
3220 | 3234 | | |
| 3235 | + | |
| 3236 | + | |
| 3237 | + | |
| 3238 | + | |
| 3239 | + | |
| 3240 | + | |
| 3241 | + | |
| 3242 | + | |
| 3243 | + | |
| 3244 | + | |
| 3245 | + | |
| 3246 | + | |
| 3247 | + | |
| 3248 | + | |
| 3249 | + | |
| 3250 | + | |
| 3251 | + | |
| 3252 | + | |
| 3253 | + | |
| 3254 | + | |
| 3255 | + | |
| 3256 | + | |
| 3257 | + | |
| 3258 | + | |
| 3259 | + | |
| 3260 | + | |
| 3261 | + | |
| 3262 | + | |
| 3263 | + | |
| 3264 | + | |
| 3265 | + | |
| 3266 | + | |
3221 | 3267 | | |
3222 | 3268 | | |
3223 | 3269 | | |
| |||
0 commit comments