Repository navigation
Commit b464997
committed
fix: remove the spill replay pool now that DataFusion 56 has #25383
DataFusion 56 includes apache/datafusion#25383, which leaves room for a
final aggregate to read its spill files back, so SpillReplayPool is no
longer needed. It is also now harmful: #25383 holds replay headroom on a
sibling reservation while the spill merge picks its files, so the pool
took every refused read-buffer request for a replay and granted it. The
merge then opened every spill file and starved the next aggregate, which
failed the #6880 PartialMerge test with a 2 MiB pool.
The #6254 and #6857 read-back tests, which the pool was added for, pass
without it. The sort-merge join #6880 test now uses an 8 MiB pool,
because with DataFusion 56 its PartialMerge fits in 12 MiB or more
without spilling.
Closes #6583.1 parent 9156480 commit b464997
6 files changed
Lines changed: 18 additions & 428 deletions
File tree
- docs/source/contributor-guide
- native/core/src/execution
- memory_pools
- planner
- spark/src/test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
300 | 300 | | |
301 | 301 | | |
302 | 302 | | |
303 | | - | |
304 | | - | |
305 | | - | |
| 303 | + | |
| 304 | + | |
306 | 305 | | |
307 | 306 | | |
308 | 307 | | |
309 | | - | |
310 | | - | |
311 | | - | |
| 308 | + | |
312 | 309 | | |
313 | 310 | | |
314 | 311 | | |
| |||
463 | 460 | | |
464 | 461 | | |
465 | 462 | | |
466 | | - | |
467 | | - | |
468 | | - | |
469 | | - | |
470 | | - | |
471 | | - | |
472 | | - | |
473 | | - | |
474 | | - | |
475 | | - | |
476 | | - | |
477 | | - | |
478 | 463 | | |
479 | 464 | | |
480 | 465 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | 24 | | |
26 | 25 | | |
27 | 26 | | |
| |||
154 | 153 | | |
155 | 154 | | |
156 | 155 | | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
157 | 160 | | |
158 | 161 | | |
159 | | - | |
| 162 | + | |
160 | 163 | | |
161 | 164 | | |
162 | 165 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | 23 | | |
25 | 24 | | |
26 | 25 | | |
| |||
30 | 29 | | |
31 | 30 | | |
32 | 31 | | |
33 | | - | |
34 | 32 | | |
35 | 33 | | |
36 | 34 | | |
| |||
94 | 92 | | |
95 | 93 | | |
96 | 94 | | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
| 95 | + | |
101 | 96 | | |
102 | 97 | | |
103 | 98 | | |
104 | 99 | | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
113 | 105 | | |
114 | 106 | | |
115 | 107 | | |
| |||
123 | 115 | | |
124 | 116 | | |
125 | 117 | | |
126 | | - | |
127 | 118 | | |
128 | 119 | | |
129 | 120 | | |
| |||
0 commit comments