Repository navigation
Commit ba7c892
authored
fix: let a spilled final aggregate read its spill files back past its memory share (apache#6544)
* fix: let a spilled final aggregate read its spill files back past its memory share
DataFusion 55's FinalHashAggregateStream replays its merged spill files through a stream that
can't spill, so a refused memory request there fails the task. The merge's read buffers are a
sibling reservation of the same consumer and take as many spill files as fit, so the replay often
finds the consumer's share already taken (apache#6254).
Both Comet pools now record that request as overcommit instead of refusing it. They recognize it
as a request from a FinalHashAggregateStream consumer while another of its reservations holds
memory, which in DataFusion 55.1 happens only during the replay. The replay emits its finished
groups after every batch, so the overcommit stays around one batch of groups, and releases repay
it first. Refusals while the aggregate reads its input, and while the merge picks its files, are
unchanged.
Remove this once Comet's DataFusion includes apache/datafusion#25383.
* fix: let an ordered final aggregate read its spill files back too
DataFusion runs a final aggregate whose input is sorted on some of its
grouping keys as an OrderedFinalAggregateStream. It merges and replays its
spill files the same way FinalHashAggregateStream does, so its replay
failed the task the same way. Treat its consumer as a final aggregate too.
Explain in spill_replay.rs why recording the replay's request is safe: the
replay asks for memory only after it has aggregated a batch, so the memory
already exists, as it does for a grow. Describe the exception in the memory
management guide, which said the fair pool always refuses a request that
fails its local checks.
* docs: tighten the spill replay comments and guide
The fair pool is the only one with a share and a pool total to skip, so say
so. The greedy pool takes its tracking lock for other consumers when Spark
refuses them, not never. Point whoever removes the workaround at the tests
that show whether the replay still needs it.
* test: share the spill replay tests between the pools
fair_pool.rs and unified_pool.rs each had a copy of the same two spill
replay scenarios. Run them once, in spill_replay.rs, against both pool
types as createPlan builds them, which also checks that the wrappers pass
register through to the greedy pool's tracking. Keep a greedy pool test for
its per-consumer map, and drop a test import that the pool's own imports
now cover.
Share the final aggregate spill checks between the two apache#6254 tests in
CometAggregateSuite, and use checkCometAnswer, which collects once and
labels the Comet answer as Comet's.
* refactor: move the spill replay workaround into a pool wrapper
The workaround for apache#6254 lived in both Comet pools. The fair pool sent its
three refusals through a check, and the greedy pool tracked each final
aggregate across its reservations. apache#5613 reworks the fair pool's try_grow,
and apache#6583 has to take the workaround out again, so both would have had to
rework the pools.
SpillReplayPool, in spill_replay.rs, now wraps the tracked Comet pool
instead. It keeps a total for each final aggregate's consumer. When the pool
refuses a request from one while another of its reservations holds memory,
it calls the pool's grow, which skips the fair pool's limits and carries
what Spark doesn't grant as overcommit. fair_pool.rs and unified_pool.rs are
back to main's versions, and overcommit() looks through the wrapper.
The Rust tests run against both pool types as createPlan builds them. They
now reach the fair pool's pool-limit refusal too, and check that a failed
Spark call during the replay is returned rather than recorded.
The guide adds the wrapper to the pool stack and keeps a short section. The
DataFusion 55.1 facts that an upgrade has to re-check stay in
spill_replay.rs, which names apache#6583 next to the removal condition.1 parent bd7cfcb commit ba7c892
4 files changed
Lines changed: 490 additions & 9 deletions
File tree
- docs/source/contributor-guide
- native/core/src/execution/memory_pools
- spark/src/test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
260 | 261 | | |
261 | 262 | | |
262 | 263 | | |
263 | | - | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
264 | 267 | | |
265 | 268 | | |
266 | 269 | | |
| |||
346 | 349 | | |
347 | 350 | | |
348 | 351 | | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
349 | 364 | | |
350 | 365 | | |
351 | 366 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
| |||
70 | 72 | | |
71 | 73 | | |
72 | 74 | | |
73 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
74 | 79 | | |
75 | 80 | | |
76 | | - | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
77 | 85 | | |
78 | 86 | | |
79 | 87 | | |
| |||
85 | 93 | | |
86 | 94 | | |
87 | 95 | | |
| 96 | + | |
88 | 97 | | |
89 | 98 | | |
90 | 99 | | |
| |||
0 commit comments