Repository navigation
Commit 68a0148
fix: RepartitionExec deadlock when its spill pool holds two open files (apache#24891)
Closes apache#24883
Under a memory limit, a grouped aggregation on two or more partitions can
stop and never complete: when `RepartitionExec` cannot reserve memory for
a batch it writes the batch to its spill pool and sends a marker, and the
reader then blocks on the pool until it gets a batch. When two input
tasks spill at the same time the pool holds two open files. The reader
read only the oldest file and, when that file had no unread batch, waited
for it, while the batch it needed was in the newer file. Once every
channel had data the gate closed and the writers waited too, so nothing
could wake the reader.
`SpillPoolReader` now keeps every file it received (oldest first) and
returns the first available batch. A file that has no unread batch is
skipped; the reader still waits on the oldest file that has unread
batches whose read is not complete, so a single writer keeps FIFO order.
The writer side is unchanged.
Also backported:
- `test: reproduce the RepartitionExec spill pool deadlock with a SQL
query` (`datafusion/core/tests/memory_limit/repartition_mem_limit.rs`),
which sums the `spill_count` metric and fails if no attempt spilled.
- `test: add a scenario fuzzer for the spill pool contract`, including
`test_read_error_is_reported_to_the_reader`, a test that a second
writer's batch is readable while the first writer is held, and
`spill_pool_scenario_fuzz` (default 50 seeds).
- the extended-CI job sets `DATAFUSION_SPILL_POOL_FUZZ_ITERATIONS: 1000`.
Backport notes: applied by hand because `git cherry-pick -x` is not
usable here (the worktree's git directory is not writable from this
sandbox). All three files applied byte-identically; the only
fork-specific deltas in the touched regions are pre-existing cosmetic
differences (`#[deprecated]` without `since`, `sort_unstable` vs `sort`),
which this commit does not touch.
(cherry picked from commit 09a2aff)
(backport of apache#24891, 09a2aff)
Signed-off-by: discord9 <[email protected]>
(cherry picked from commit ac60580950ee42b7358a74fdf1203a0dc5e84992)1 parent a710e34 commit 68a0148
3 files changed
Lines changed: 874 additions & 93 deletions
File tree
- .github/workflows
- datafusion
- core/tests/memory_limit
- physical-plan/src/spill
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
91 | 94 | | |
92 | 95 | | |
93 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
0 commit comments