Repository navigation
Commit 2d1aab3
authored
fix: size JVM shuffle pointer array growth from the array, not the data pages (apache#5907)
CometShuffleExternalSorter.growPointerArrayIfNecessary sized the replacement
pointer array from SpillSorter.getMemoryUsage(), which includes every data
page, so the first growth requested an array proportional to the page size
instead of doubling the array as Spark's ShuffleExternalSorter does. Add
SpillSorter.getPointerArrayMemoryUsage() and use it for growth.1 parent 9b63e7d commit 2d1aab3
3 files changed
Lines changed: 66 additions & 1 deletion
File tree
- spark/src
- main/java/org/apache/spark/shuffle/sort
- test/scala/org/apache/spark/shuffle/sort
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
254 | 254 | | |
255 | 255 | | |
256 | 256 | | |
257 | | - | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
258 | 261 | | |
259 | 262 | | |
260 | 263 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
183 | 198 | | |
184 | 199 | | |
185 | 200 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
259 | 259 | | |
260 | 260 | | |
261 | 261 | | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
262 | 309 | | |
0 commit comments