Repository navigation
Commit d21b24e
committed
perf: release the shuffle decoder at EOF and fold it into the scan input state
ShuffleScanExec now keeps the input batch and its decoder in one
ScanInputState behind a single mutex, still shared by plan and stream
clones, and releases the decoder's retained zstd workspace when the
input reaches EOF instead of holding it until the plan drops.
The ipc_decode bench adds a 400-row case, zstd levels 1 and 3, an LZ4
control, a per-reader-lifetime variant that counts context startup and
cleanup, and a mutex-locked variant mirroring the operator's lock.
The decode-context reuse test now decodes its reference with a new
ShuffleDecodeContext per frame; read_ipc_compressed uses the thread-local
cached context, so the old reference was itself a reused context.1 parent 3fb42e1 commit d21b24e
4 files changed
Lines changed: 141 additions & 43 deletions
File tree
- native
- core/src/execution/operators
- shuffle
- benches
- src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
| 65 | + | |
| 66 | + | |
71 | 67 | | |
72 | 68 | | |
73 | 69 | | |
| |||
113 | 109 | | |
114 | 110 | | |
115 | 111 | | |
116 | | - | |
117 | | - | |
| 112 | + | |
118 | 113 | | |
119 | 114 | | |
120 | 115 | | |
| |||
126 | 121 | | |
127 | 122 | | |
128 | 123 | | |
129 | | - | |
| 124 | + | |
130 | 125 | | |
131 | 126 | | |
132 | 127 | | |
| |||
138 | 133 | | |
139 | 134 | | |
140 | 135 | | |
141 | | - | |
142 | | - | |
| 136 | + | |
| 137 | + | |
143 | 138 | | |
144 | 139 | | |
145 | 140 | | |
146 | 141 | | |
147 | | - | |
| 142 | + | |
148 | 143 | | |
149 | 144 | | |
150 | 145 | | |
151 | | - | |
| 146 | + | |
152 | 147 | | |
153 | 148 | | |
154 | 149 | | |
| |||
238 | 233 | | |
239 | 234 | | |
240 | 235 | | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
241 | 259 | | |
242 | 260 | | |
243 | 261 | | |
| |||
370 | 388 | | |
371 | 389 | | |
372 | 390 | | |
373 | | - | |
| 391 | + | |
374 | 392 | | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
| 393 | + | |
379 | 394 | | |
380 | 395 | | |
381 | 396 | | |
| |||
398 | 413 | | |
399 | 414 | | |
400 | 415 | | |
401 | | - | |
| 416 | + | |
402 | 417 | | |
403 | 418 | | |
404 | 419 | | |
| |||
428 | 443 | | |
429 | 444 | | |
430 | 445 | | |
431 | | - | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
432 | 451 | | |
433 | 452 | | |
434 | 453 | | |
| |||
503 | 522 | | |
504 | 523 | | |
505 | 524 | | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
506 | 554 | | |
507 | 555 | | |
508 | 556 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | | - | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
34 | 38 | | |
35 | 39 | | |
36 | 40 | | |
37 | 41 | | |
38 | 42 | | |
39 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
40 | 47 | | |
41 | 48 | | |
42 | 49 | | |
| |||
81 | 88 | | |
82 | 89 | | |
83 | 90 | | |
84 | | - | |
| 91 | + | |
85 | 92 | | |
86 | 93 | | |
87 | | - | |
| 94 | + | |
88 | 95 | | |
89 | 96 | | |
90 | 97 | | |
91 | 98 | | |
92 | 99 | | |
93 | | - | |
94 | | - | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
95 | 121 | | |
96 | 122 | | |
97 | | - | |
| 123 | + | |
98 | 124 | | |
99 | 125 | | |
100 | 126 | | |
101 | 127 | | |
102 | | - | |
| 128 | + | |
103 | 129 | | |
104 | 130 | | |
105 | 131 | | |
| |||
112 | 138 | | |
113 | 139 | | |
114 | 140 | | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | 141 | | |
120 | 142 | | |
121 | 143 | | |
| |||
133 | 155 | | |
134 | 156 | | |
135 | 157 | | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
142 | 164 | | |
143 | 165 | | |
144 | 166 | | |
145 | 167 | | |
146 | 168 | | |
| 169 | + | |
147 | 170 | | |
148 | 171 | | |
149 | 172 | | |
| |||
152 | 175 | | |
153 | 176 | | |
154 | 177 | | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
155 | 199 | | |
156 | 200 | | |
157 | 201 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
179 | 179 | | |
180 | 180 | | |
181 | 181 | | |
182 | | - | |
183 | | - | |
184 | | - | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
185 | 185 | | |
186 | 186 | | |
187 | 187 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
320 | 320 | | |
321 | 321 | | |
322 | 322 | | |
323 | | - | |
| 323 | + | |
324 | 324 | | |
325 | 325 | | |
326 | 326 | | |
327 | 327 | | |
328 | 328 | | |
329 | 329 | | |
330 | 330 | | |
331 | | - | |
| 331 | + | |
| 332 | + | |
332 | 333 | | |
333 | 334 | | |
334 | 335 | | |
335 | 336 | | |
336 | 337 | | |
337 | 338 | | |
338 | 339 | | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
339 | 345 | | |
340 | 346 | | |
341 | 347 | | |
| |||
0 commit comments