Repository navigation
Commit 08ec7be
committed
refactor: borrow shuffle write scratch per call and cap retained capacity
The write scratch is now threaded through write and flush as a borrow
instead of moving through the constructor, so one convention covers all
task-scoped shuffle scratch, and an error mid-partition can no longer
strand the buffer inside a dropped writer. The writer asserts a fresh
scratch is drained and that every call passes the same buffer, since a
swapped buffer would silently lose unflushed bytes. Flush shrinks the
scratch back to the configured write-buffer size, bounding retained
capacity that could previously reach the buffer size plus the largest
block. The producer asserts the batch-ref slice covers every buffered
batch, and the criterion suite gains a high-partition group so
partition scaling stays visible to benchmark checks.1 parent 6f93521 commit 08ec7be
6 files changed
Lines changed: 275 additions & 73 deletions
File tree
- native/shuffle
- benches
- src
- partitioners
- writers
- local
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
177 | 206 | | |
178 | 207 | | |
179 | 208 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
58 | 65 | | |
59 | 66 | | |
60 | 67 | | |
| |||
209 | 216 | | |
210 | 217 | | |
211 | 218 | | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
212 | 233 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1170 | 1170 | | |
1171 | 1171 | | |
1172 | 1172 | | |
1173 | | - | |
1174 | 1173 | | |
| 1174 | + | |
1175 | 1175 | | |
1176 | | - | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
1177 | 1179 | | |
1178 | | - | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
1179 | 1183 | | |
1180 | 1184 | | |
1181 | 1185 | | |
| |||
1187 | 1191 | | |
1188 | 1192 | | |
1189 | 1193 | | |
1190 | | - | |
1191 | 1194 | | |
| 1195 | + | |
1192 | 1196 | | |
1193 | | - | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
1194 | 1200 | | |
1195 | | - | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
1196 | 1204 | | |
1197 | 1205 | | |
1198 | 1206 | | |
| |||
1292 | 1300 | | |
1293 | 1301 | | |
1294 | 1302 | | |
1295 | | - | |
1296 | 1303 | | |
| 1304 | + | |
1297 | 1305 | | |
1298 | | - | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
1299 | 1309 | | |
1300 | | - | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
1301 | 1313 | | |
1302 | 1314 | | |
1303 | 1315 | | |
| |||
0 commit comments