Repository navigation
Commit 78834e5
committed
let claude change it back from mmap to vec deque
```
mmap vs VecDeque (branch/main medians, same box, same day)
┌──────────────────────────────────┬──────────────────────────────────┬──────────────────────────────────┐
│ │ mmap + pool │ VecDeque │
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
│ TPC-DS Q1 / Q43 / Q36 / Q27 │ 1.02 / 1.09 / 0.96 / 1.02 │ 1.06 / 0.98 / 1.04 / 1.05 │
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
│ TPC-DS Q76 / Q81 / Q23 / Q39 │ 1.09 / 1.05 / 1.14 / 1.10 │ 1.03 / 1.02 / 1.08 / 1.07 │
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
│ TPC-DS Q11 / Q57 / others │ 1.07 / 1.10 / 1.00–1.06 │ 1.07 / 1.07 / 0.99–1.03 │
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
│ external_aggr Q1 (64/32/16 MB) │ 1.19 / 1.08 / 1.05 │ 1.22 / 1.12 / 1.13 │
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
│ external_aggr Q2 (512…32 MB) │ 1.03 / 0.99 / 1.07 / 1.06 / 1.09 │ 1.05 / 0.94 / 1.08 / 1.07 / 1.04 │
├──────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
│ ClickBench Q33 / Q16 / Q18 / Q32 │ 1.02 / 1.11 / 0.87 / ~1.05 │ 1.17 / 1.09 / 1.10 / 1.07 │
└──────────────────────────────────┴──────────────────────────────────┴──────────────────────────────────┘
So: VecDeque wins on TPC-DS (mid-size aggregations, less setup and teardown), loses clearly on the big single-column string aggregations. Q33's profile shows why: GroupValuesBytesView::intern +24% and BlockedCount::update_batch +53%, from the two-level lookup (VecDeque slot → Vec → element) on every row versus one flat offset with mmap.
```1 parent bb36d00 commit 78834e5
7 files changed
Lines changed: 271 additions & 599 deletions
File tree
- datafusion
- expr-common/src/blocked_helpers
- functions-aggregate/src
- physical-expr-common/src
- physical-plan/src/aggregates_blocked
- group_values/multi_group_by
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
341 | 341 | | |
342 | 342 | | |
343 | 343 | | |
| 344 | + | |
| 345 | + | |
344 | 346 | | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
349 | 351 | | |
350 | 352 | | |
351 | 353 | | |
| |||
359 | 361 | | |
360 | 362 | | |
361 | 363 | | |
362 | | - | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
363 | 400 | | |
364 | 401 | | |
365 | 402 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | 77 | | |
91 | 78 | | |
92 | 79 | | |
93 | | - | |
94 | | - | |
95 | | - | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
96 | 84 | | |
97 | 85 | | |
98 | 86 | | |
| |||
158 | 146 | | |
159 | 147 | | |
160 | 148 | | |
161 | | - | |
162 | | - | |
163 | 149 | | |
164 | 150 | | |
165 | 151 | | |
166 | 152 | | |
167 | | - | |
168 | | - | |
169 | | - | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
170 | 158 | | |
171 | | - | |
| 159 | + | |
172 | 160 | | |
173 | 161 | | |
174 | 162 | | |
| |||
365 | 353 | | |
366 | 354 | | |
367 | 355 | | |
368 | | - | |
| 356 | + | |
| 357 | + | |
369 | 358 | | |
370 | 359 | | |
371 | 360 | | |
372 | 361 | | |
373 | 362 | | |
374 | | - | |
| 363 | + | |
375 | 364 | | |
376 | 365 | | |
377 | 366 | | |
| |||
408 | 397 | | |
409 | 398 | | |
410 | 399 | | |
411 | | - | |
| 400 | + | |
| 401 | + | |
412 | 402 | | |
413 | 403 | | |
414 | 404 | | |
| |||
0 commit comments