Repository navigation
Commit 5b45ae4
authored
Compose and drive a V2 scan in DataFusion (#10708)
Third of a stack of three that lands the planner-driven V2 scan for
DataFusion behind a switch, cut from #10694. Stacked on #10707 (the
planner chain), itself on #10706. With this merged, `scan_v2` runs TPC-H
end to end; the perf and diagnostics commits of #10694 follow as
separate PRs on top.
## Measured on its own
Against the V1 scan, locally (14 cores, interleaved A/B/B/A, 10 and 5
iterations a run) at the same code before the stack was cut
(da8a7c8):
| query | SF1 | SF10 |
|---|---|---|
| q1 | 1.074 | 1.024 |
| q6 | 0.966 | 0.832 |
| q10 | 1.025 | 0.882 |
| q12 | 0.939 | 0.811 |
| q13 | 1.640 | 1.125 |
| q19 | 0.996 | 0.909 |
| geomean | 1.085 | 0.924 |
SF10 improves over #10707 (0.977 to 0.924) as the per-step task hops go;
SF1 does not yet, and q13 at SF1 is far behind because every partition
still plans the filter and projection itself (the plan-once and
interning commits of #10694 fix that, taking the full branch to 0.963 at
SF1). This PR's base is a branch, so CI's benchmark dispatch does not
run on it.
---------
Signed-off-by: Joe Isaacs <[email protected]>1 parent 33675d9 commit 5b45ae4
18 files changed
Lines changed: 807 additions & 639 deletions
File tree
- benchmarks/compress-bench/src
- vortex-bench/src/random_access
- vortex-datafusion/src/persistent
- vortex-file/src/v2
- vortex-layout/src
- plan/physical
- plans
- scan
- execution
- v2
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | 6 | | |
8 | 7 | | |
9 | 8 | | |
10 | 9 | | |
11 | 10 | | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| 23 | + | |
| 24 | + | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
| |||
96 | 99 | | |
97 | 100 | | |
98 | 101 | | |
99 | | - | |
| 102 | + | |
| 103 | + | |
100 | 104 | | |
101 | | - | |
| 105 | + | |
102 | 106 | | |
103 | 107 | | |
104 | | - | |
105 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
106 | 113 | | |
107 | 114 | | |
108 | 115 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
38 | 40 | | |
39 | 41 | | |
40 | 42 | | |
| |||
190 | 192 | | |
191 | 193 | | |
192 | 194 | | |
193 | | - | |
194 | | - | |
195 | | - | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
196 | 198 | | |
197 | 199 | | |
198 | 200 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| 56 | + | |
56 | 57 | | |
57 | 58 | | |
| 59 | + | |
| 60 | + | |
58 | 61 | | |
59 | 62 | | |
60 | 63 | | |
61 | 64 | | |
62 | 65 | | |
| 66 | + | |
63 | 67 | | |
64 | 68 | | |
| 69 | + | |
65 | 70 | | |
66 | 71 | | |
67 | 72 | | |
| |||
77 | 82 | | |
78 | 83 | | |
79 | 84 | | |
| 85 | + | |
80 | 86 | | |
81 | 87 | | |
82 | 88 | | |
| |||
116 | 122 | | |
117 | 123 | | |
118 | 124 | | |
119 | | - | |
| 125 | + | |
120 | 126 | | |
121 | 127 | | |
122 | 128 | | |
| |||
437 | 443 | | |
438 | 444 | | |
439 | 445 | | |
440 | | - | |
441 | | - | |
| 446 | + | |
442 | 447 | | |
443 | 448 | | |
444 | 449 | | |
| |||
462 | 467 | | |
463 | 468 | | |
464 | 469 | | |
465 | | - | |
466 | | - | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
467 | 499 | | |
468 | 500 | | |
469 | 501 | | |
| |||
0 commit comments