Repository navigation
Commit 444cbfc
perf: reuse column normalization context across expressions (apache#25010)
## Which issue does this PR close?
- Addresses fix (1) in apache#24777. Constructor-based logical-plan decoding
remains separate.
## Rationale for this change
Column normalization repeatedly collects fallback schemas and traverses
the input plan for USING columns. Wide expression lists and projections
repeat that work for the same immutable plan.
## What changes are included in this PR?
- Introduce a private, lazy normalization context reused across columns
and expression lists.
- Share the context in sort normalization and validated projection
construction, including wildcard expansion.
- Keep already-qualified columns and expressions that do not need
normalization on the existing fast path.
- Add benchmarks for qualified and unqualified expressions and
projection construction at several schema widths.
## What is the testing strategy for this PR?
- Add `normalize_batch_schema_precedence`, `normalize_batch_using_join`,
and `normalize_batch_skips_unused_plan_context` to cover schema
precedence, USING joins, ambiguity/error order, sort options, and lazy
handling of qualified columns and literals.
- In balanced local `release-nonlto` runs, constructing a 2,000-column
unqualified projection falls from about 100 ms to 35 ms. This measures
projection construction, not full protobuf decoding; small controls
remain noisy.
- Reproduce with `cargo bench -p datafusion-expr --bench
normalize_columns --profile release-nonlto`.
- Focused expression and SQL tests pass. The required extended workspace
test command also passes, including all 511 SQL logic-test files.
- Expression-crate Clippy passes with all targets and features enabled.
The complete documented `dev/rust_lint.sh` also passes, including strict
workspace documentation checks.
- Full-workspace Clippy with all features enabled hits the existing
PostgreSQL decimal-formatting lint in apache#24974; the affected source is
unchanged here.
## Are there any user-facing changes?
No public API or name-resolution behavior changes are intended.
Normalization reuses plan context instead of collecting it for each
column; existing schema lookup costs remain.1 parent b8c1506 commit 444cbfc
5 files changed
Lines changed: 249 additions & 32 deletions
File tree
- datafusion/expr
- benches
- src
- expr_rewriter
- logical_plan
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
75 | 76 | | |
76 | 77 | | |
77 | 78 | | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
68 | | - | |
| 67 | + | |
69 | 68 | | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
78 | 112 | | |
79 | | - | |
80 | | - | |
| 113 | + | |
| 114 | + | |
81 | 115 | | |
82 | 116 | | |
83 | 117 | | |
| |||
118 | 152 | | |
119 | 153 | | |
120 | 154 | | |
| 155 | + | |
121 | 156 | | |
122 | 157 | | |
123 | | - | |
| 158 | + | |
124 | 159 | | |
125 | 160 | | |
126 | 161 | | |
127 | 162 | | |
128 | 163 | | |
129 | 164 | | |
130 | 165 | | |
| 166 | + | |
131 | 167 | | |
132 | 168 | | |
133 | 169 | | |
134 | 170 | | |
135 | | - | |
| 171 | + | |
| 172 | + | |
136 | 173 | | |
137 | 174 | | |
138 | 175 | | |
| |||
382 | 419 | | |
383 | 420 | | |
384 | 421 | | |
385 | | - | |
| 422 | + | |
386 | 423 | | |
387 | 424 | | |
388 | 425 | | |
| |||
491 | 528 | | |
492 | 529 | | |
493 | 530 | | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
494 | 618 | | |
495 | 619 | | |
496 | 620 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| |||
1132 | 1132 | | |
1133 | 1133 | | |
1134 | 1134 | | |
1135 | | - | |
1136 | | - | |
1137 | | - | |
1138 | | - | |
1139 | | - | |
1140 | | - | |
1141 | | - | |
1142 | | - | |
1143 | | - | |
1144 | | - | |
1145 | | - | |
1146 | | - | |
| 1135 | + | |
1147 | 1136 | | |
1148 | 1137 | | |
1149 | 1138 | | |
| |||
2082 | 2071 | | |
2083 | 2072 | | |
2084 | 2073 | | |
| 2074 | + | |
2085 | 2075 | | |
2086 | 2076 | | |
2087 | 2077 | | |
| |||
2100 | 2090 | | |
2101 | 2091 | | |
2102 | 2092 | | |
2103 | | - | |
| 2093 | + | |
2104 | 2094 | | |
2105 | 2095 | | |
2106 | 2096 | | |
| |||
2122 | 2112 | | |
2123 | 2113 | | |
2124 | 2114 | | |
2125 | | - | |
| 2115 | + | |
2126 | 2116 | | |
2127 | 2117 | | |
2128 | 2118 | | |
2129 | 2119 | | |
2130 | 2120 | | |
2131 | 2121 | | |
2132 | 2122 | | |
2133 | | - | |
| 2123 | + | |
2134 | 2124 | | |
2135 | 2125 | | |
2136 | 2126 | | |
| |||
0 commit comments