Repository navigation
Commit fea28d7
perf(contrib-delta) P7j: pre-parse session TZ and key injectors by op kind
Two perf-sweep items from apache#135:
#7 parse_delta_partition_scalar TZ parse-once. The per-row
chrono_tz::Tz::from_str (or fixed-offset parse) was happening inside
parse_delta_partition_scalar for every TIMESTAMP partition value, but the
session TZ string doesn't change within a scan. Introduce SessionTimezone
enum (Tz | Offset | Invalid), parse once in build_delta_partitioned_files,
pass the parsed value through. parse_delta_partition_scalar's signature gains
&SessionTimezone and keeps session_tz: &str only for the error message.
#2 PlanDataInjector lookup by op kind. injectPlanData was running
`for (injector <- injectors if injector.canInject(op))` against every
operator in the tree; for a 50-op plan with 3 injectors that's 150
canInject calls just to find no match on most ops. Add `opStructCase` to the
PlanDataInjector trait, build a Map[OpStructCase, PlanDataInjector] once at
object init, and look up by op.getOpStructCase before any canInject call.
Iceberg/NativeScan/Delta injectors set their own opStructCase.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 56c2b01 commit fea28d7
3 files changed
Lines changed: 103 additions & 55 deletions
File tree
- contrib/delta
- native/src
- src/main/scala/org/apache/spark/sql/comet
- spark/src/main/scala/org/apache/spark/sql/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
46 | 100 | | |
47 | 101 | | |
48 | 102 | | |
49 | 103 | | |
50 | 104 | | |
51 | 105 | | |
52 | 106 | | |
| 107 | + | |
53 | 108 | | |
54 | 109 | | |
55 | 110 | | |
| |||
76 | 131 | | |
77 | 132 | | |
78 | 133 | | |
79 | | - | |
| 134 | + | |
80 | 135 | | |
81 | 136 | | |
82 | 137 | | |
| |||
110 | 165 | | |
111 | 166 | | |
112 | 167 | | |
| 168 | + | |
113 | 169 | | |
114 | 170 | | |
115 | 171 | | |
116 | 172 | | |
117 | 173 | | |
118 | | - | |
119 | 174 | | |
120 | 175 | | |
121 | 176 | | |
| |||
155 | 210 | | |
156 | 211 | | |
157 | 212 | | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
191 | 216 | | |
192 | 217 | | |
193 | 218 | | |
194 | 219 | | |
195 | 220 | | |
196 | | - | |
197 | | - | |
198 | | - | |
| 221 | + | |
| 222 | + | |
199 | 223 | | |
200 | 224 | | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
201 | 228 | | |
202 | | - | |
203 | | - | |
204 | 229 | | |
205 | 230 | | |
206 | 231 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
47 | 49 | | |
48 | 50 | | |
49 | 51 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
69 | 76 | | |
70 | 77 | | |
71 | 78 | | |
| |||
101 | 108 | | |
102 | 109 | | |
103 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
104 | 118 | | |
105 | 119 | | |
106 | 120 | | |
| |||
114 | 128 | | |
115 | 129 | | |
116 | 130 | | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
132 | 149 | | |
133 | 150 | | |
134 | 151 | | |
| |||
172 | 189 | | |
173 | 190 | | |
174 | 191 | | |
| 192 | + | |
| 193 | + | |
175 | 194 | | |
176 | 195 | | |
177 | 196 | | |
| |||
211 | 230 | | |
212 | 231 | | |
213 | 232 | | |
| 233 | + | |
| 234 | + | |
214 | 235 | | |
215 | 236 | | |
216 | 237 | | |
| |||
0 commit comments