Repository navigation
Commit b5e6db9
[SPARK-41416][SQL] Cover the A2 outputRemap path, remaining allowed types, and LocalRelation self-joins
Follow-up on the review of #58424, addressing the remaining non-blocking
items from cloud-fan's latest APPROVE.
Tests:
* Extend "Pattern A': direct InSubquery self-join is rewritten" to loop
the same query over a real Parquet table and a VALUES-backed temp view,
so the LocalRelation branch of `isRowBagRepeatable` /
`sameLeafSnapshotIdentity` -- a documented supported leaf -- gets an
actual firing case instead of appearing only as an unrelated outer-query
companion (e.g. `D` in the Pattern A2 tests).
* Extend "Pattern A2: nested self-join is rewritten" so the wrapper Project
aliases both self-join columns and the top-level Project selects the
sjRight-derived one, forcing the `outputRemap` /
`remapNamedExpressionAttributes` path in `rewriteNestedSelfJoin`. Every
other A2 test's top-level Project only ever selected an unrelated column
from the outer relation, so that path was never exercised.
* Add one new test, positive firing + rewrite-on/off parity, for the
allowed type families that had none: Decimal, Boolean, Date, Timestamp,
TimestampNTZ, Binary. Using the same type for both the equi-key and the
neq column in one query exercises `isSafeEquiKeyType` and
`isSafeNeqColumnType` together per type.
Comment:
* Fix `createTable`'s docstring, which still claimed a VALUES temp view
"cannot be used" for a self-joined fixture -- contradicting the
class-level comment above it, which already explains that a
VALUES-backed `LocalRelation` canonicalizes correctly too and is simply
not the shape this rule targets.
No production code changes; sql/testOnly *RewriteSelfJoinInequalityToAggregateSuite
passes (38/38).
Co-authored-by: Guo Wangyang <[email protected]>
Co-authored-by: Lipeng Zhu <[email protected]>
Signed-off-by: hengrui hu <[email protected]>1 parent 36bc398 commit b5e6db9
1 file changed
Lines changed: 74 additions & 19 deletions
File tree
- sql/core/src/test/scala/org/apache/spark/sql/execution
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
109 | | - | |
110 | | - | |
| 109 | + | |
| 110 | + | |
111 | 111 | | |
112 | 112 | | |
113 | 113 | | |
| |||
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
162 | | - | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
163 | 165 | | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
169 | 183 | | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
177 | 193 | | |
178 | 194 | | |
179 | 195 | | |
| |||
217 | 233 | | |
218 | 234 | | |
219 | 235 | | |
220 | | - | |
| 236 | + | |
221 | 237 | | |
222 | 238 | | |
| 239 | + | |
| 240 | + | |
223 | 241 | | |
224 | 242 | | |
225 | 243 | | |
226 | 244 | | |
227 | 245 | | |
228 | 246 | | |
229 | | - | |
230 | | - | |
| 247 | + | |
| 248 | + | |
231 | 249 | | |
232 | | - | |
| 250 | + | |
233 | 251 | | |
234 | 252 | | |
235 | 253 | | |
| |||
1230 | 1248 | | |
1231 | 1249 | | |
1232 | 1250 | | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
1233 | 1288 | | |
1234 | 1289 | | |
1235 | 1290 | | |
| |||
0 commit comments