Repository navigation
Commit ae427aa
authored
fix: cast lambda variables to their declared type instead of erroring (apache#25418) (#186)
## Which issue does this PR close?
- Closes apache#25411
## Rationale for this change
`create_physical_expr` required an `Expr::LambdaVariable`'s field to
equal the planning schema's field and returned a plan error otherwise.
The two can legitimately differ when a plan producer records a type that
Substrait cannot express, such as dictionary encoding or a string view
(Utf8view), since the recorded field and the field derived by
`lambda_parameters` come from independent sources.
## What changes are included in this PR?
Cast the variable to the recorded type instead, so the declared type is
enforced rather than asserted. This is the same approach already used
for `ScalarFunction` arguments, which the `TypeCoercion` analyzer casts
to their coerced types instead of requiring them to match. Incompatible
types still fail, as a cast error.
## What is the testing strategy for this PR?
I added
- Unit tests covering dictionary-encoded and `Utf8View` list elements
with the lambda variable declared as plain `Utf8`, plus plan-shape tests
pinning that the cast is inserted on a mismatch and omitted when the
types already agree.
- A Substrait integration test with a plan whose read schema declares a
dictionary-encoded list element while the lambda carries its own plain
`Utf8` parameter type.
## Are there any user-facing changes?
No, this is a bug fix
(cherry picked from commit 5da95ac)1 parent 7269807 commit ae427aa
7 files changed
Lines changed: 709 additions & 19 deletions
File tree
- datafusion
- expr/src
- functions-nested/src
- optimizer/src/analyzer
- substrait/tests
- cases
- testdata/test_plans
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1219 | 1219 | | |
1220 | 1220 | | |
1221 | 1221 | | |
1222 | | - | |
1223 | | - | |
1224 | | - | |
1225 | | - | |
1226 | | - | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
1227 | 1225 | | |
1228 | | - | |
| 1226 | + | |
1229 | 1227 | | |
1230 | | - | |
1231 | | - | |
1232 | | - | |
1233 | | - | |
1234 | | - | |
1235 | | - | |
1236 | | - | |
1237 | | - | |
1238 | | - | |
1239 | | - | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
| 1240 | + | |
1240 | 1241 | | |
1241 | 1242 | | |
1242 | 1243 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
191 | 191 | | |
192 | 192 | | |
193 | 193 | | |
194 | | - | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
195 | 198 | | |
196 | | - | |
| 199 | + | |
197 | 200 | | |
198 | | - | |
| 201 | + | |
199 | 202 | | |
200 | 203 | | |
201 | 204 | | |
| |||
490 | 493 | | |
491 | 494 | | |
492 | 495 | | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
493 | 566 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
166 | 176 | | |
167 | 177 | | |
168 | 178 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
340 | 340 | | |
341 | 341 | | |
342 | 342 | | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
343 | 436 | | |
0 commit comments