Repository navigation
Commit 587e74c
authored
fix: preserve short-circuit evaluation in Boolean CASE simplification (apache#25143)
## Which issue does this PR close?
Closes apache#25136.
## Rationale for this change
`CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END`
currently becomes an `AND` predicate. On input containing `'abc'`,
predicate evaluation can reach the cast even though the `WHEN` excluded
that row.
## What changes are included in this PR?
Restrict the searched Boolean CASE-to-AND/OR rewrite to columns or
literals in every `WHEN` condition and every `THEN`/`ELSE` output. Apply
the same restriction to the companion inversion rule. Even a nonvolatile
first condition can fail, so it must remain evaluated when both outputs
are identical constants.
This is deliberately conservative: it does not attempt general
expression fallibility analysis or change predicate ordering. More
complex conditional expressions retain their CASE structure. Existing
column/literal rewrites remain available, and two optimizer tests record
the intentionally reduced simplification.
Four EXPLAIN snapshots in `null_aware_mark_join.slt` and `subquery.slt`
also change. CASE expressions generated for ANY comparisons survive
until EXISTS decorrelation, allowing the first mark column to be reused
instead of duplicating its join. A correlated `max(...) IS NULL`
projection retains CASE under the stricter first-WHEN guard. Existing
query-result expectations are unchanged; an additional result test
covers non-NULL maxima, all-NULL groups, empty groups, and NULL
correlation keys.
## What is the testing strategy for this PR?
The SQL regressions in `case.slt` cover guarded casts in filters and
projections, true/false/NULL/implicit ELSE, a fallible later WHEN with
literal outputs, guarded division, and a selected invalid branch that
must still error. Five queries failed against current main before the
optimizer change. The regressions also cover a volatile first WHEN and a
fallible first WHEN with identical false outputs, which must still raise
the cast error.
Initial validation at `49c02ec7` with pinned Rust 1.97.0 (before the
first-WHEN review correction):
- `cargo fmt --all` and `git diff --check` passed.
- `cargo clippy --all-targets --all-features -- -D warnings` passed.
- The complete `./dev/rust_lint.sh` suite passed.
- The contributor-guide extended workspace command passed: 11,341 Rust
tests, 8 ignored, and all 513 SQL test files. No additional test
exclusions were needed.
Latest follow-up validation: all 787 optimizer tests and all 512 SQL
files supported by the existing review-fix binary passed, including six
focused CASE/subquery files. `cargo fmt --all`, `git diff --check`, and
`cargo clippy --all-targets --all-features -- -D warnings` passed with
pinned Rust 1.97.0. The full `./dev/rust_lint.sh` suite also passed.
Hosted CI for the follow-up remains pending.
The nearby `simplify_function_over_case` microbenchmark was run against
current main with separately rebuilt binaries (Rust 1.98.0, dev profile,
short Criterion samples). The quieter comparison reported no change or
changes within the noise threshold. It exercises string-output CASE, not
the retained Boolean CASE workloads, so this is not a general
performance claim.
## Are there any user-facing changes?
Guarded Boolean CASE expressions preserve branch-local evaluation. Some
optimized plans retain CASE expressions where Boolean lowering was
previously applied; there is no public API change.
Generated with OpenAI Codex assistance, including implementation, tests,
and this description.1 parent 16ecdc9 commit 587e74c
4 files changed
Lines changed: 241 additions & 170 deletions
File tree
- datafusion
- optimizer/src/simplify_expressions
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1575 | 1575 | | |
1576 | 1576 | | |
1577 | 1577 | | |
1578 | | - | |
| 1578 | + | |
| 1579 | + | |
| 1580 | + | |
1579 | 1581 | | |
1580 | 1582 | | |
1581 | 1583 | | |
| |||
1586 | 1588 | | |
1587 | 1589 | | |
1588 | 1590 | | |
1589 | | - | |
| 1591 | + | |
| 1592 | + | |
1590 | 1593 | | |
1591 | 1594 | | |
1592 | 1595 | | |
| |||
1643 | 1646 | | |
1644 | 1647 | | |
1645 | 1648 | | |
1646 | | - | |
| 1649 | + | |
| 1650 | + | |
1647 | 1651 | | |
1648 | 1652 | | |
1649 | 1653 | | |
| |||
2463 | 2467 | | |
2464 | 2468 | | |
2465 | 2469 | | |
| 2470 | + | |
| 2471 | + | |
| 2472 | + | |
| 2473 | + | |
| 2474 | + | |
| 2475 | + | |
| 2476 | + | |
| 2477 | + | |
| 2478 | + | |
| 2479 | + | |
| 2480 | + | |
| 2481 | + | |
| 2482 | + | |
| 2483 | + | |
2466 | 2484 | | |
2467 | 2485 | | |
2468 | 2486 | | |
| |||
2608 | 2626 | | |
2609 | 2627 | | |
2610 | 2628 | | |
2611 | | - | |
| 2629 | + | |
| 2630 | + | |
2612 | 2631 | | |
2613 | 2632 | | |
2614 | 2633 | | |
2615 | 2634 | | |
2616 | | - | |
| 2635 | + | |
| 2636 | + | |
| 2637 | + | |
2617 | 2638 | | |
2618 | 2639 | | |
2619 | 2640 | | |
| |||
4241 | 4262 | | |
4242 | 4263 | | |
4243 | 4264 | | |
4244 | | - | |
4245 | | - | |
4246 | | - | |
| 4265 | + | |
| 4266 | + | |
| 4267 | + | |
| 4268 | + | |
4247 | 4269 | | |
4248 | | - | |
4249 | | - | |
4250 | | - | |
4251 | | - | |
4252 | | - | |
4253 | | - | |
4254 | | - | |
4255 | | - | |
4256 | | - | |
4257 | | - | |
4258 | | - | |
| 4270 | + | |
| 4271 | + | |
| 4272 | + | |
| 4273 | + | |
4259 | 4274 | | |
4260 | | - | |
4261 | | - | |
4262 | | - | |
4263 | | - | |
4264 | | - | |
4265 | | - | |
4266 | | - | |
4267 | | - | |
4268 | | - | |
4269 | | - | |
| 4275 | + | |
| 4276 | + | |
| 4277 | + | |
| 4278 | + | |
4270 | 4279 | | |
4271 | | - | |
4272 | | - | |
4273 | | - | |
4274 | | - | |
4275 | | - | |
4276 | | - | |
4277 | | - | |
4278 | | - | |
4279 | | - | |
4280 | | - | |
4281 | | - | |
4282 | | - | |
4283 | | - | |
| 4280 | + | |
| 4281 | + | |
| 4282 | + | |
| 4283 | + | |
4284 | 4284 | | |
4285 | | - | |
4286 | | - | |
4287 | | - | |
4288 | | - | |
4289 | | - | |
4290 | | - | |
4291 | | - | |
4292 | | - | |
4293 | | - | |
4294 | | - | |
4295 | | - | |
4296 | | - | |
4297 | | - | |
4298 | | - | |
4299 | | - | |
4300 | | - | |
| 4285 | + | |
| 4286 | + | |
| 4287 | + | |
| 4288 | + | |
4301 | 4289 | | |
4302 | | - | |
| 4290 | + | |
| 4291 | + | |
| 4292 | + | |
| 4293 | + | |
| 4294 | + | |
| 4295 | + | |
| 4296 | + | |
| 4297 | + | |
| 4298 | + | |
| 4299 | + | |
| 4300 | + | |
| 4301 | + | |
| 4302 | + | |
| 4303 | + | |
| 4304 | + | |
| 4305 | + | |
| 4306 | + | |
| 4307 | + | |
| 4308 | + | |
| 4309 | + | |
| 4310 | + | |
| 4311 | + | |
| 4312 | + | |
| 4313 | + | |
| 4314 | + | |
| 4315 | + | |
| 4316 | + | |
| 4317 | + | |
| 4318 | + | |
| 4319 | + | |
| 4320 | + | |
4303 | 4321 | | |
4304 | | - | |
4305 | | - | |
4306 | | - | |
4307 | | - | |
4308 | | - | |
4309 | | - | |
4310 | | - | |
4311 | | - | |
4312 | | - | |
4313 | | - | |
| 4322 | + | |
| 4323 | + | |
| 4324 | + | |
| 4325 | + | |
| 4326 | + | |
| 4327 | + | |
| 4328 | + | |
| 4329 | + | |
| 4330 | + | |
| 4331 | + | |
| 4332 | + | |
| 4333 | + | |
| 4334 | + | |
| 4335 | + | |
| 4336 | + | |
| 4337 | + | |
| 4338 | + | |
| 4339 | + | |
| 4340 | + | |
| 4341 | + | |
| 4342 | + | |
| 4343 | + | |
| 4344 | + | |
| 4345 | + | |
| 4346 | + | |
| 4347 | + | |
4314 | 4348 | | |
4315 | 4349 | | |
4316 | 4350 | | |
| |||
4354 | 4388 | | |
4355 | 4389 | | |
4356 | 4390 | | |
4357 | | - | |
4358 | | - | |
4359 | | - | |
4360 | | - | |
| 4391 | + | |
| 4392 | + | |
| 4393 | + | |
| 4394 | + | |
| 4395 | + | |
| 4396 | + | |
| 4397 | + | |
4361 | 4398 | | |
4362 | 4399 | | |
4363 | 4400 | | |
4364 | 4401 | | |
4365 | 4402 | | |
4366 | 4403 | | |
4367 | | - | |
4368 | | - | |
4369 | | - | |
| 4404 | + | |
4370 | 4405 | | |
4371 | 4406 | | |
4372 | 4407 | | |
| |||
4407 | 4442 | | |
4408 | 4443 | | |
4409 | 4444 | | |
4410 | | - | |
4411 | | - | |
4412 | | - | |
| 4445 | + | |
| 4446 | + | |
| 4447 | + | |
4413 | 4448 | | |
4414 | 4449 | | |
4415 | 4450 | | |
4416 | | - | |
4417 | | - | |
4418 | | - | |
4419 | | - | |
4420 | | - | |
4421 | | - | |
4422 | | - | |
4423 | | - | |
4424 | | - | |
4425 | | - | |
4426 | | - | |
| 4451 | + | |
| 4452 | + | |
4427 | 4453 | | |
4428 | 4454 | | |
4429 | 4455 | | |
4430 | | - | |
4431 | | - | |
4432 | | - | |
| 4456 | + | |
| 4457 | + | |
| 4458 | + | |
| 4459 | + | |
4433 | 4460 | | |
4434 | 4461 | | |
4435 | 4462 | | |
| |||
4626 | 4653 | | |
4627 | 4654 | | |
4628 | 4655 | | |
4629 | | - | |
4630 | | - | |
4631 | | - | |
4632 | | - | |
4633 | | - | |
4634 | | - | |
4635 | | - | |
4636 | | - | |
4637 | | - | |
4638 | | - | |
4639 | | - | |
4640 | | - | |
4641 | | - | |
4642 | | - | |
4643 | | - | |
4644 | | - | |
4645 | 4656 | | |
4646 | 4657 | | |
4647 | 4658 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
906 | 906 | | |
907 | 907 | | |
908 | 908 | | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
0 commit comments