Repository navigation
Commit 23c1b59
authored
Add regression tests for nullable NOT IN expressions (apache#25994)
## Which issue does this PR close?
Closes apache#25473.
Closes apache#25474.
## Rationale for this change
Issues apache#25473 and apache#25474 reported wrong results for `NOT IN (subquery)`
when the subquery column is `NOT NULL` but the value being compared can
evaluate to `NULL`.
This includes NULL constants such as `CAST(NULL AS INT)` and nullable
expressions over `NOT NULL` columns, such as a `CASE` expression without
an `ELSE`.
The fix itself landed in apache#25338, which determines null-awareness from
the full `IN` operand expressions. This PR adds the missing regression
coverage for both issues so this behavior does not regress.
## What changes are included in this PR?
This is a test-only change. There are no production code or public API
changes.
It adds optimizer unit tests that verify:
- A NULL constant compared against a non-nullable subquery column
produces a projected `__correlated_sq_1_value` and a `null_aware`
`LeftAnti` join.
- A non-nullable expression such as `test.c + 1`, where `test.c` is
non-nullable, continues to use a regular `LeftAnti` join without
`null_aware`.
It adds SQL logic tests for null-aware anti joins covering:
- `CAST(NULL AS INT)`, untyped `NULL`, and `NULLIF(1, 1)`.
- Nullable expressions over `NOT NULL` columns.
- Nullable expressions on the subquery side.
- NULL values against an empty subquery.
- A non-NULL constant control.
- A non-nullable `x + 1` control.
- Logical and physical `EXPLAIN` plans for the NULL constant, nullable
`CASE`, and non-nullable `x + 1` cases.
- The NULL constant and nullable `CASE` cases with
`datafusion.execution.target_partitions = 1`.
It also adds SQL logic tests for the mark-join path, including the `...
NOT IN (...) OR x = 99` form and a `SELECT`-list control that verifies
the mark evaluates to `NULL`.
## Are these changes tested?
Yes. This PR adds regression tests in:
- `datafusion/optimizer/src/decorrelate_predicate_subquery.rs`
- `datafusion/sqllogictest/test_files/null_aware_anti_join.slt`
- `datafusion/sqllogictest/test_files/null_aware_mark_join.slt`
The new tests verify both result correctness and expected
logical/physical plan shapes, including that nullable operands use
null-aware joins while non-nullable operands remain non-null-aware.
Verification:
```bash
cargo test -p datafusion-optimizer decorrelate_predicate_subquery
cargo test -p datafusion-sqllogictest --test sqllogictests -- null_aware
cargo test -p datafusion-sqllogictest --test sqllogictests -- subquery_projection
```
## Are there any user-facing changes?
No new user-facing behavior is introduced by this PR. The underlying
correctness fix already landed in apache#25338.
This PR only adds regression coverage for that behavior and does not
change any public APIs.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.1 parent 11c8c3d commit 23c1b59
3 files changed
Lines changed: 284 additions & 0 deletions
File tree
- datafusion
- optimizer/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2050 | 2050 | | |
2051 | 2051 | | |
2052 | 2052 | | |
| 2053 | + | |
| 2054 | + | |
| 2055 | + | |
| 2056 | + | |
| 2057 | + | |
| 2058 | + | |
| 2059 | + | |
| 2060 | + | |
| 2061 | + | |
| 2062 | + | |
| 2063 | + | |
| 2064 | + | |
| 2065 | + | |
| 2066 | + | |
| 2067 | + | |
| 2068 | + | |
| 2069 | + | |
| 2070 | + | |
| 2071 | + | |
| 2072 | + | |
| 2073 | + | |
| 2074 | + | |
| 2075 | + | |
| 2076 | + | |
| 2077 | + | |
| 2078 | + | |
| 2079 | + | |
| 2080 | + | |
| 2081 | + | |
| 2082 | + | |
| 2083 | + | |
| 2084 | + | |
| 2085 | + | |
| 2086 | + | |
| 2087 | + | |
| 2088 | + | |
| 2089 | + | |
| 2090 | + | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
| 2094 | + | |
| 2095 | + | |
| 2096 | + | |
| 2097 | + | |
| 2098 | + | |
| 2099 | + | |
| 2100 | + | |
| 2101 | + | |
| 2102 | + | |
| 2103 | + | |
| 2104 | + | |
| 2105 | + | |
| 2106 | + | |
2053 | 2107 | | |
2054 | 2108 | | |
2055 | 2109 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
942 | 942 | | |
943 | 943 | | |
944 | 944 | | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
998 | 998 | | |
999 | 999 | | |
1000 | 1000 | | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
0 commit comments