Repository navigation
Commit 79b4d33
committed
fix: walk an adaptive cache's final plan and keep the streamed order through semi joins
Review feedback on the streamed order that a native outer hash join declares on its probe input.
The serde walk that decides whether a cache or checkpoint can be trusted now:
- reads the final plan of an adaptive cached plan, as Spark's cache scan does, since the
adaptive plan reports no order itself and the walk stopped there;
- follows only the streamed side of a hash join, and counts an outer join as declared only
when it sent its order and its probe input is a scan in its own native plan;
- no longer trusts a leaf that reports an order unless it is a scan or a range, which covers
an RDD checkpoint whose plan is gone.
The planner now walks through right semi, anti and mark hash joins as well as inner ones, since
DataFusion builds their probe indices in ascending order whether or not the probe declares an
order. Only a right outer join depends on the declaration.
Tests cover a cached NOT IN with AQE off and on, through Spark's cache scan and through the native
one, a checkpointed NOT IN, an outer join over a cache written by a declared outer join and by one
over a sample, an outer join over a semi join, a NOT IN below an inner join's build side, and a
broadcast outer join over a sorted cache with default settings. The test that compared Comet with
itself now compares against the rows.1 parent 2e0fb63 commit 79b4d33
4 files changed
Lines changed: 538 additions & 154 deletions
File tree
- native/core/src/execution
- spark/src
- main/scala/org/apache/spark/sql/comet
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2981 | 2981 | | |
2982 | 2982 | | |
2983 | 2983 | | |
2984 | | - | |
2985 | | - | |
| 2984 | + | |
| 2985 | + | |
| 2986 | + | |
| 2987 | + | |
| 2988 | + | |
2986 | 2989 | | |
2987 | | - | |
| 2990 | + | |
| 2991 | + | |
| 2992 | + | |
| 2993 | + | |
| 2994 | + | |
| 2995 | + | |
| 2996 | + | |
2988 | 2997 | | |
2989 | 2998 | | |
2990 | 2999 | | |
| |||
6190 | 6199 | | |
6191 | 6200 | | |
6192 | 6201 | | |
6193 | | - | |
6194 | | - | |
6195 | | - | |
6196 | | - | |
6197 | | - | |
| 6202 | + | |
| 6203 | + | |
| 6204 | + | |
| 6205 | + | |
| 6206 | + | |
| 6207 | + | |
| 6208 | + | |
6198 | 6209 | | |
6199 | 6210 | | |
6200 | 6211 | | |
| |||
6205 | 6216 | | |
6206 | 6217 | | |
6207 | 6218 | | |
6208 | | - | |
6209 | | - | |
| 6219 | + | |
| 6220 | + | |
| 6221 | + | |
| 6222 | + | |
| 6223 | + | |
| 6224 | + | |
| 6225 | + | |
| 6226 | + | |
| 6227 | + | |
| 6228 | + | |
| 6229 | + | |
| 6230 | + | |
| 6231 | + | |
| 6232 | + | |
| 6233 | + | |
| 6234 | + | |
| 6235 | + | |
| 6236 | + | |
| 6237 | + | |
| 6238 | + | |
| 6239 | + | |
| 6240 | + | |
| 6241 | + | |
| 6242 | + | |
| 6243 | + | |
| 6244 | + | |
| 6245 | + | |
| 6246 | + | |
| 6247 | + | |
| 6248 | + | |
| 6249 | + | |
| 6250 | + | |
| 6251 | + | |
| 6252 | + | |
| 6253 | + | |
| 6254 | + | |
| 6255 | + | |
| 6256 | + | |
| 6257 | + | |
| 6258 | + | |
| 6259 | + | |
| 6260 | + | |
| 6261 | + | |
| 6262 | + | |
| 6263 | + | |
| 6264 | + | |
6210 | 6265 | | |
6211 | 6266 | | |
6212 | | - | |
6213 | | - | |
6214 | | - | |
6215 | | - | |
| 6267 | + | |
| 6268 | + | |
| 6269 | + | |
6216 | 6270 | | |
6217 | 6271 | | |
6218 | 6272 | | |
6219 | | - | |
6220 | | - | |
6221 | | - | |
6222 | | - | |
| 6273 | + | |
| 6274 | + | |
| 6275 | + | |
| 6276 | + | |
| 6277 | + | |
| 6278 | + | |
| 6279 | + | |
| 6280 | + | |
| 6281 | + | |
| 6282 | + | |
| 6283 | + | |
| 6284 | + | |
| 6285 | + | |
6223 | 6286 | | |
6224 | 6287 | | |
6225 | | - | |
6226 | | - | |
6227 | | - | |
6228 | | - | |
6229 | | - | |
| 6288 | + | |
| 6289 | + | |
6230 | 6290 | | |
6231 | 6291 | | |
6232 | 6292 | | |
6233 | 6293 | | |
6234 | | - | |
| 6294 | + | |
6235 | 6295 | | |
6236 | 6296 | | |
6237 | 6297 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
| |||
2884 | 2885 | | |
2885 | 2886 | | |
2886 | 2887 | | |
2887 | | - | |
| 2888 | + | |
| 2889 | + | |
2888 | 2890 | | |
2889 | 2891 | | |
2890 | 2892 | | |
| |||
2929 | 2931 | | |
2930 | 2932 | | |
2931 | 2933 | | |
2932 | | - | |
2933 | | - | |
2934 | | - | |
2935 | | - | |
2936 | | - | |
2937 | | - | |
| 2934 | + | |
| 2935 | + | |
| 2936 | + | |
| 2937 | + | |
| 2938 | + | |
| 2939 | + | |
| 2940 | + | |
| 2941 | + | |
| 2942 | + | |
| 2943 | + | |
| 2944 | + | |
| 2945 | + | |
| 2946 | + | |
| 2947 | + | |
| 2948 | + | |
2938 | 2949 | | |
| 2950 | + | |
| 2951 | + | |
2939 | 2952 | | |
2940 | 2953 | | |
2941 | | - | |
2942 | | - | |
2943 | | - | |
2944 | | - | |
2945 | 2954 | | |
| 2955 | + | |
| 2956 | + | |
| 2957 | + | |
| 2958 | + | |
| 2959 | + | |
| 2960 | + | |
| 2961 | + | |
| 2962 | + | |
| 2963 | + | |
| 2964 | + | |
| 2965 | + | |
| 2966 | + | |
| 2967 | + | |
| 2968 | + | |
| 2969 | + | |
| 2970 | + | |
| 2971 | + | |
2946 | 2972 | | |
2947 | 2973 | | |
2948 | 2974 | | |
2949 | 2975 | | |
2950 | 2976 | | |
| 2977 | + | |
| 2978 | + | |
| 2979 | + | |
| 2980 | + | |
2951 | 2981 | | |
2952 | 2982 | | |
2953 | 2983 | | |
| 2984 | + | |
| 2985 | + | |
| 2986 | + | |
| 2987 | + | |
| 2988 | + | |
| 2989 | + | |
| 2990 | + | |
| 2991 | + | |
| 2992 | + | |
| 2993 | + | |
| 2994 | + | |
| 2995 | + | |
| 2996 | + | |
| 2997 | + | |
| 2998 | + | |
| 2999 | + | |
| 3000 | + | |
| 3001 | + | |
| 3002 | + | |
| 3003 | + | |
| 3004 | + | |
| 3005 | + | |
| 3006 | + | |
| 3007 | + | |
| 3008 | + | |
| 3009 | + | |
2954 | 3010 | | |
2955 | 3011 | | |
2956 | 3012 | | |
| |||
0 commit comments