Repository navigation
Commit 5ee8d8f
committed
review: tighten the CometRule docstring, test and docs
The docstring claimed operator conversion against unconverted Spark scans
converts nothing. That is wrong: with spark.comet.convert.parquet.enabled=true
CometExecRule bridges the scan with a CometSparkToColumnarExec and converts the
operators above it, and it converts local table scans, empty relations and
shuffles with no scan conversion at all. State the narrower property instead,
drop the archaeology about what this PR changed, and record how the pass is
named in Spark's plan change log.
The test named an invariant that only held because
spark.comet.convert.parquet.enabled defaults to false, and it never ran
CometRule. Pin the conf explicitly and assert both directions: CometExecRule
alone leaves the FileSourceScanExec in place, CometRule on the same query
produces a CometNativeScanExec. Each direction needs its own plan, because
fallback reasons are node tags and CometNativeScan.isSupported declines a scan
already carrying one.
Also switch RevertNativeForTransitionHeavyStagesSuite.applyFullColumnarPipeline,
which chained the two rules by hand, to CometRule, and update
plugin_overview.md to describe the single registered rule and its two phases.1 parent 8dd66d4 commit 5ee8d8f
4 files changed
Lines changed: 47 additions & 31 deletions
File tree
- docs/source/contributor-guide
- spark/src
- main/scala/org/apache/comet/rules
- test/scala/org/apache/comet/rules
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
51 | | - | |
52 | | - | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
55 | 59 | | |
56 | 60 | | |
57 | 61 | | |
| |||
68 | 72 | | |
69 | 73 | | |
70 | 74 | | |
71 | | - | |
| 75 | + | |
72 | 76 | | |
73 | 77 | | |
74 | 78 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
36 | 34 | | |
37 | | - | |
38 | | - | |
39 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
40 | 39 | | |
41 | 40 | | |
42 | 41 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1086 | 1086 | | |
1087 | 1087 | | |
1088 | 1088 | | |
1089 | | - | |
| 1089 | + | |
1090 | 1090 | | |
1091 | 1091 | | |
1092 | 1092 | | |
1093 | 1093 | | |
1094 | | - | |
1095 | | - | |
1096 | | - | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
1097 | 1103 | | |
1098 | | - | |
1099 | | - | |
1100 | | - | |
1101 | | - | |
1102 | 1104 | | |
1103 | 1105 | | |
1104 | | - | |
1105 | | - | |
1106 | | - | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
1107 | 1119 | | |
1108 | | - | |
1109 | | - | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
1110 | 1123 | | |
1111 | 1124 | | |
1112 | 1125 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
48 | | - | |
49 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
0 commit comments