Repository navigation
Commit d36ad9f
committed
fix: escape the output path before parsing it as a URI
The output path reaches both write serdes as `Path.toString`, which decodes
percent escapes: a directory containing a space or a literal `%` yields a
string that is not a valid URI, and `URI.create` throws on it.
On Spark 4.0+ that exception escaped `CometWriteFiles.convert` and failed the
query. On 3.x `CometDataWritingCommand.convert` caught it and silently handed
the write back to Spark, so the native writer was never used for those paths.
Round-trip through `Path` instead, which re-escapes. Only the scheme and
authority reach `extractObjectStoreOptions`, but parsing has to succeed to get
at them.
Also assert that the INSERT INTO visibility test's write actually went native,
rather than inferring it from the read-back.1 parent df96895 commit d36ad9f
3 files changed
Lines changed: 42 additions & 7 deletions
File tree
- spark/src
- main/scala/org/apache/comet/serde/operator
- test/scala/org/apache/comet/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | 22 | | |
24 | 23 | | |
25 | 24 | | |
| |||
129 | 128 | | |
130 | 129 | | |
131 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
132 | 134 | | |
133 | | - | |
| 135 | + | |
134 | 136 | | |
135 | 137 | | |
136 | 138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | 22 | | |
24 | 23 | | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
144 | 148 | | |
145 | | - | |
| 149 | + | |
146 | 150 | | |
147 | 151 | | |
148 | 152 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
865 | 865 | | |
866 | 866 | | |
867 | 867 | | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
868 | 889 | | |
869 | 890 | | |
870 | 891 | | |
| |||
937 | 958 | | |
938 | 959 | | |
939 | 960 | | |
940 | | - | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
941 | 966 | | |
942 | 967 | | |
943 | 968 | | |
| |||
1172 | 1197 | | |
1173 | 1198 | | |
1174 | 1199 | | |
1175 | | - | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
1176 | 1205 | | |
1177 | 1206 | | |
1178 | 1207 | | |
| |||
1191 | 1220 | | |
1192 | 1221 | | |
1193 | 1222 | | |
1194 | | - | |
| 1223 | + | |
1195 | 1224 | | |
1196 | 1225 | | |
1197 | 1226 | | |
| |||
0 commit comments