Repository navigation
Commit 7c2e0eb
(Fix)[hive-writer] Fixed the issue when partition values contain spaces when writing to s3. (apache#35645)
## Proposed changes
Issue Number: close apache#31442
(Fix) [hive-writer] Fixed the issue when partition values contain spaces
when writing to s3.
### Error msg
```
org.apache.doris.common.UserException: errCode = 2, detailMessage = java.net.URISyntaxException: Illegal character in path at index 114: oss://xxxxxxxxxxx/hive/tpcds1000_partition_oss/call_center/cc_call_center_sk=1/cc_mkt_class=A bit narrow forms matter animals. Consist/cc_market_manager=Daniel Weller/cc_rec_end_date=2001-12-31/f6b5ff4253414b06-9fd365ef68e5ddc5_133f02fb-a7e0-4109-9100-fb748a28259e-0.zlib.orc
at org.apache.doris.common.util.S3URI.validateUri(S3URI.java:134) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.S3URI.parseUri(S3URI.java:120) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.S3URI.<init>(S3URI.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.S3URI.create(S3URI.java:108) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.fs.obj.S3ObjStorage.deleteObject(S3ObjStorage.java:194) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.fs.remote.ObjFileSystem.delete(ObjFileSystem.java:150) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.fs.remote.SwitchingFileSystem.delete(SwitchingFileSystem.java:92) ~[doris-fe.jar:1.2-
```
### Root Cause
Hadoop partition names will encode some special characters, but not
space characters, which is different from URI encoding. Therefore, an
error will be reported when constructing URI.
### Solution
The solution is to use regular expressions to parse URI, and then pass
in each part of URI to construct URI. This URI constructor will encode
each part of URI.1 parent 4520f1c commit 7c2e0eb
2 files changed
Lines changed: 43 additions & 9 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/common/util
- test/java/org/apache/doris/common/util
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
| |||
60 | 62 | | |
61 | 63 | | |
62 | 64 | | |
63 | | - | |
64 | | - | |
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
68 | 71 | | |
69 | 72 | | |
70 | 73 | | |
| |||
117 | 120 | | |
118 | 121 | | |
119 | 122 | | |
120 | | - | |
| 123 | + | |
| 124 | + | |
121 | 125 | | |
122 | 126 | | |
123 | 127 | | |
| |||
127 | 131 | | |
128 | 132 | | |
129 | 133 | | |
130 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
131 | 149 | | |
132 | | - | |
| 150 | + | |
133 | 151 | | |
134 | 152 | | |
135 | 153 | | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
136 | 157 | | |
137 | 158 | | |
138 | 159 | | |
| |||
143 | 164 | | |
144 | 165 | | |
145 | 166 | | |
146 | | - | |
| 167 | + | |
147 | 168 | | |
148 | 169 | | |
149 | 170 | | |
| |||
173 | 194 | | |
174 | 195 | | |
175 | 196 | | |
176 | | - | |
| 197 | + | |
177 | 198 | | |
178 | 199 | | |
179 | 200 | | |
| |||
201 | 222 | | |
202 | 223 | | |
203 | 224 | | |
204 | | - | |
| 225 | + | |
205 | 226 | | |
206 | 227 | | |
207 | 228 | | |
| |||
226 | 247 | | |
227 | 248 | | |
228 | 249 | | |
229 | | - | |
| 250 | + | |
230 | 251 | | |
231 | 252 | | |
232 | 253 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
170 | 170 | | |
171 | 171 | | |
172 | 172 | | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
173 | 186 | | |
174 | 187 | | |
175 | 188 | | |
| |||
0 commit comments