Repository navigation
Commit 968a0f7
[fix](fe) Use the urlconnection HTTP client in the iceberg storage connectivity probe
### What problem does this PR solve?
Issue Number: None
Related PR: #69031
Problem Summary: The iceberg 1.12.0 upgrade defaults iceberg-aws's HTTP
client to `urlconnection` in `IcebergCatalogFactory.buildBaseCatalogProperties`,
because iceberg-aws's default `apache` type is `Apache5HttpClient` since
1.12.0 and needs an AWS SDK newer than the 2.29.52 Doris pins (#58566).
The CREATE CATALOG storage connectivity probe does not go through that
method. `IcebergConnector.buildStorageProbeProperties` builds its own map
(only `s3.*` and `client.region`) and `probeStorage` loads an `S3FileIO`
from it, so the probe fell back to `apache` and building the S3 client
threw `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`.
That is an `Error`, not caught by the probe's `catch (Exception e)`, so it
never became a "connectivity test failed" result: CREATE CATALOG with
static S3 credentials failed with an unexpected exception whether the
credentials were right or wrong.
- Move the default into `IcebergCatalogFactory.applyDefaultAwsHttpClient`
and apply it in both `buildBaseCatalogProperties` and
`buildStorageProbeProperties`.
- Copy the catalog's `http-client.*` settings (client type, proxy,
timeouts) into the probe map, so the probe reaches storage the same way
the catalog does; `s3.*` still comes only from the bound storage.
The other places the connector builds AWS clients do not need it: every
catalog flavor (DLF included) goes through `buildBaseCatalogProperties`;
the s3tables control-plane client uses the s3tables library's own
`HttpClientProperties`, whose `apache` type is the v4 `ApacheHttpClient`
shipped with 2.29.52; DLF's own `S3Client` sets `UrlConnectionHttpClient`
explicitly; the STS assume-role client is a plain AWS SDK builder.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Added IcebergConnectorTestConnectionTest#storageProbePropertiesBuildAnS3ClientOnTheShippedSdk
(builds a real S3 client builder from the probe map on the shipped
SDK) and #storageProbeUsesTheCatalogHttpClientSettings.
- `build.sh --fe` passes. Ran every iceberg FE unit test with
run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg,
be-java-extensions/iceberg-metadata-scanner): 1652 tests, 0 failures,
5 skipped.
- The external_table_p0/test_connection regression cases were not run
(no docker regression environment available locally).
- Behavior changed: No (fixes a regression of the iceberg 1.12.0 upgrade
in #69031 before it is released)
- Does this need documentation: No
Co-authored-by: Isaac <[email protected]>1 parent a7d5511 commit 968a0f7
3 files changed
Lines changed: 70 additions & 10 deletions
File tree
- fe/fe-connector/fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
156 | 156 | | |
157 | 157 | | |
158 | 158 | | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
| 159 | + | |
164 | 160 | | |
165 | 161 | | |
166 | 162 | | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
167 | 175 | | |
168 | 176 | | |
169 | 177 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
173 | 173 | | |
174 | 174 | | |
175 | 175 | | |
| 176 | + | |
| 177 | + | |
176 | 178 | | |
177 | 179 | | |
178 | 180 | | |
| |||
469 | 471 | | |
470 | 472 | | |
471 | 473 | | |
472 | | - | |
| 474 | + | |
473 | 475 | | |
474 | 476 | | |
475 | 477 | | |
476 | 478 | | |
477 | | - | |
| 479 | + | |
478 | 480 | | |
479 | 481 | | |
480 | 482 | | |
| |||
510 | 512 | | |
511 | 513 | | |
512 | 514 | | |
513 | | - | |
514 | | - | |
| 515 | + | |
| 516 | + | |
515 | 517 | | |
516 | 518 | | |
517 | 519 | | |
| |||
530 | 532 | | |
531 | 533 | | |
532 | 534 | | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
533 | 543 | | |
534 | 544 | | |
535 | 545 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
125 | 127 | | |
126 | 128 | | |
127 | 129 | | |
128 | | - | |
| 130 | + | |
129 | 131 | | |
130 | 132 | | |
131 | 133 | | |
| |||
135 | 137 | | |
136 | 138 | | |
137 | 139 | | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
138 | 180 | | |
139 | 181 | | |
140 | 182 | | |
| |||
0 commit comments