Skip to content

Commit 968a0f7

Browse files
laskoviymishkaIsaac
andcommitted
[fix](fe) Use the urlconnection HTTP client in the iceberg storage connectivity probe
### What problem does this PR solve? Issue Number: None Related PR: #69031 Problem Summary: The iceberg 1.12.0 upgrade defaults iceberg-aws's HTTP client to `urlconnection` in `IcebergCatalogFactory.buildBaseCatalogProperties`, because iceberg-aws's default `apache` type is `Apache5HttpClient` since 1.12.0 and needs an AWS SDK newer than the 2.29.52 Doris pins (#58566). The CREATE CATALOG storage connectivity probe does not go through that method. `IcebergConnector.buildStorageProbeProperties` builds its own map (only `s3.*` and `client.region`) and `probeStorage` loads an `S3FileIO` from it, so the probe fell back to `apache` and building the S3 client threw `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`. That is an `Error`, not caught by the probe's `catch (Exception e)`, so it never became a "connectivity test failed" result: CREATE CATALOG with static S3 credentials failed with an unexpected exception whether the credentials were right or wrong. - Move the default into `IcebergCatalogFactory.applyDefaultAwsHttpClient` and apply it in both `buildBaseCatalogProperties` and `buildStorageProbeProperties`. - Copy the catalog's `http-client.*` settings (client type, proxy, timeouts) into the probe map, so the probe reaches storage the same way the catalog does; `s3.*` still comes only from the bound storage. The other places the connector builds AWS clients do not need it: every catalog flavor (DLF included) goes through `buildBaseCatalogProperties`; the s3tables control-plane client uses the s3tables library's own `HttpClientProperties`, whose `apache` type is the v4 `ApacheHttpClient` shipped with 2.29.52; DLF's own `S3Client` sets `UrlConnectionHttpClient` explicitly; the STS assume-role client is a plain AWS SDK builder. ### Release note None ### Check List (For Author) - Test: Unit Test - Added IcebergConnectorTestConnectionTest#storageProbePropertiesBuildAnS3ClientOnTheShippedSdk (builds a real S3 client builder from the probe map on the shipped SDK) and #storageProbeUsesTheCatalogHttpClientSettings. - `build.sh --fe` passes. Ran every iceberg FE unit test with run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg, be-java-extensions/iceberg-metadata-scanner): 1652 tests, 0 failures, 5 skipped. - The external_table_p0/test_connection regression cases were not run (no docker regression environment available locally). - Behavior changed: No (fixes a regression of the iceberg 1.12.0 upgrade in #69031 before it is released) - Does this need documentation: No Co-authored-by: Isaac <[email protected]>
1 parent a7d5511 commit 968a0f7

3 files changed

Lines changed: 70 additions & 10 deletions

File tree

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -156,14 +156,22 @@ public static Map<String, String> buildBaseCatalogProperties(Map<String, String>
156156
opts.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
157157
}
158158
appendManifestCacheProperties(props, opts, metaCacheWeightLimited);
159-
// iceberg-aws's default "apache" client type is Apache5HttpClient since iceberg 1.12, which needs an AWS SDK
160-
// newer than the one Doris pins (2.29.x, see #58566), so every S3FileIO / Glue / STS client iceberg builds
161-
// would fail to load. url-connection-client ships with that SDK. The key travels with S3FileIO's properties,
162-
// so the BE metadata scanner, which rebuilds the FileIO from them, uses the same client.
163-
opts.putIfAbsent(HttpClientProperties.CLIENT_TYPE, HttpClientProperties.CLIENT_TYPE_URLCONNECTION);
159+
applyDefaultAwsHttpClient(opts);
164160
return opts;
165161
}
166162

163+
/**
164+
* Defaults iceberg-aws's HTTP client to {@code urlconnection} unless {@code opts} sets one. iceberg-aws's default
165+
* "apache" client type is Apache5HttpClient since iceberg 1.12, which needs an AWS SDK newer than the one Doris
166+
* pins (2.29.x, see #58566), so every S3FileIO / Glue / STS client iceberg builds from a map without this key fails
167+
* with NoClassDefFoundError; url-connection-client ships with that SDK. Every map that ends up building an
168+
* iceberg AWS client must go through here. The key travels with S3FileIO's properties, so the BE metadata
169+
* scanner, which rebuilds the FileIO from them, uses the same client.
170+
*/
171+
static void applyDefaultAwsHttpClient(Map<String, String> opts) {
172+
opts.putIfAbsent(HttpClientProperties.CLIENT_TYPE, HttpClientProperties.CLIENT_TYPE_URLCONNECTION);
173+
}
174+
167175
/**
168176
* Mirrors legacy {@code AbstractIcebergProperties.addManifestCacheProperties}: pass through any
169177
* explicitly-set {@code io.manifest.cache.*} keys, then — only when the user did NOT set

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,8 @@ public class IcebergConnector implements Connector {
173173

174174
/** The key BE looks up (case-sensitively) to learn which location to probe. */
175175
private static final String BE_TEST_LOCATION = "test_location";
176+
/** Prefix of iceberg-aws's HTTP client settings ({@code http-client.type}, proxy, timeouts). */
177+
private static final String HTTP_CLIENT_PROPERTY_PREFIX = "http-client.";
176178

177179
private final Map<String, String> properties;
178180
// The bound connector-level facts (flavor, type-mapping switches, namespace level), the single
@@ -469,12 +471,12 @@ private ConnectorTestResult probeStorage(String catalogType) {
469471

470472
// Map Doris s3.* keys to Iceberg S3FileIO keys and force static credentials (disable
471473
// remote/vended signing) so the probe validates exactly what the user configured.
472-
Map<String, String> ioProps = buildStorageProbeProperties(catalogType, chosenStorage.get());
474+
Map<String, String> ioProps = buildStorageProbeProperties(catalogType, chosenStorage.get(), properties);
473475

474476
// Load S3FileIO reflectively via CatalogUtil so this module needs no compile-time AWS SDK
475477
// dependency; the AWS SDK is resolved from the shared runtime classpath at execution time.
476478
// Pin the TCCL to the plugin loader for the probe: iceberg-aws builds its S3 client lazily and
477-
// resolves org.apache.iceberg.aws.ApacheHttpClientConfigurations via DynMethods (default loader =
479+
// resolves its org.apache.iceberg.aws.*HttpClientConfigurations class via DynMethods (default loader =
478480
// TCCL). This CREATE-CATALOG thread runs under the default 'app' TCCL, which would return the
479481
// fe-core copy of the class and ClassCast against the child-loaded plugin copy the rest of the
480482
// iceberg-aws stack uses — the SAME split-brain TcclPinningConnectorContext guards on the commit
@@ -510,8 +512,8 @@ private ConnectorTestResult probeStorage(String catalogType) {
510512
return probeStorageFromBackend(location);
511513
}
512514

513-
static Map<String, String> buildStorageProbeProperties(
514-
String catalogType, S3CompatibleFileSystemProperties storageProperties) {
515+
static Map<String, String> buildStorageProbeProperties(String catalogType,
516+
S3CompatibleFileSystemProperties storageProperties, Map<String, String> catalogProperties) {
515517
String endpoint = storageProperties.getEndpoint();
516518
if (IcebergCatalogProperties.TYPE_DLF.equalsIgnoreCase(catalogType)
517519
&& "OSS".equals(storageProperties.providerName())) {
@@ -530,6 +532,14 @@ static Map<String, String> buildStorageProbeProperties(
530532
if (!isBlank(storageProperties.getRegion())) {
531533
ioProps.put("client.region", storageProperties.getRegion());
532534
}
535+
// The probe builds its own S3FileIO, so it must reach the store the way the catalog does: same HTTP client
536+
// and proxy settings, and the same default client type as every other iceberg AWS client.
537+
for (Map.Entry<String, String> e : catalogProperties.entrySet()) {
538+
if (e.getKey().startsWith(HTTP_CLIENT_PROPERTY_PREFIX)) {
539+
ioProps.put(e.getKey(), e.getValue());
540+
}
541+
}
542+
IcebergCatalogFactory.applyDefaultAwsHttpClient(ioProps);
533543
return ioProps;
534544
}
535545

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTestConnectionTest.java‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,10 @@
2121
import org.apache.doris.connector.spi.ConnectorStorageContext;
2222
import org.apache.doris.connector.spi.ConnectorTestResult;
2323

24+
import org.apache.iceberg.aws.HttpClientProperties;
2425
import org.junit.jupiter.api.Assertions;
2526
import org.junit.jupiter.api.Test;
27+
import software.amazon.awssdk.services.s3.S3Client;
2628

2729
import java.lang.reflect.Method;
2830
import java.util.HashMap;
@@ -125,7 +127,7 @@ public void dlfStorageProbeUsesBoundOssCredentialsAndNormalizedEndpoint() {
125127
.usePathStyle("false");
126128

127129
Map<String, String> ioProps = IcebergConnector.buildStorageProbeProperties(
128-
IcebergCatalogProperties.TYPE_DLF, oss);
130+
IcebergCatalogProperties.TYPE_DLF, oss, new HashMap<>());
129131

130132
Assertions.assertEquals("http://s3.oss-cn-hangzhou-internal.aliyuncs.com", ioProps.get("s3.endpoint"));
131133
Assertions.assertEquals("bound-ak", ioProps.get("s3.access-key-id"));
@@ -135,6 +137,46 @@ public void dlfStorageProbeUsesBoundOssCredentialsAndNormalizedEndpoint() {
135137
Assertions.assertEquals("false", ioProps.get("s3.path-style-access"));
136138
}
137139

140+
@Test
141+
public void storageProbePropertiesBuildAnS3ClientOnTheShippedSdk() {
142+
// WHY: the probe builds its own S3FileIO from this map, outside buildBaseCatalogProperties. iceberg 1.12's
143+
// default "apache" client is Apache5HttpClient, absent from the AWS SDK Doris ships, so CREATE CATALOG failed
144+
// with NoClassDefFoundError (an Error the probe's catch(Exception) never turns into a test result).
145+
// MUTATION: drop the default -> "apache" -> NoClassDefFoundError -> red.
146+
Map<String, String> ioProps = IcebergConnector.buildStorageProbeProperties(
147+
IcebergCatalogProperties.TYPE_REST,
148+
new FakeS3CompatibleStorageProperties("S3").endpoint("http://127.0.0.1:1")
149+
.region("us-east-1").accessKey("ak").secretKey("sk"),
150+
new HashMap<>());
151+
152+
Assertions.assertEquals(HttpClientProperties.CLIENT_TYPE_URLCONNECTION,
153+
ioProps.get(HttpClientProperties.CLIENT_TYPE));
154+
Assertions.assertDoesNotThrow(
155+
() -> new HttpClientProperties(ioProps).applyHttpClientConfigurations(S3Client.builder()));
156+
}
157+
158+
@Test
159+
public void storageProbeUsesTheCatalogHttpClientSettings() {
160+
// WHY: the probe must reach storage the way the catalog will (proxy, timeouts), so the catalog's
161+
// http-client.* keys win over the default. MUTATION: not copying them -> the probe ignores the proxy -> red.
162+
Map<String, String> catalogProperties = new HashMap<>();
163+
catalogProperties.put(HttpClientProperties.CLIENT_TYPE, HttpClientProperties.CLIENT_TYPE_URLCONNECTION);
164+
catalogProperties.put("http-client.proxy-endpoint", "http://proxy:3128");
165+
catalogProperties.put("s3.endpoint", "http://catalog-only-key");
166+
167+
Map<String, String> ioProps = IcebergConnector.buildStorageProbeProperties(
168+
IcebergCatalogProperties.TYPE_REST,
169+
new FakeS3CompatibleStorageProperties("S3").endpoint("http://127.0.0.1:1")
170+
.region("us-east-1").accessKey("ak").secretKey("sk"),
171+
catalogProperties);
172+
173+
Assertions.assertEquals("http://proxy:3128", ioProps.get("http-client.proxy-endpoint"));
174+
Assertions.assertEquals(HttpClientProperties.CLIENT_TYPE_URLCONNECTION,
175+
ioProps.get(HttpClientProperties.CLIENT_TYPE));
176+
Assertions.assertEquals("http://127.0.0.1:1", ioProps.get("s3.endpoint"),
177+
"only http-client.* keys are copied; the storage binder stays the source of s3.*");
178+
}
179+
138180
@Test
139181
public void dlfQualifiedOssWarehouseUsesSelectedStorageNormalizerForProbe() throws Exception {
140182
String warehouse = "oss://bucket.oss-cn-hangzhou.aliyuncs.com/warehouse";

0 commit comments

Comments
 (0)