Skip to content

Commit af5c7cc

Browse files
dengfangyuanclaude
andcommitted
[fix](paimon) do not let the Hadoop S3 mirror overwrite explicit options
The mirrorHadoopS3OptionsToCatalogOptions loop set every fs.s3a.* entry from the generated Hadoop configuration unconditionally, so for a catalog with static s3 keys plus an explicit assumed-role credentials provider the Hadoop side's SimpleAWSCredentialsProvider replaced the requested provider in the catalog options — the warehouse then re-opened under the base identity and failed with AccessDenied. The mirror now fills only options the catalog has not already set: the user's explicit fs.s3a.* entries keep precedence over the generated defaults. This affects the filesystem, HMS and JDBC paimon catalog variants alike. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 792aa85 commit af5c7cc

2 files changed

Lines changed: 38 additions & 1 deletion

File tree

‎fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,13 @@ protected void mirrorHadoopS3OptionsToCatalogOptions(Configuration conf) {
146146
}
147147
conf.forEach(entry -> {
148148
String key = entry.getKey();
149-
if (key.startsWith(HADOOP_S3A_PREFIX) && StringUtils.isNotBlank(entry.getValue())) {
149+
// Fill only ABSENT options: the user's explicit fs.s3a.* catalog options
150+
// (e.g. an assumed-role credentials provider set via a paimon.fs.s3a.*
151+
// property) take precedence over the Hadoop-generated defaults (the
152+
// SimpleAWSCredentialsProvider the static S3 keys produce); an
153+
// unconditional set would re-open the warehouse under the base identity.
154+
if (key.startsWith(HADOOP_S3A_PREFIX) && StringUtils.isNotBlank(entry.getValue())
155+
&& catalogOptions.get(key) == null) {
150156
catalogOptions.set(key, entry.getValue());
151157
}
152158
});

‎fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/AbstractPaimonPropertiesTest.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,6 +250,37 @@ void testForwardTableDefaultOptionsToPaimonCatalog() {
250250
"7", testProps.getCatalogOptionsMap().get("table-default.scan.snapshot-id"));
251251
}
252252

253+
@Test
254+
public void testHadoopS3MirrorDoesNotOverwriteExplicitProvider() {
255+
// Static S3 keys plus an explicit assumed-role credentials provider: the
256+
// Hadoop configuration carries the SimpleAWSCredentialsProvider the static
257+
// keys produce, but the user's provider entry is already in the catalog
258+
// options — the mirror must fill only absent options, or the warehouse
259+
// re-opens under the base identity and fails AccessDenied.
260+
Map<String, String> input = new HashMap<>();
261+
input.put("warehouse", "s3://tmp/warehouse");
262+
input.put("paimon.fs.s3a.aws.credentials.provider",
263+
"org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider");
264+
TestPaimonProperties testProps = new TestPaimonProperties(input);
265+
testProps.buildCatalogOptions();
266+
267+
// The Hadoop side only knows the Simple provider generated from the static
268+
// keys, plus defaults the catalog has not set yet. Mirror BEFORE the first
269+
// getCatalogOptionsMap() call: the map is memoized once read.
270+
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration(false);
271+
conf.set("fs.s3a.aws.credentials.provider",
272+
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");
273+
conf.set("fs.s3a.connection.maximum", "100");
274+
testProps.mirrorHadoopS3OptionsToCatalogOptions(conf);
275+
276+
Map<String, String> options = testProps.getCatalogOptionsMap();
277+
Assertions.assertEquals("org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider",
278+
options.get("fs.s3a.aws.credentials.provider"),
279+
"the explicit user provider must survive the Hadoop mirror");
280+
Assertions.assertEquals("100", options.get("fs.s3a.connection.maximum"),
281+
"absent defaults are still filled");
282+
}
283+
253284
@Test
254285
public void testWeightGovernanceDisablesSdkMetadataCacheUnlessUserConfigured() {
255286
// Doris weight governance owns retention: Paimon's own CachingCatalog would retain

0 commit comments

Comments
 (0)