Skip to content

Commit d23d5df

Browse files
morningmanclaude
andcommitted
fix: FIX-STATIC-CREDS-BE — normalize static object-store creds to BE-canonical AWS_*
Finding B-9 (BLOCKER, rereview2). The paimon connector copied static catalog-level storage credentials/config verbatim into the BE scan-node properties: PaimonScanPlanProvider.getScanNodeProperties iterated the raw catalog properties and emitted location.<rawkey> for any s3./oss./cos./obs./ hadoop./fs./dfs./hive. prefix; the fe-core bridge only strips the location. prefix. BE's native (FILE_S3) reader understands ONLY AWS_ACCESS_KEY/ AWS_SECRET_KEY/AWS_ENDPOINT/AWS_REGION/AWS_TOKEN, so static s3.access_key/ oss.access_key on a private bucket reached BE unintelligible -> no usable credentials -> 403. This is the third credential seam (static->BE-scan), missed by both the prior round and the 8 fixes (review §9.3); the catalog- FileIO seam (FIX-STORAGE-CREDS) and the vended seam (FIX-REST-VENDED) were already closed. Root cause: legacy PaimonScanNode.getLocationProperties returns only CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap) (the canonical AWS_*/hadoop/dfs map). The cutover replaced that single normalized call with a raw prefix-copy loop; the connector cannot import fe-core's StorageProperties so it had no access to the normalization. Solution (D-048, user-signed full legacy-parity scope): new no-op-default SPI ConnectorContext.getBackendStorageProperties(); DefaultConnectorContext returns getBackendPropertiesFromStorageMap over the storagePropertiesSupplier already wired in FIX-URI-NORMALIZE (no ctor change, CredentialUtils already imported). The connector replaces its raw prefix-copy loop with a context-gated overlay of that map; the vended overlay stays after it (vended wins on collision, legacy precedence). Object-store creds -> AWS_*; HDFS -> canonical hadoop/dfs (preserves user overrides + adds the legacy defaults, folding in the §211 MINOR); drops the non-parity hive.* passthrough. Investigated the AWS_CREDENTIALS_PROVIDER_TYPE=ANONYMOUS two-step edge and confirmed via BE s3_util.cpp (both providers prefer explicit ak/sk over cred_provider_type) that it is harmless — no regression. Connector import-gate stays clean. Tests: fe-core DefaultConnectorContextBackendStoragePropsTest (OSS static creds -> AWS_*, raw alias absent; no-supplier -> empty); connector PaimonScanPlanProviderTest (+getScanNodePropertiesNormalizesStaticCreds raw alias not shipped; modified vended-overlay collision to canonical keys; renamed no-context test -> emits no storage props). Fail-before/pass-after proven by reverting the connector change (2/3 go red). Module 217/0/0 (1 CI-gated skip), checkstyle clean, import-gate clean. Live private-bucket native-read e2e is CI-gated (not run). SPI RFC §22 (E14). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 741169e commit d23d5df

9 files changed

Lines changed: 337 additions & 26 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -369,14 +369,17 @@ public Map<String, String> getScanNodeProperties(
369369
props.put("paimon.options_json", sb.toString());
370370
}
371371

372-
// Location / storage properties (static catalog-level keys)
373-
for (Map.Entry<String, String> entry : properties.entrySet()) {
374-
String key = entry.getKey();
375-
if (key.startsWith("hadoop.") || key.startsWith("fs.")
376-
|| key.startsWith("dfs.") || key.startsWith("hive.")
377-
|| key.startsWith("s3.") || key.startsWith("cos.")
378-
|| key.startsWith("oss.") || key.startsWith("obs.")) {
379-
props.put("location." + key, entry.getValue());
372+
// FIX-STATIC-CREDS-BE (B-9): static catalog-level storage credentials/config, normalized to
373+
// BE-canonical keys (AWS_* for object stores, hadoop/dfs for HDFS). Ports legacy
374+
// PaimonScanNode.getLocationProperties() = getBackendPropertiesFromStorageMap(storagePropertiesMap):
375+
// BE's native (FILE_S3) reader understands ONLY the canonical keys, so the raw catalog aliases
376+
// (s3.access_key, oss.access_key, …) must be translated before they leave FE — copying them
377+
// verbatim gives the native reader no usable creds (403 on a private bucket). The connector
378+
// cannot import fe-core StorageProperties -> it delegates to the ConnectorContext seam. Empty
379+
// when no context (offline unit tests) -> no storage props emitted (never the broken raw aliases).
380+
if (context != null) {
381+
for (Map.Entry<String, String> e : context.getBackendStorageProperties().entrySet()) {
382+
props.put("location." + e.getKey(), e.getValue());
380383
}
381384
}
382385

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java‎

Lines changed: 69 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -521,9 +521,11 @@ public void extractVendedTokenEmptyForNullAndNonRestFileIO() {
521521
"a non-RESTTokenFileIO table must yield no vended token");
522522
}
523523

524-
/** A ConnectorContext whose vendStorageCredentials returns a fixed normalized map (the engine's
525-
* StorageProperties normalization is exercised by the fe-core DefaultConnectorContextVendTest). */
526-
private static ConnectorContext vendingContext(Map<String, String> fixed) {
524+
/** A ConnectorContext whose getBackendStorageProperties / vendStorageCredentials return fixed
525+
* normalized maps. The engine's real StorageProperties normalization is exercised by the fe-core
526+
* DefaultConnectorContextBackendStoragePropsTest / DefaultConnectorContextVendTest; here we pin the
527+
* connector wiring (overlay order + that the raw catalog aliases are NOT shipped). */
528+
private static ConnectorContext scanContext(Map<String, String> backendStatic, Map<String, String> vended) {
527529
return new ConnectorContext() {
528530
@Override
529531
public String getCatalogName() {
@@ -535,13 +537,56 @@ public long getCatalogId() {
535537
return 0;
536538
}
537539

540+
@Override
541+
public Map<String, String> getBackendStorageProperties() {
542+
return backendStatic;
543+
}
544+
538545
@Override
539546
public Map<String, String> vendStorageCredentials(Map<String, String> raw) {
540-
return fixed;
547+
return vended;
541548
}
542549
};
543550
}
544551

552+
@Test
553+
public void getScanNodePropertiesNormalizesStaticCreds() {
554+
FakePaimonTable table = new FakePaimonTable(
555+
"t1", rowType("id"), Collections.emptyList(), Collections.emptyList());
556+
PaimonTableHandle handle = new PaimonTableHandle(
557+
"db1", "t1", Collections.emptyList(), Collections.emptyList());
558+
handle.setPaimonTable(table);
559+
560+
// The connector holds the RAW catalog aliases; the engine seam returns the BE-canonical map.
561+
Map<String, String> props = new HashMap<>();
562+
props.put("s3.access_key", "raw-ak");
563+
props.put("s3.secret_key", "raw-sk");
564+
565+
Map<String, String> backendStatic = new HashMap<>();
566+
backendStatic.put("AWS_ACCESS_KEY", "ak");
567+
backendStatic.put("AWS_SECRET_KEY", "sk");
568+
backendStatic.put("AWS_ENDPOINT", "ep");
569+
570+
PaimonScanPlanProvider provider = new PaimonScanPlanProvider(
571+
props, new RecordingPaimonCatalogOps(),
572+
scanContext(backendStatic, Collections.emptyMap()));
573+
574+
Map<String, String> scanProps = provider.getScanNodeProperties(
575+
null, handle, Collections.emptyList(), Optional.empty());
576+
577+
// WHY (BLOCKER B-9): BE's native (FILE_S3) reader understands ONLY AWS_* keys. The connector
578+
// must ship the engine-normalized canonical creds under location.*, NOT the raw catalog aliases
579+
// (s3.access_key/…) which BE cannot read (403 on a private bucket). MUTATION: re-introducing the
580+
// raw passthrough -> location.s3.access_key present / location.AWS_ACCESS_KEY absent -> red.
581+
Assertions.assertEquals("ak", scanProps.get("location.AWS_ACCESS_KEY"));
582+
Assertions.assertEquals("sk", scanProps.get("location.AWS_SECRET_KEY"));
583+
Assertions.assertEquals("ep", scanProps.get("location.AWS_ENDPOINT"));
584+
Assertions.assertFalse(scanProps.containsKey("location.s3.access_key"),
585+
"the raw catalog alias must NOT reach BE (that is the B-9 bug)");
586+
Assertions.assertFalse(scanProps.containsKey("location.s3.secret_key"),
587+
"the raw catalog alias must NOT reach BE (that is the B-9 bug)");
588+
}
589+
545590
@Test
546591
public void getScanNodePropertiesOverlaysVendedCreds() {
547592
FakePaimonTable table = new FakePaimonTable(
@@ -550,52 +595,59 @@ public void getScanNodePropertiesOverlaysVendedCreds() {
550595
"db1", "t1", Collections.emptyList(), Collections.emptyList());
551596
handle.setPaimonTable(table);
552597

598+
// Static (engine-normalized) creds; vended (REST per-table) creds collide on AWS_ACCESS_KEY /
599+
// AWS_ENDPOINT and must WIN (legacy precedence: vended overlays static).
600+
Map<String, String> backendStatic = new HashMap<>();
601+
backendStatic.put("AWS_ACCESS_KEY", "static-ak");
602+
backendStatic.put("AWS_ENDPOINT", "static-ep");
603+
553604
Map<String, String> vended = new HashMap<>();
554605
vended.put("AWS_ACCESS_KEY", "vended-ak");
555606
vended.put("AWS_SECRET_KEY", "vended-sk");
556607
vended.put("AWS_TOKEN", "vended-tok");
557-
vended.put("s3.endpoint", "vended-ep"); // collides with the static s3.endpoint below
608+
vended.put("AWS_ENDPOINT", "vended-ep");
558609

559-
Map<String, String> props = new HashMap<>();
560-
props.put("s3.endpoint", "static-ep");
561610
PaimonScanPlanProvider provider = new PaimonScanPlanProvider(
562-
props, new RecordingPaimonCatalogOps(), vendingContext(vended));
611+
new HashMap<>(), new RecordingPaimonCatalogOps(), scanContext(backendStatic, vended));
563612

564613
Map<String, String> scanProps = provider.getScanNodeProperties(
565614
null, handle, Collections.emptyList(), Optional.empty());
566615

567616
// WHY (BLOCKER): native-reader REST tables must receive normalized vended AWS_* creds under
568617
// location.*; without them BE hits the object store with no credentials (403). Vended overlays
569618
// static (legacy precedence). MUTATION: no overlay loop / context not threaded -> AWS_* absent
570-
// -> red; overlaying BEFORE the static loop -> the colliding location.s3.endpoint keeps the
571-
// static value -> red.
619+
// -> red; overlaying static AFTER vended -> the colliding location.AWS_ACCESS_KEY/ENDPOINT keep
620+
// the static value -> red.
572621
Assertions.assertEquals("vended-ak", scanProps.get("location.AWS_ACCESS_KEY"));
573622
Assertions.assertEquals("vended-sk", scanProps.get("location.AWS_SECRET_KEY"));
574623
Assertions.assertEquals("vended-tok", scanProps.get("location.AWS_TOKEN"));
575-
Assertions.assertEquals("vended-ep", scanProps.get("location.s3.endpoint"),
624+
Assertions.assertEquals("vended-ep", scanProps.get("location.AWS_ENDPOINT"),
576625
"vended creds must overlay (win over) the static location key on collision");
577626
}
578627

579628
@Test
580-
public void getScanNodePropertiesNoContextUnchanged() {
629+
public void getScanNodePropertiesNoContextNoStorageProps() {
581630
FakePaimonTable table = new FakePaimonTable(
582631
"t1", rowType("id"), Collections.emptyList(), Collections.emptyList());
583632
PaimonTableHandle handle = new PaimonTableHandle(
584633
"db1", "t1", Collections.emptyList(), Collections.emptyList());
585634
handle.setPaimonTable(table);
586635

587636
Map<String, String> props = new HashMap<>();
588-
props.put("s3.endpoint", "static-ep");
637+
props.put("s3.access_key", "raw-ak");
589638
// 2-arg ctor -> context == null (the offline harness path).
590639
PaimonScanPlanProvider provider = new PaimonScanPlanProvider(props, new RecordingPaimonCatalogOps());
591640

592641
Map<String, String> scanProps = provider.getScanNodeProperties(
593642
null, handle, Collections.emptyList(), Optional.empty());
594643

595-
// WHY: with no context (offline / no vended support) the static-only behavior is preserved —
596-
// no overlay, no NPE. MUTATION: NPE on null context, or adding vended keys -> red.
597-
Assertions.assertEquals("static-ep", scanProps.get("location.s3.endpoint"));
644+
// WHY: the connector cannot normalize static creds without the engine seam, so with no context
645+
// (offline only — production always wires one) it emits NO storage props — never the broken raw
646+
// aliases that BE cannot read. MUTATION: NPE on null context, or re-adding the raw passthrough
647+
// -> location.s3.access_key present -> red.
648+
Assertions.assertFalse(scanProps.containsKey("location.s3.access_key"),
649+
"no context -> the raw alias must not be shipped to BE");
598650
Assertions.assertFalse(scanProps.containsKey("location.AWS_ACCESS_KEY"),
599-
"no context -> no vended overlay");
651+
"no context -> no normalized overlay");
600652
}
601653
}

‎fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,4 +162,27 @@ default Map<String, String> vendStorageCredentials(Map<String, String> rawVended
162162
default String normalizeStorageUri(String rawUri) {
163163
return rawUri;
164164
}
165+
166+
/**
167+
* Returns the catalog's static storage credentials/config normalized to BE-canonical scan
168+
* properties: object-store creds as {@code AWS_ACCESS_KEY} / {@code AWS_SECRET_KEY} /
169+
* {@code AWS_TOKEN} / {@code AWS_ENDPOINT} / {@code AWS_REGION}, and HDFS config as the resolved
170+
* {@code hadoop.*} / {@code dfs.*} keys (user overrides plus the legacy-derived defaults). The
171+
* engine runs the same {@code CredentialUtils.getBackendPropertiesFromStorageMap} that legacy /
172+
* iceberg / hive use over the catalog's parsed {@code StorageProperties} map — the single source of
173+
* truth — so there is no re-ported normalization that could drift.
174+
*
175+
* <p>BE's native (FILE_S3) reader understands ONLY these canonical keys. A connector that copies
176+
* the raw catalog aliases ({@code s3.access_key}, {@code oss.access_key}, …) to BE hands the native
177+
* reader no usable credentials → 403 on a private bucket. A connector that emits static storage
178+
* props to BE MUST source them from this hook.
179+
*
180+
* <p>The default returns empty (no normalization machinery / no storage map), so every other
181+
* connector — and any credential-less (e.g. local-filesystem) warehouse — is unaffected.
182+
*
183+
* @return the BE-facing normalized storage-property map, or empty when none
184+
*/
185+
default Map<String, String> getBackendStorageProperties() {
186+
return Collections.emptyMap();
187+
}
165188
}

‎fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,17 @@ public Map<String, String> vendStorageCredentials(Map<String, String> rawVendedC
178178
}
179179
}
180180

181+
@Override
182+
public Map<String, String> getBackendStorageProperties() {
183+
// Mirror legacy PaimonScanNode.getLocationProperties(): translate the catalog's parsed
184+
// StorageProperties map into BE-canonical scan keys (AWS_* for object stores, hadoop/dfs for
185+
// HDFS) via the SAME CredentialUtils.getBackendPropertiesFromStorageMap legacy/iceberg/hive use
186+
// — single source of truth, no drift. The map is already validated at catalog creation, so this
187+
// does not throw; an empty map (non-plugin ctor / local-FS warehouse) yields an empty result
188+
// (no overlay) — correct parity, unlike normalizeStorageUri which must fail-loud on a bad path.
189+
return CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesSupplier.get());
190+
}
191+
181192
@Override
182193
public String normalizeStorageUri(String rawUri) {
183194
if (Strings.isNullOrEmpty(rawUri)) {
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector;
19+
20+
import org.apache.doris.common.security.authentication.ExecutionAuthenticator;
21+
import org.apache.doris.datasource.property.storage.StorageProperties;
22+
23+
import org.junit.jupiter.api.Assertions;
24+
import org.junit.jupiter.api.Test;
25+
26+
import java.util.HashMap;
27+
import java.util.List;
28+
import java.util.Map;
29+
import java.util.function.Function;
30+
import java.util.function.Supplier;
31+
import java.util.stream.Collectors;
32+
33+
/**
34+
* FIX-STATIC-CREDS-BE (B-9) fe-core bridge test: pins that
35+
* {@link DefaultConnectorContext#getBackendStorageProperties} translates the catalog's parsed
36+
* {@code StorageProperties} map into the BE-canonical {@code AWS_*} keys (the same
37+
* {@code CredentialUtils.getBackendPropertiesFromStorageMap} legacy {@code PaimonScanNode} returns
38+
* from {@code getLocationProperties()}). The paimon connector cannot import that machinery, so this
39+
* hook is its only access; without it the connector ships raw {@code s3.access_key}/{@code oss.*}
40+
* aliases to BE, whose native (FILE_S3) reader understands only {@code AWS_*} -> no usable
41+
* credentials -> 403 on a private bucket. FAILS before the fix (the method is a no-op default
42+
* returning empty).
43+
*/
44+
public class DefaultConnectorContextBackendStoragePropsTest {
45+
46+
private static final Supplier<ExecutionAuthenticator> NOOP_AUTH =
47+
() -> new ExecutionAuthenticator() {};
48+
49+
/** A context whose storage-props supplier yields a real OSS storage-properties map, built with
50+
* the same {@code StorageProperties.createAll} machinery a real OSS catalog uses. */
51+
private static DefaultConnectorContext ossContext() throws Exception {
52+
Map<String, String> oss = new HashMap<>();
53+
oss.put("oss.endpoint", "oss-cn-beijing.aliyuncs.com");
54+
oss.put("oss.access_key", "ak");
55+
oss.put("oss.secret_key", "sk");
56+
List<StorageProperties> all = StorageProperties.createAll(oss);
57+
Map<StorageProperties.Type, StorageProperties> map = all.stream()
58+
.collect(Collectors.toMap(StorageProperties::getType, Function.identity(), (a, b) -> a));
59+
return new DefaultConnectorContext("c", 1L, NOOP_AUTH, () -> map);
60+
}
61+
62+
@Test
63+
public void normalizesStaticOssCredsToBackendAwsProps() throws Exception {
64+
// WHY (BLOCKER B-9): the BE native S3/object-store reader consumes ONLY canonical AWS_* keys;
65+
// the raw oss.access_key/oss.secret_key catalog aliases are unintelligible to it. The bridge
66+
// must run getBackendPropertiesFromStorageMap to produce them. MUTATION: returning the no-op
67+
// default (empty), or echoing the raw oss.* keys -> AWS_ACCESS_KEY absent -> red.
68+
Map<String, String> be = ossContext().getBackendStorageProperties();
69+
70+
Assertions.assertEquals("ak", be.get("AWS_ACCESS_KEY"));
71+
Assertions.assertEquals("sk", be.get("AWS_SECRET_KEY"));
72+
Assertions.assertNotNull(be.get("AWS_ENDPOINT"), "endpoint must be emitted as canonical AWS_ENDPOINT");
73+
Assertions.assertFalse(be.containsKey("oss.access_key"),
74+
"the raw catalog alias must NOT survive to BE (that is the B-9 bug)");
75+
}
76+
77+
@Test
78+
public void noStorageMapYieldsEmpty() {
79+
// WHY: a context with no storage map (non-plugin ctor, or a credential-less local-FS warehouse)
80+
// must short-circuit to empty -> no overlay, parity with legacy
81+
// getBackendPropertiesFromStorageMap({}). MUTATION: NPE, or fabricating props from nothing -> red.
82+
Assertions.assertTrue(new DefaultConnectorContext("c", 1L).getBackendStorageProperties().isEmpty());
83+
}
84+
}

0 commit comments

Comments
 (0)