You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d23d5df
Browse filesBrowse the repository at this point in the historyBrowse files
fix: FIX-STATIC-CREDS-BE — normalize static object-store creds to BE-canonical AWS_*
Finding B-9 (BLOCKER, rereview2). The paimon connector copied static
catalog-level storage credentials/config verbatim into the BE scan-node
properties: PaimonScanPlanProvider.getScanNodeProperties iterated the raw
catalog properties and emitted location.<rawkey> for any s3./oss./cos./obs./
hadoop./fs./dfs./hive. prefix; the fe-core bridge only strips the location.
prefix. BE's native (FILE_S3) reader understands ONLY AWS_ACCESS_KEY/
AWS_SECRET_KEY/AWS_ENDPOINT/AWS_REGION/AWS_TOKEN, so static s3.access_key/
oss.access_key on a private bucket reached BE unintelligible -> no usable
credentials -> 403. This is the third credential seam (static->BE-scan),
missed by both the prior round and the 8 fixes (review §9.3); the catalog-
FileIO seam (FIX-STORAGE-CREDS) and the vended seam (FIX-REST-VENDED) were
already closed.
Root cause: legacy PaimonScanNode.getLocationProperties returns only
CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap) (the
canonical AWS_*/hadoop/dfs map). The cutover replaced that single normalized
call with a raw prefix-copy loop; the connector cannot import fe-core's
StorageProperties so it had no access to the normalization.
Solution (D-048, user-signed full legacy-parity scope): new no-op-default SPI
ConnectorContext.getBackendStorageProperties(); DefaultConnectorContext returns
getBackendPropertiesFromStorageMap over the storagePropertiesSupplier already
wired in FIX-URI-NORMALIZE (no ctor change, CredentialUtils already imported).
The connector replaces its raw prefix-copy loop with a context-gated overlay of
that map; the vended overlay stays after it (vended wins on collision, legacy
precedence). Object-store creds -> AWS_*; HDFS -> canonical hadoop/dfs
(preserves user overrides + adds the legacy defaults, folding in the §211
MINOR); drops the non-parity hive.* passthrough. Investigated the
AWS_CREDENTIALS_PROVIDER_TYPE=ANONYMOUS two-step edge and confirmed via BE
s3_util.cpp (both providers prefer explicit ak/sk over cred_provider_type) that
it is harmless — no regression. Connector import-gate stays clean.
Tests: fe-core DefaultConnectorContextBackendStoragePropsTest (OSS static creds
-> AWS_*, raw alias absent; no-supplier -> empty); connector
PaimonScanPlanProviderTest (+getScanNodePropertiesNormalizesStaticCreds raw
alias not shipped; modified vended-overlay collision to canonical keys; renamed
no-context test -> emits no storage props). Fail-before/pass-after proven by
reverting the connector change (2/3 go red). Module 217/0/0 (1 CI-gated skip),
checkstyle clean, import-gate clean. Live private-bucket native-read e2e is
CI-gated (not run). SPI RFC §22 (E14).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java
+11-8Lines changed: 11 additions & 8 deletions
Original file line number
Diff line number
Diff line change
@@ -369,14 +369,17 @@ public Map<String, String> getScanNodeProperties(
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java
+69-17Lines changed: 69 additions & 17 deletions
Original file line number
Diff line number
Diff line change
@@ -521,9 +521,11 @@ public void extractVendedTokenEmptyForNullAndNonRestFileIO() {
521
521
"a non-RESTTokenFileIO table must yield no vended token");
522
522
}
523
523
524
-
/** A ConnectorContext whose vendStorageCredentials returns a fixed normalized map (the engine's
525
-
* StorageProperties normalization is exercised by the fe-core DefaultConnectorContextVendTest). */
0 commit comments