You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 635aee0
Browse filesBrowse the repository at this point in the historyBrowse files
[perf](catalog) fe-connector-iceberg: derive the vended storage config once per scan, not per file (PERF-06)
Root cause: for a REST catalog with vended credentials, iceberg scan planning
re-derived the whole StorageProperties map (StorageProperties.createAll + a hadoop
Configuration build) for EVERY data file and EVERY delete file, via
context.normalizeStorageUri(uri, token). But the vended token is scan-invariant
(extracted once, the same Map instance threads through every range builder), and
the derivation is a pure function of the token, independent of the per-file URI.
So a 50k-file scan burned tens of seconds of FE CPU (doubled on merge-on-read)
rebuilding an identical map O(N_files + N_deletes) times.
Fix (connector-side "prepare once, apply many"): add a scan-scoped SPI seam
ConnectorContext.newStorageUriNormalizer(token) that returns a UnaryOperator<String>
which bakes the token-derived config in once and then normalizes each URI cheaply.
DefaultConnectorContext overrides it to build the effective map lazily-memoized on
the first non-empty URI, byte-identical to normalizeStorageUri(uri, token) (same
empty-uri short-circuit, same vended-replaces-static precedence, same fail-loud
LocationPath, same exception timing). The SPI default folds back to a per-call
normalizeStorageUri, so every other connector is unaffected. The iceberg provider
builds the normalizer once at each of the three scan-start sites (synchronous data,
streaming, position_deletes) and threads it through buildRange / convertDelete /
buildPositionDeleteRange in place of the token; TcclPinningConnectorContext delegates
the new method to the raw context so iceberg gets the hoist (this path is pure fe-core,
no TCCL pin needed). DefaultConnectorContext is per-catalog and shared across queries,
so this scan-scoped hoist deliberately keeps no cross-query state and retains no
credentials (the normalizer is a scan-local, single-threaded, GC'd at scan end).
Not in scope: the write path getBackendFileType is O(1) per write, not per data file.
Measurement / guard: one vended scan now derives the config once instead of
O(N_files + N_deletes) times. A new connector-side guard test asserts a 3-file scan
enters newStorageUriNormalizer exactly once while normalizing all three paths; new
fe-core parity tests assert the normalizer's output is byte-identical to
normalizeStorageUri across the vended / static / empty-uri / fail-loud cases.
Tests: fe-connector-iceberg full module 974 UT green (0 fail / 1 skip); fe-core
DefaultConnectorContextNormalizeUriTest 13 UT green. Checkstyle clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java
+10Lines changed: 10 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -30,6 +30,7 @@
30
30
importjava.util.Objects;
31
31
importjava.util.concurrent.Callable;
32
32
importjava.util.function.Supplier;
33
+
importjava.util.function.UnaryOperator;
33
34
34
35
/**
35
36
* A {@link ConnectorContext} decorator that pins the thread-context classloader (TCCL) to the iceberg plugin
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java
+38-10Lines changed: 38 additions & 10 deletions
Original file line number
Diff line number
Diff line change
@@ -83,6 +83,7 @@
83
83
importjava.util.NoSuchElementException;
84
84
importjava.util.Optional;
85
85
importjava.util.Set;
86
+
importjava.util.function.UnaryOperator;
86
87
87
88
/**
88
89
* Tests for {@link IcebergScanPlanProvider}. T01 pinned the capability constants + that {@code planScan}
@@ -1559,7 +1560,7 @@ public void convertDeletePositionDeleteCarriesBoundsAndFormat() {
1559
1560
// POSITION_DELETES (non-PUFFIN) -> content 1, parquet/orc format, [lower,upper] bounds decoded from the
1560
1561
// delete file's DELETE_FILE_POS bounds. MUTATION: wrong content id / dropped bounds / wrong format -> red.
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java
+14Lines changed: 14 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -29,6 +29,7 @@
29
29
importjava.util.List;
30
30
importjava.util.Map;
31
31
importjava.util.concurrent.Callable;
32
+
importjava.util.function.UnaryOperator;
32
33
33
34
/**
34
35
* Hand-written {@link ConnectorContext} test double (no Mockito), adapted verbatim from the paimon
@@ -60,6 +61,9 @@ final class RecordingConnectorContext implements ConnectorContext {
0 commit comments