Skip to content

Commit 2dc1fc2

Browse files
morningmanclaude
andcommitted
[fix](be-java-extensions) address the second 2026-09-14 review round
Four P1s, all about the hadoop FileSystem cache: one key, three lifetimes. Isolated plugins lost the credential-aware cache key. A plugin's classloader has the platform loader as its parent and bundles a vanilla hadoop-common, so org.apache.hadoop.fs.FileSystem inside paimon, iceberg and hudi was hadoop's own. Its Cache.Key ignores the doris.fs.cache.key.<scheme> fingerprint FE tags every storage map with, and every simple-auth scan runs under one UGI per hadoop.username (PreExecutionAuthenticatorCache), so two catalogs reaching the same scheme://authority with different credentials or namenode addresses shared one cached filesystem and read through whichever was built first. On master the patched class reached the scanners from the system classpath, where start_be.sh puts hadoop-deps.jar first; a plugin directory has no such line. Each of the three plugins now declares hadoop-deps - transitives excluded, so only the patched FileSystem joins the closure - and the jar's new Doris-Shadows-Classes manifest entry tells PluginRuntime to search it before every other jar in the directory (by name, hadoop-common-*.jar would win). The layout check holds the attribute to its word: a shadowing jar must carry exactly the classes it names, and only those may duplicate a sibling's. Declared ahead of hadoop-common so surefire runs on the same copy; each plugin test asserts two fingerprints key two cache entries, and loading the deployed directories through PluginRuntime shows all three plugins resolving FileSystem from hadoop-deps. The FE planning path opened filesystems nobody could close. HudiScanPlanProvider.planScan and getScanNodeProperties built their metaClients under whatever UGI the planning thread had - the FE login user, since the engine only pins the classloader - so their filesystems were cached where HudiConnector.close(), which releases the connector's own scope UGI, never looked; every altered or rotated catalog configuration left its S3 client and executor threads behind until FE restart. The provider now runs both inside the connector's execute-wrapper (unwrapped, so its own failures still reach the engine as they are), the scope test pins that the executor it is handed runs under the connector's scope, and a provider test with a real empty table pins that the metaClient's filesystem is opened under the executor's UGI. A pinned hive-sync read was pruned against today's HMS. listsPartitionsAtSnapshot answers false for use_hive_sync_partition so the engine will not prune a FOR TIME AS OF read against a listing that can only shrink, but applyFilter still pruned partition predicates against hmsClient.listPartitionNames and stored the result, which applySnapshot preserves and resolvePartitions reads as the scan set: a partition dropped and unsynced since the pin turned `FOR TIME AS OF '<old>' WHERE part1 = 'GONE'` into zero rows. A pinned read now prunes the Hudi metadata listing instead - the universe the unpruned pinned scan walks itself, so pruning it can never lose a partition the scan would have read, and it costs no extra listing because resolvePartitions short-circuits on the pruned set. The p2 dropped-partition case gains the predicate reads. The BE hudi scanner's filesystem scopes were never evicted. Each distinct hadoop_conf.* map got a UGI of its own, under which the scanner turned the cache back on, and the map only grew: credential rotation and CREATE/query/DROP cycles turned the old per-query leak into a per-configuration one. HudiFileSystemScopes now counts the scanners holding each scope - acquired in open, released in close, so a failed open lets go too, and a close that runs twice releases once - and a sweeper closes a scope's filesystems through FileSystem.closeAllForUGI, the one handle Hadoop offers, once it has had no holder for ten minutes. Removal happens under the acquire lock and the close outside it, so a scan either holds the entry or gets a fresh UGI whose cache entries the close cannot touch; the race is exercised in the test. With the patched FileSystem in the plugin the UGI no longer carries the credential separation, only the handle to close with. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01222rs932qTF8BjpDrYjUEj
1 parent 8be943a commit 2dc1fc2

27 files changed

Lines changed: 1600 additions & 130 deletions

File tree

‎fe/be-java-extensions/hadoop-deps/pom.xml‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,13 @@ under the License.
3737
whole into lib/hadoop_hdfs/ instead.
3838

3939
Consequences worth knowing before editing:
40-
- It is invisible to every plugin. A plugin that reads through hadoop carries its own copy
41-
of hadoop-common and its own filesystem implementations; nothing here reaches it.
40+
- The DEPLOYMENT is invisible to every plugin. A plugin that reads through hadoop carries
41+
its own copy of hadoop-common and its own filesystem implementations; nothing in
42+
lib/hadoop_hdfs reaches it. The module JAR is a different matter: it holds nothing but
43+
the Doris-patched org.apache.hadoop.fs.FileSystem (the credential-aware cache key, see
44+
the class header), and every plugin that bundles hadoop-common declares this artifact
45+
so that copy lands in its plugin directory too - ahead of hadoop-common's, which is
46+
what the Doris-Shadows-Classes manifest attribute below arranges.
4247
- Which schemes actually arrive here is decided in FE. StorageRegistry maps s3a/s3n/cosn/
4348
oss/obs/cos/gs/abfs*/wasb* to the native C++ reader, so only hdfs/viewfs/jfs/local, the
4449
empty scheme, oss-dls (JindoFS) and OneLake's abfs (hadoop-azure) come through libhdfs.
@@ -169,6 +174,22 @@ under the License.
169174
<artifactId>maven-jar-plugin</artifactId>
170175
<configuration>
171176
<finalName>${project.artifactId}</finalName>
177+
<archive>
178+
<manifestEntries>
179+
<!--
180+
This jar exists to be found AHEAD of hadoop-common's copy of the class it
181+
patches. On the system classpath that is a matter of order in start_be.sh /
182+
start_fe.sh / cloud's start.sh. Inside a plugin directory there is no such
183+
script: PluginRuntime sorts the jars by name (hadoop-common-*.jar would win),
184+
and this attribute is what tells it to search this jar first. The value names
185+
what is shadowed; the layout check (tools/be-java-plugins/
186+
check_plugin_layout.py) verifies the jar carries exactly those classes and
187+
lets only them duplicate another jar's. Merged with the parent's
188+
Doris-Jni-Plugin-Api-Version entry, not replacing it.
189+
-->
190+
<Doris-Shadows-Classes>org.apache.hadoop.fs.FileSystem</Doris-Shadows-Classes>
191+
</manifestEntries>
192+
</archive>
172193
</configuration>
173194
</plugin>
174195
<plugin>

‎fe/be-java-extensions/hadoop-deps/src/main/java/org/apache/hadoop/fs/FileSystem.java‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,17 @@
1919
// DORIS-PATCH: this file is a verbatim copy of org.apache.hadoop.fs.FileSystem
2020
// from hadoop-common 3.4.2 (sources jar). This class shadows the vanilla one
2121
// in BOTH JVMs, always ahead of hadoop-common on the classpath:
22-
// - BE: build.sh copies hadoop-deps.jar to be/lib/hadoop_hdfs/, which
23-
// bin/start_be.sh adds before be/lib/hadoop_hdfs/lib/*.jar, covering the
24-
// whole BE JVM including all JNI scanners.
22+
// - BE, system classpath: build.sh copies hadoop-deps.jar to
23+
// be/lib/hadoop_hdfs/, which bin/start_be.sh adds before
24+
// be/lib/hadoop_hdfs/lib/*.jar. That is what C++ libhdfs runs on.
25+
// - BE, Java plugins: a plugin classloader cannot see the system classpath
26+
// and bundles its own hadoop-common, so every plugin that does (paimon,
27+
// iceberg, hudi) also declares this module and gets this jar in its
28+
// plugin directory. The jar's Doris-Shadows-Classes manifest entry
29+
// (hadoop-deps/pom.xml) makes PluginRuntime search it before the rest.
2530
// - FE: fe-core depends on this module (transitives excluded), so the jar
26-
// lands in fe/lib/, and bin/start_fe.sh prepends it explicitly.
31+
// lands in fe/lib/, and bin/start_fe.sh prepends it explicitly; plugin
32+
// classloaders resolve org.apache.hadoop.* parent-first, so they see it.
2733
//
2834
// The only functional change: Cache.Key carries one extra dimension read from
2935
// the configuration property "doris.fs.cache.key". FE injects a per-catalog /

‎fe/be-java-extensions/hadoop-hudi-scanner/pom.xml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,34 @@ under the License.
198198
<version>${project.version}</version>
199199
</dependency>
200200

201+
<!--
202+
The Doris-patched org.apache.hadoop.fs.FileSystem, to be found AHEAD of hadoop-common's
203+
copy. Its Cache.Key carries the credential fingerprint FE tags every storage map with
204+
(doris.fs.cache.key.<scheme>); the vanilla class ignores that property, and this
205+
plugin's simple-auth scans all run under one UGI per hadoop.username
206+
(PreExecutionAuthenticatorCache), so on the vanilla key two catalogs reaching the same
207+
scheme://authority with different credentials or namenode addresses would share one
208+
cached filesystem and read through whichever was built first. Before this module was a
209+
plugin the patched class arrived from BE's system classpath, where start_be.sh puts
210+
hadoop-deps.jar first; a plugin cannot see that classpath, so it carries the jar itself,
211+
and the jar's Doris-Shadows-Classes manifest entry is what makes PluginRuntime search it
212+
before hadoop-common (by name, hadoop-common-*.jar would win). Declared BEFORE
213+
hadoop-common because surefire's classpath follows declaration order, so the tests run
214+
on the copy the deployed plugin runs on - HadoopHudiPluginTest asserts that.
215+
Every transitive is excluded: they are the libhdfs runtime that module deploys to
216+
lib/hadoop_hdfs, and this plugin declares the hadoop artifacts it needs itself.
217+
-->
218+
<dependency>
219+
<groupId>org.apache.doris</groupId>
220+
<artifactId>hadoop-deps</artifactId>
221+
<version>${project.version}</version>
222+
<exclusions>
223+
<exclusion>
224+
<groupId>*</groupId>
225+
<artifactId>*</artifactId>
226+
</exclusion>
227+
</exclusions>
228+
</dependency>
201229
<!--
202230
Hadoop, declared here rather than taken transitively so this plugin has exactly one copy
203231
at depth one. Until this module became a plugin it declared hadoop provided and let BE's

‎fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java‎

Lines changed: 79 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,6 @@
4343
import org.apache.hadoop.mapred.JobConf;
4444
import org.apache.hadoop.mapred.RecordReader;
4545
import org.apache.hadoop.mapred.Reporter;
46-
import org.apache.hadoop.security.UserGroupInformation;
4746
import org.apache.hadoop.util.ReflectionUtils;
4847
import org.apache.hudi.common.model.HoodieLogFile;
4948
import org.apache.hudi.common.util.Option;
@@ -64,7 +63,6 @@
6463
import java.util.Properties;
6564
import java.util.TreeMap;
6665
import java.util.concurrent.Callable;
67-
import java.util.concurrent.ConcurrentHashMap;
6866
import java.util.regex.Pattern;
6967
import java.util.stream.Collectors;
7068
import java.util.stream.IntStream;
@@ -80,12 +78,6 @@ public class HadoopHudiJniScanner extends JniScanner {
8078
// fs.s3a.impl.disable.cache and its per-scheme siblings, as the FE emits them.
8179
private static final Pattern FS_DISABLE_CACHE = Pattern.compile("fs\\..+\\.impl\\.disable\\.cache");
8280

83-
// One UGI per distinct filesystem configuration, which is what keys Hadoop's FileSystem cache to
84-
// the credentials that opened it. See createFileSystemScope. Never evicted on purpose: an entry is
85-
// one UGI, there is one per catalog storage config, and dropping one would strand the filesystems
86-
// cached under it - a live scan may still be reading through them.
87-
private static final ConcurrentHashMap<String, UserGroupInformation> FS_SCOPES = new ConcurrentHashMap<>();
88-
8981
// Hudi data info
9082
private final String basePath;
9183
private final String dataFilePath;
@@ -115,8 +107,12 @@ public class HadoopHudiJniScanner extends JniScanner {
115107
private final int fetchSize;
116108

117109
private final PreExecutionAuthenticator preExecutionAuthenticator;
118-
// Null when this scanner does not own its filesystems; see createFileSystemScope.
119-
private final UserGroupInformation fileSystemScope;
110+
// The configuration this scanner reads under, digested, and the user name its scope carries;
111+
// both null when the scanner does not own its filesystems. See fileSystemScopeKey.
112+
private final String fileSystemScopeKey;
113+
private final String fileSystemScopeUser;
114+
// Held from open to close; null outside that window. See HudiFileSystemScopes.
115+
private HudiFileSystemScopes.Hold fileSystemScope;
120116

121117
public HadoopHudiJniScanner(int fetchSize, Map<String, String> params) {
122118
this.basePath = params.get("base_path");
@@ -152,10 +148,12 @@ public HadoopHudiJniScanner(int fetchSize, Map<String, String> params) {
152148
}
153149
}
154150
this.preExecutionAuthenticator = PreExecutionAuthenticatorCache.getAuthenticator(fsOptionsProps);
155-
this.fileSystemScope = createFileSystemScope();
156-
if (fileSystemScope != null) {
151+
this.fileSystemScopeUser = fileSystemScopeUser();
152+
this.fileSystemScopeKey = fileSystemScopeUser == null ? null : fileSystemScopeKey(fsOptionsProps);
153+
if (fileSystemScopeKey != null) {
157154
// Hadoop's FileSystem cache is the only thing standing between this scanner and a
158-
// filesystem leaked on every get(); the scope above is what makes it safe to use.
155+
// filesystem leaked on every get(); the scope acquired in openInternal is what makes the
156+
// cached filesystems closable again.
159157
fsOptionsProps.replaceAll((key, value) -> FS_DISABLE_CACHE.matcher(key).matches() ? "false" : value);
160158
}
161159

@@ -171,44 +169,40 @@ public HadoopHudiJniScanner(int fetchSize, Map<String, String> params) {
171169
}
172170

173171
/**
174-
* The {@link UserGroupInformation} this scanner reads under, chosen so that Hadoop's
175-
* {@link FileSystem} cache can stay ON without letting two catalogs share one filesystem.
176-
*
177-
* <p>The FE hands every S3-compatible catalog {@code fs.s3a.impl.disable.cache=true} (the HDFS
178-
* builder does the same for {@code fs.hdfs.}), because that cache is keyed on
179-
* (scheme, authority, ugi) and ignores credentials - and every non-Kerberos catalog arrives here
180-
* under the SAME ugi, {@code HadoopSimpleAuthenticator}'s {@code createRemoteUser(hadoop.username)},
181-
* whose own cache key is just that user name. Two catalogs on one bucket would otherwise read
182-
* through whichever S3AFileSystem was built first, with its credentials.
172+
* The user name of the {@code UserGroupInformation} this scanner reads under, or null when it
173+
* reads under the authenticator's own.
183174
*
184-
* <p>Disabling the cache does stop that, and leaks instead. Every {@code FileSystem.get()} inside
185-
* hudi-hadoop-mr then builds a fresh S3AFileSystem and a fresh AWS SDK client, and nobody closes
186-
* them: one query reaches for the base file, each log file and the timeline through some two dozen
187-
* of those calls. The filesystems are collected, but each SDK client leaves a scheduled executor
188-
* that its own worker threads keep alive - about 118 threads per query, measured - until the JVM
189-
* cannot start a thread and the BE aborts with
190-
* {@code std::system_error: thread constructor failed}.
175+
* <p>Hadoop's {@link FileSystem} cache stays ON in this scanner, and that is a lifecycle decision
176+
* before it is anything else. The FE used to hand every S3-compatible catalog
177+
* {@code fs.s3a.impl.disable.cache=true}, and with the cache off every {@code FileSystem.get()}
178+
* inside hudi-hadoop-mr built a fresh S3AFileSystem and a fresh AWS SDK client that nobody closed:
179+
* one query reaches for the base file, each log file and the timeline through some two dozen of
180+
* those calls, and each SDK client leaves a scheduled executor its own worker threads keep alive
181+
* - about 118 threads per query, measured - until the JVM cannot start a thread and the BE aborts
182+
* with {@code std::system_error: thread constructor failed}.
191183
*
192-
* <p>So the cache goes back on, and the credential separation it was disabled for is provided by
193-
* the cache key itself: one UGI per distinct filesystem configuration, from {@link #FS_SCOPES}.
194-
* Same credentials reuse a filesystem, different credentials cannot. The UGI carries the SAME user
195-
* name the simple authenticator would have used - it is a second Subject for one user, not another
196-
* user - so an {@code hdfs://} warehouse still sees the identity it always saw.
184+
* <p>With the cache on, what keeps two catalogs from sharing one filesystem is the cache key. This
185+
* plugin carries the Doris-patched {@code FileSystem} (hadoop-deps), whose key includes the
186+
* {@code doris.fs.cache.key.<scheme>} fingerprint FE derives from each catalog's storage
187+
* configuration - so same credentials reuse a filesystem and different ones cannot, whichever UGI
188+
* the scan runs under. What the per-configuration UGI adds is a handle to CLOSE with: the cached
189+
* filesystems are keyed on it, {@code FileSystem.closeAllForUGI} is the only way Hadoop offers to
190+
* let a set of them go, and {@link HudiFileSystemScopes} closes each configuration's set once no
191+
* scanner has read through it for a while. The UGI carries the SAME user name the simple
192+
* authenticator would have used - it is a second Subject for one user, not another user - so an
193+
* {@code hdfs://} warehouse still sees the identity it always saw.
197194
*
198-
* <p>Kerberos is the case this cannot serve: {@code createRemoteUser} would drop the credentials the
199-
* ticket carries. There the scanner keeps the authenticator's own UGI, which is already cached per
200-
* principal and so already partitions the filesystem cache; this returns null and the FE's setting
201-
* is left alone.
195+
* <p>Kerberos is the case this cannot serve: {@code createRemoteUser} would drop the credentials
196+
* the ticket carries. There the scanner keeps the authenticator's own UGI, which is cached per
197+
* principal, and the FE's cache setting is left alone; this returns null.
202198
*/
203-
private UserGroupInformation createFileSystemScope() {
199+
private String fileSystemScopeUser() {
204200
HadoopAuthenticator authenticator = preExecutionAuthenticator.getHadoopAuthenticator();
205201
if (authenticator == null || authenticator instanceof HadoopKerberosAuthenticator) {
206202
return null;
207203
}
208204
try {
209-
String userName = authenticator.getUGI().getUserName();
210-
return FS_SCOPES.computeIfAbsent(fileSystemScopeKey(),
211-
key -> UserGroupInformation.createRemoteUser(userName));
205+
return authenticator.getUGI().getUserName();
212206
} catch (Exception e) {
213207
LOG.warn("failed to derive a FileSystem scope for the hudi scanner, keeping the shared one", e);
214208
return null;
@@ -217,15 +211,23 @@ private UserGroupInformation createFileSystemScope() {
217211

218212
/**
219213
* Identifies a filesystem configuration - endpoint, credentials, everything the FE sent - so that
220-
* {@link #FS_SCOPES} hands the same UGI to two scanners exactly when they may share a filesystem.
221-
* Digested rather than used directly because the properties hold secrets and a map key is easy to
222-
* print by accident.
214+
* {@link HudiFileSystemScopes} hands the same UGI to two scanners exactly when they may share a
215+
* filesystem. Digested rather than used directly because the properties hold secrets and a map key
216+
* is easy to print by accident. Null when the digest cannot be computed, in which case the scanner
217+
* reads under the authenticator's UGI like a Kerberos one. Package-private so a test can name the
218+
* scope a scanner built from a given configuration holds.
223219
*/
224-
private String fileSystemScopeKey() throws NoSuchAlgorithmException {
220+
static String fileSystemScopeKey(Map<String, String> fsOptionsProps) {
225221
StringBuilder canonical = new StringBuilder();
226222
new TreeMap<>(fsOptionsProps).forEach((k, v) -> canonical.append(k).append('=').append(v).append('\n'));
227-
byte[] digest = MessageDigest.getInstance("SHA-256")
228-
.digest(canonical.toString().getBytes(StandardCharsets.UTF_8));
223+
byte[] digest;
224+
try {
225+
digest = MessageDigest.getInstance("SHA-256")
226+
.digest(canonical.toString().getBytes(StandardCharsets.UTF_8));
227+
} catch (NoSuchAlgorithmException e) {
228+
LOG.warn("failed to derive a FileSystem scope for the hudi scanner, keeping the shared one", e);
229+
return null;
230+
}
229231
StringBuilder hex = new StringBuilder(digest.length * 2);
230232
for (byte b : digest) {
231233
hex.append(Character.forDigit((b >> 4) & 0xF, 16)).append(Character.forDigit(b & 0xF, 16));
@@ -241,20 +243,44 @@ private String fileSystemScopeKey() throws NoSuchAlgorithmException {
241243
* {@code createRemoteUser} of the same name, and the simple authenticator adds nothing else.
242244
*/
243245
private <T> T executeInFileSystemScope(Callable<T> task) throws Exception {
244-
if (fileSystemScope == null) {
246+
HudiFileSystemScopes.Hold hold = fileSystemScope;
247+
if (hold == null) {
245248
return preExecutionAuthenticator.execute(task);
246249
}
247250
try {
248-
return fileSystemScope.doAs((PrivilegedExceptionAction<T>) task::call);
251+
return hold.ugi().doAs((PrivilegedExceptionAction<T>) task::call);
249252
} catch (UndeclaredThrowableException e) {
250253
// doAs only wraps checked exceptions it does not declare; the callers report the cause.
251254
Throwable cause = e.getUndeclaredThrowable();
252255
throw cause instanceof Exception ? (Exception) cause : e;
253256
}
254257
}
255258

259+
/**
260+
* Takes this scanner's hold on its configuration's scope. From here until {@link #closeInternal}
261+
* the filesystems it opens are cached under that scope's UGI and cannot be closed from under it;
262+
* see {@link HudiFileSystemScopes}.
263+
*/
264+
private void acquireFileSystemScope() {
265+
if (fileSystemScopeKey != null && fileSystemScope == null) {
266+
fileSystemScope = HudiFileSystemScopes.shared().acquire(fileSystemScopeKey, fileSystemScopeUser);
267+
}
268+
}
269+
270+
/** Gives the hold back. Idempotent, like the close that calls it. */
271+
private void releaseFileSystemScope() {
272+
HudiFileSystemScopes.Hold hold = fileSystemScope;
273+
fileSystemScope = null;
274+
if (hold != null) {
275+
hold.release();
276+
}
277+
}
278+
256279
@Override
257280
protected void openInternal() throws IOException {
281+
// Acquired here and not in the constructor: BE only closes a scanner whose open succeeded, so
282+
// a hold taken before this point would have no release when construction failed midway.
283+
acquireFileSystemScope();
258284
try {
259285
executeInFileSystemScope(() -> {
260286
initRequiredColumnsAndTypes();
@@ -316,6 +342,9 @@ protected void closeInternal() throws IOException {
316342
} catch (IOException e) {
317343
LOG.warn("failed to close hadoop hudi jni scanner", e);
318344
throw new IOException("failed to close hadoop hudi jni scanner: " + e.getMessage(), e);
345+
} finally {
346+
// After the reader: the reader reads through the filesystems the scope keeps open.
347+
releaseFileSystemScope();
319348
}
320349
}
321350

0 commit comments

Comments
 (0)