You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2dc1fc2
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](be-java-extensions) address the second 2026-09-14 review round
Four P1s, all about the hadoop FileSystem cache: one key, three
lifetimes.
Isolated plugins lost the credential-aware cache key. A plugin's
classloader has the platform loader as its parent and bundles a vanilla
hadoop-common, so org.apache.hadoop.fs.FileSystem inside paimon,
iceberg and hudi was hadoop's own. Its Cache.Key ignores the
doris.fs.cache.key.<scheme> fingerprint FE tags every storage map with,
and every simple-auth scan runs under one UGI per hadoop.username
(PreExecutionAuthenticatorCache), so two catalogs reaching the same
scheme://authority with different credentials or namenode addresses
shared one cached filesystem and read through whichever was built
first. On master the patched class reached the scanners from the system
classpath, where start_be.sh puts hadoop-deps.jar first; a plugin
directory has no such line. Each of the three plugins now declares
hadoop-deps - transitives excluded, so only the patched FileSystem
joins the closure - and the jar's new Doris-Shadows-Classes manifest
entry tells PluginRuntime to search it before every other jar in the
directory (by name, hadoop-common-*.jar would win). The layout check
holds the attribute to its word: a shadowing jar must carry exactly the
classes it names, and only those may duplicate a sibling's. Declared
ahead of hadoop-common so surefire runs on the same copy; each plugin
test asserts two fingerprints key two cache entries, and loading the
deployed directories through PluginRuntime shows all three plugins
resolving FileSystem from hadoop-deps.
The FE planning path opened filesystems nobody could close.
HudiScanPlanProvider.planScan and getScanNodeProperties built their
metaClients under whatever UGI the planning thread had - the FE login
user, since the engine only pins the classloader - so their filesystems
were cached where HudiConnector.close(), which releases the connector's
own scope UGI, never looked; every altered or rotated catalog
configuration left its S3 client and executor threads behind until FE
restart. The provider now runs both inside the connector's
execute-wrapper (unwrapped, so its own failures still reach the engine
as they are), the scope test pins that the executor it is handed runs
under the connector's scope, and a provider test with a real empty
table pins that the metaClient's filesystem is opened under the
executor's UGI.
A pinned hive-sync read was pruned against today's HMS.
listsPartitionsAtSnapshot answers false for use_hive_sync_partition so
the engine will not prune a FOR TIME AS OF read against a listing that
can only shrink, but applyFilter still pruned partition predicates
against hmsClient.listPartitionNames and stored the result, which
applySnapshot preserves and resolvePartitions reads as the scan set:
a partition dropped and unsynced since the pin turned
`FOR TIME AS OF '<old>' WHERE part1 = 'GONE'` into zero rows. A pinned
read now prunes the Hudi metadata listing instead - the universe the
unpruned pinned scan walks itself, so pruning it can never lose a
partition the scan would have read, and it costs no extra listing
because resolvePartitions short-circuits on the pruned set. The p2
dropped-partition case gains the predicate reads.
The BE hudi scanner's filesystem scopes were never evicted. Each
distinct hadoop_conf.* map got a UGI of its own, under which the scanner
turned the cache back on, and the map only grew: credential rotation
and CREATE/query/DROP cycles turned the old per-query leak into a
per-configuration one. HudiFileSystemScopes now counts the scanners
holding each scope - acquired in open, released in close, so a failed
open lets go too, and a close that runs twice releases once - and a
sweeper closes a scope's filesystems through FileSystem.closeAllForUGI,
the one handle Hadoop offers, once it has had no holder for ten
minutes. Removal happens under the acquire lock and the close outside
it, so a scan either holds the entry or gets a fresh UGI whose cache
entries the close cannot touch; the race is exercised in the test.
With the patched FileSystem in the plugin the UGI no longer carries the
credential separation, only the handle to close with.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01222rs932qTF8BjpDrYjUEj
0 commit comments