Repository navigation
Commit 11ff064
[fix](be-java-extensions) address the fourth review round of the plugin isolation change
Forty-five findings from the c1aa5b1 review. Three of them are answered rather
than changed and are listed at the end.
BLOCKER
- F-01 JvmStats::init() took its env through JvmLauncher::attach_current_thread(),
which calls ensure_jvm() - and init() is reached from _bootstrap(), inside
ensure_jvm()'s own call_once. Every BE built from the previous commit would hang
on the first JVM it created, on the first Java query or the first libhdfs read.
New JvmLauncher::ScopedVmEnv attaches WITHOUT ensure_jvm() and, for the life of
the guard, primes Jni::Env::Get()'s thread-local env, restoring what it found
rather than clearing it (the bootstrap thread has one). The comment above
init_jvm_metrics() said the position before reset_tls_env() was what kept this
safe; it never was, since the metrics daemon reruns the same code on a thread
that was never primed - rewritten to say what actually holds it.
MAJOR
- F-05 The same guard closes the other half. JvmStats::refresh() entered off the
SPI gate but everything it allocated was released through it, because the RAII
wrappers ask Env::Get() for an env of their own: on a BE whose doris-jni-spi.jar
did not resolve, each destructor logged "Can't destroy Jni Ref" and returned
without deleting. Holding the guard across the whole body puts the body and
every destructor on the fast path. Was masked by F-01 - no JVM, no refresh.
- F-02/F-03/F-06/F-09 releaseSupersededScopes() decided what to close by "belongs
to my catalog id and is not my key", which has no ordering: fileSystemScope() is
per-instance and lazy, so an ALTER-superseded connector computing its scope late
would close the live one's filesystems and orphan its own entry forever. Replaced
by a reference count. FS_SCOPES maps a configuration digest - no catalog id
prefix, restoring the contract fileSystemScopeKey() documents - to (ugi, owners);
fileSystemScope() takes one hold, close() gives it back, and the last holder
closes the filesystems. Every path that ends a connector goes through close()
(ALTER and CREATE-time re-registration via ExternalCatalog.resetToUninitialized,
DROP via closeResources, and the CatalogFactory throwaway via
initLocalObjectsImpl), which is also what F-06 needed: a dropped catalog used to
release nothing at all. REFRESH CATALOG does NOT rebuild the connector - it goes
through RefreshManager.onRefreshCache - so the two comments claiming it does are
corrected. The in-flight window close() has is now stated honestly instead of
being argued away: it is the same window hmsClient.close() has always had.
New HudiConnectorFileSystemScopeTest, seven cases; mutation verified (ignore the
count -> two cases fail).
- F-04 hadoop-aws's AssumedRoleCredentialProvider resolves
software.amazon.awssdk.services.sts.* from its constant pool, and Doris names
that class by string whenever a catalog carries s3.role_arn - also for
web_identity, container and the default provider chain an EKS/IRSA deployment
lands on. hudi and paimon declared s3, apache-client and s3-transfer-manager but
not sts; iceberg did. Declared in both. Not kms: CSE-KMS additionally needs
software.amazon.encryption.s3, which no plugin and no preload classpath ever had.
MINOR
- F-24 The closure gate walked from Doris's own jars only, so it never entered
hadoop-aws or hadoop-huaweicloud and could not see either of the two shipped
holes above. The jdeps root set is now Doris's jars PLUS every jar holding a
class Doris writes into a hadoop Configuration (fs.*.impl and the two
credentials-provider keys, eight class names from fe/fe-filesystem/**). Run
against a built output tree it reports exactly sts for hudi and paimon and
commons-lang for all three, and nothing else after the allowlist below. This is
not reflection analysis: every reference it then finds is an ordinary
constant-pool entry. The new CLOSURE_ALLOWLIST entries - CSE-KMS, the AWS SDK's
shaded httpclient, the SDK v1 adapter, the log4j1 bridge, OBS's okhttp platform
probe, Guava's removed CheckedFuture, the S3A committers - were each checked
with javap against the deployed jar rather than assumed. _names is memoized,
which the api-version docstring claimed already (F-42).
- F-12/F-33/F-36 jni_plugin_fs_dir was honoured by the JVM half only.
bin/start_be.sh now reads it out of be.conf the way set_tcmalloc_heap_limit reads
mem_limit (the export loop above takes UPPERCASE keys only), and enumerates every
subdirectory holding jars, which is what PluginRuntime does - so the two readers
of "one directory" really do see the same jars. The two deprecated locations keep
their jar-presence fallback and WARN. scan_class_path() reads the config too, and
loses the "cannot consult config" justification, which was false: it runs from
_build_options(), well after config load.
- F-13 checksumVerifier("") is null, so a catalog created without
jdbc_driver_checksum wrote no LOADED_UNDER entry; declaring a checksum later then
read previous == null and skipped the discard, serving the OLD bytes for the life
of the process - the case the discard exists for. LOADED_UNDER is written
whenever a loader is created, with an UNDECLARED sentinel. F-29: the locked path
re-reads the cache after verifyOnce, as the fast path already did and documented.
- F-11 The sibling-delegation completeness lock was one-way: it fires when a
delegation is dropped and says nothing when one is added, which is how
listsPartitionsAtSnapshot - the path a hudi-on-HMS table takes for every
time-travel query - reached this branch undriven. Added a source scan asserting
every method of HiveConnectorMetadata that forwards to a sibling is classified,
and drove the two that were not (listsPartitionsAtSnapshot, listFileSizes).
Mutation verified: an unclassified forwarder fails the case by name.
- F-16 FROZEN_TYPES was hand-written, so adding a public top-level SPI type - a
MAJOR change by this file's own contract - kept the gate green. Derived from the
built artifact instead. Byte-identical to the recorded baseline; mutation
verified with a new public interface in the SPI package.
- F-14 The BE class-name gate read jni_plugin_registry.cpp only. The two classes
Util::_init_jni_base() resolves inline - JniUtil and JNINativeMethod, both
rewritten by this PR's package moves - are now scanned as well. A wrong name
there fails the base, which disables every Java feature at once.
- F-15 "doris_be_test creates no JVM" had no enforcement left after the assertion
became a GTEST_SKIP. A global test environment asserts it once, after every case,
where the question is decidable; skipped when disabled cases are being run.
- F-21 The new null guard in GetJniExceptionMsg checked two of the three statics it
can dereference. throwable_to_stack_trace_id_ is assigned last, so a failure at
the last step of Env::Init() left the guard passing and log_stack (default true)
calling CallStaticObjectMethod with a null jmethodID.
- F-22 The comment protecting ensure_jni_base() contradicted its own declared
friend. The invariant that actually makes GetJNIEnvSlowPath safe - the bootstrap
always completed the call_once first - is written down and pinned with a DCHECK.
- F-17 The JindoFS duplicate-class exemptions outlived the copy loop this commit's
predecessor deleted, permanently waving through ten class paths in iceberg and
paimon. Removed.
- F-18 The docker fixture looked for JuiceFS under output/be/lib/juicefs; this head
deploys plugins/jni_fs/juicefs. Added, old globs kept for already-deployed CI
clusters.
- F-19 run-fs-benchmark.sh and run-task-executor-simulator.sh were rewritten for
the new layout in the same commit that gave start_be.sh -Dlog4j2.configurationFile
and did not get it, so both lost hadoop's INFO and WARN.
- F-20 build.sh summarised the layout gate as "every plugin's dependency closure is
complete". Rewritten to what it checks, with what it does not see named:
plugins/jni_fs, and anything reached by a ServiceLoader.
- F-08 The scope machinery had no test at all; it now has seven. F-10, F-26 The
enableFileSystemCache javadoc argued from one call site: the scan-planning one
runs with no doAs, so its filesystems are cached under the login user and are not
covered by the release - now stated, along with the silent override of an
operator's own disable.cache flag. F-25 Every default-path test had moved to the
two-argument seam; two cases drive the production one-argument form again.
NITS
- F-27, F-28 Two stale claims: the aggregate WARN this commit's predecessor added,
and "the same view call the scan uses" (it is not, and listPartitionPathsAsOf's
own javadoc already said so). F-30 The shared-jar count and package were wrong -
jindo-sdk carries 19 hadoop classes, juicefs 4, and none of juicefs's are in
org.apache.hadoop.fs. F-31 A missing or empty plugins/jni_fs left no trace on the
Java side, which is what "jfs:// stopped working after the upgrade" looks like
from in there; both now log at INFO. F-32 PROTOCOL.md still said a plugin's
classpath is the jars in its directory. F-34 The -Xmx boundary case asserted
-Xmx8g, so every wrong threshold up to 8589934591 stayed green; the boundary is
named exactly now. F-35 A comment credited a reordering with protecting "both
metrics daemons", which no longer come through there. F-37 The wasb:// assertion
proves resolution, not linking - said so rather than leaving false confidence.
F-38 An in-place upgrade keeps conf/, so the new hadoop logging configuration
never lands on exactly the deployments most likely to want it; start_be.sh now
says so. F-39 The bthread branch of _create_hdfs_fs ran the JVM bootstrap on an
anonymous std::thread with no thread context - named hdfs_fs_connect, with
SCOPED_INIT_THREAD_CONTEXT. F-40 The healthy shape of /api/jni_plugin_status was
the only one of four missing javaSupport and registryInitialized. F-43 hudi's
commons-lang was scoped to the obs profile, but hive-apache-shade's
ValidReadTxnList references the UNRELOCATED org.apache.commons.lang and the jar
ships neither - moved to the main dependencies for hudi only; iceberg and paimon
are correct as they are. F-44, F-45 Both are documented where the code is, since
the claims they correct live in the release note: the FE still writes
trinoconnector%g.log, and EXPIRATION_TIME is accepted but no longer acted on.
ANSWERED RATHER THAN CHANGED
- F-23 Adding kms next to sts would be ineffective - not added, as the finding
itself recommends.
- F-41 Does not reproduce. <optional> in jni-util.h is already in the same block as
<atomic>, <cstdlib> and <string>; the only standalone one in the diff is in
jni_plugin_status_action.cpp, separated from <fmt/format.h>, which is the correct
grouping.
VERIFICATION
BE: the eight changed translation units and the two changed test files pass
-fsyntax-only against the real compile flags (tests with -fno-access-control, as
doris_be_test is built). No BE unit tests were executed. FE: BUILD SUCCESS across
the touched modules, 0 checkstyle violations, every affected suite green. The three
changed shell scripts pass bash -n, and start_be.sh's new block was run against four
fixtures (default, custom shared path, a value written with ${DORIS_HOME}, and no
directory at all). check_plugin_layout.py was run against a previously built output
tree; the two families it still reports there are the ones the pom changes fix.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01JsphGANprUiPXGCPQMWVmn1 parent 0c4ccf7 commit 11ff064
34 files changed
Lines changed: 1336 additions & 255 deletions
File tree
- be
- src
- common
- metrics
- io/fs/hdfs
- service/http/action
- util
- test/util
- bin
- docker/thirdparties
- fe
- be-java-extensions
- hadoop-hudi-scanner
- src/test/java/org/apache/doris/hudi
- iceberg-metadata-scanner
- jni-bootstrap/src
- main/java/org/apache/doris/jni/bootstrap
- test/java/org/apache/doris/jni/bootstrap
- jni-spi
- src/test/java/org/apache/doris/jni/spi
- paimon-scanner
- plugin-toolkit/src
- main/java/org/apache/doris/jni/toolkit/jdbc
- test/java/org/apache/doris/jni/toolkit/jdbc
- fe-connector
- fe-connector-hive/src/test/java/org/apache/doris/connector/hive
- fe-connector-hudi/src
- main/java/org/apache/doris/connector/hudi
- test/java/org/apache/doris/connector/hudi
- fe-connector-spi/src/main/java/org/apache/doris/connector/spi
- fe-connector-trino/src/main/java/org/apache/doris/connector/trino
- fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands
- tools/be-java-plugins
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1693 | 1693 | | |
1694 | 1694 | | |
1695 | 1695 | | |
1696 | | - | |
| 1696 | + | |
| 1697 | + | |
| 1698 | + | |
| 1699 | + | |
1697 | 1700 | | |
1698 | 1701 | | |
1699 | 1702 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
39 | 40 | | |
40 | 41 | | |
41 | 42 | | |
42 | | - | |
43 | | - | |
44 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
45 | 56 | | |
46 | 57 | | |
47 | 58 | | |
| |||
225 | 236 | | |
226 | 237 | | |
227 | 238 | | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
228 | 242 | | |
229 | | - | |
| 243 | + | |
230 | 244 | | |
231 | 245 | | |
232 | 246 | | |
| |||
325 | 339 | | |
326 | 340 | | |
327 | 341 | | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
328 | 345 | | |
329 | | - | |
| 346 | + | |
330 | 347 | | |
331 | 348 | | |
332 | 349 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
33 | 34 | | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| |||
223 | 225 | | |
224 | 226 | | |
225 | 227 | | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
226 | 237 | | |
227 | 238 | | |
228 | 239 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
| 87 | + | |
| 88 | + | |
88 | 89 | | |
89 | 90 | | |
90 | 91 | | |
| |||
139 | 140 | | |
140 | 141 | | |
141 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
142 | 151 | | |
143 | 152 | | |
144 | 153 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
69 | | - | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
70 | 81 | | |
71 | 82 | | |
72 | 83 | | |
| |||
90 | 101 | | |
91 | 102 | | |
92 | 103 | | |
93 | | - | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
94 | 112 | | |
95 | 113 | | |
96 | 114 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
149 | 153 | | |
150 | 154 | | |
151 | 155 | | |
| |||
1233 | 1237 | | |
1234 | 1238 | | |
1235 | 1239 | | |
1236 | | - | |
1237 | | - | |
1238 | | - | |
1239 | | - | |
| 1240 | + | |
| 1241 | + | |
| 1242 | + | |
| 1243 | + | |
| 1244 | + | |
| 1245 | + | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
1240 | 1255 | | |
1241 | 1256 | | |
1242 | 1257 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
101 | 103 | | |
102 | 104 | | |
103 | 105 | | |
| |||
387 | 389 | | |
388 | 390 | | |
389 | 391 | | |
390 | | - | |
391 | | - | |
392 | | - | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
393 | 399 | | |
394 | 400 | | |
395 | 401 | | |
| |||
474 | 480 | | |
475 | 481 | | |
476 | 482 | | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
477 | 516 | | |
478 | 517 | | |
479 | 518 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
65 | 99 | | |
66 | 100 | | |
67 | 101 | | |
| |||
78 | 112 | | |
79 | 113 | | |
80 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
81 | 120 | | |
82 | 121 | | |
83 | 122 | | |
| |||
0 commit comments