Skip to content

Commit 11ff064

Browse files
morningmanclaude
andcommitted
[fix](be-java-extensions) address the fourth review round of the plugin isolation change
Forty-five findings from the c1aa5b1 review. Three of them are answered rather than changed and are listed at the end. BLOCKER - F-01 JvmStats::init() took its env through JvmLauncher::attach_current_thread(), which calls ensure_jvm() - and init() is reached from _bootstrap(), inside ensure_jvm()'s own call_once. Every BE built from the previous commit would hang on the first JVM it created, on the first Java query or the first libhdfs read. New JvmLauncher::ScopedVmEnv attaches WITHOUT ensure_jvm() and, for the life of the guard, primes Jni::Env::Get()'s thread-local env, restoring what it found rather than clearing it (the bootstrap thread has one). The comment above init_jvm_metrics() said the position before reset_tls_env() was what kept this safe; it never was, since the metrics daemon reruns the same code on a thread that was never primed - rewritten to say what actually holds it. MAJOR - F-05 The same guard closes the other half. JvmStats::refresh() entered off the SPI gate but everything it allocated was released through it, because the RAII wrappers ask Env::Get() for an env of their own: on a BE whose doris-jni-spi.jar did not resolve, each destructor logged "Can't destroy Jni Ref" and returned without deleting. Holding the guard across the whole body puts the body and every destructor on the fast path. Was masked by F-01 - no JVM, no refresh. - F-02/F-03/F-06/F-09 releaseSupersededScopes() decided what to close by "belongs to my catalog id and is not my key", which has no ordering: fileSystemScope() is per-instance and lazy, so an ALTER-superseded connector computing its scope late would close the live one's filesystems and orphan its own entry forever. Replaced by a reference count. FS_SCOPES maps a configuration digest - no catalog id prefix, restoring the contract fileSystemScopeKey() documents - to (ugi, owners); fileSystemScope() takes one hold, close() gives it back, and the last holder closes the filesystems. Every path that ends a connector goes through close() (ALTER and CREATE-time re-registration via ExternalCatalog.resetToUninitialized, DROP via closeResources, and the CatalogFactory throwaway via initLocalObjectsImpl), which is also what F-06 needed: a dropped catalog used to release nothing at all. REFRESH CATALOG does NOT rebuild the connector - it goes through RefreshManager.onRefreshCache - so the two comments claiming it does are corrected. The in-flight window close() has is now stated honestly instead of being argued away: it is the same window hmsClient.close() has always had. New HudiConnectorFileSystemScopeTest, seven cases; mutation verified (ignore the count -> two cases fail). - F-04 hadoop-aws's AssumedRoleCredentialProvider resolves software.amazon.awssdk.services.sts.* from its constant pool, and Doris names that class by string whenever a catalog carries s3.role_arn - also for web_identity, container and the default provider chain an EKS/IRSA deployment lands on. hudi and paimon declared s3, apache-client and s3-transfer-manager but not sts; iceberg did. Declared in both. Not kms: CSE-KMS additionally needs software.amazon.encryption.s3, which no plugin and no preload classpath ever had. MINOR - F-24 The closure gate walked from Doris's own jars only, so it never entered hadoop-aws or hadoop-huaweicloud and could not see either of the two shipped holes above. The jdeps root set is now Doris's jars PLUS every jar holding a class Doris writes into a hadoop Configuration (fs.*.impl and the two credentials-provider keys, eight class names from fe/fe-filesystem/**). Run against a built output tree it reports exactly sts for hudi and paimon and commons-lang for all three, and nothing else after the allowlist below. This is not reflection analysis: every reference it then finds is an ordinary constant-pool entry. The new CLOSURE_ALLOWLIST entries - CSE-KMS, the AWS SDK's shaded httpclient, the SDK v1 adapter, the log4j1 bridge, OBS's okhttp platform probe, Guava's removed CheckedFuture, the S3A committers - were each checked with javap against the deployed jar rather than assumed. _names is memoized, which the api-version docstring claimed already (F-42). - F-12/F-33/F-36 jni_plugin_fs_dir was honoured by the JVM half only. bin/start_be.sh now reads it out of be.conf the way set_tcmalloc_heap_limit reads mem_limit (the export loop above takes UPPERCASE keys only), and enumerates every subdirectory holding jars, which is what PluginRuntime does - so the two readers of "one directory" really do see the same jars. The two deprecated locations keep their jar-presence fallback and WARN. scan_class_path() reads the config too, and loses the "cannot consult config" justification, which was false: it runs from _build_options(), well after config load. - F-13 checksumVerifier("") is null, so a catalog created without jdbc_driver_checksum wrote no LOADED_UNDER entry; declaring a checksum later then read previous == null and skipped the discard, serving the OLD bytes for the life of the process - the case the discard exists for. LOADED_UNDER is written whenever a loader is created, with an UNDECLARED sentinel. F-29: the locked path re-reads the cache after verifyOnce, as the fast path already did and documented. - F-11 The sibling-delegation completeness lock was one-way: it fires when a delegation is dropped and says nothing when one is added, which is how listsPartitionsAtSnapshot - the path a hudi-on-HMS table takes for every time-travel query - reached this branch undriven. Added a source scan asserting every method of HiveConnectorMetadata that forwards to a sibling is classified, and drove the two that were not (listsPartitionsAtSnapshot, listFileSizes). Mutation verified: an unclassified forwarder fails the case by name. - F-16 FROZEN_TYPES was hand-written, so adding a public top-level SPI type - a MAJOR change by this file's own contract - kept the gate green. Derived from the built artifact instead. Byte-identical to the recorded baseline; mutation verified with a new public interface in the SPI package. - F-14 The BE class-name gate read jni_plugin_registry.cpp only. The two classes Util::_init_jni_base() resolves inline - JniUtil and JNINativeMethod, both rewritten by this PR's package moves - are now scanned as well. A wrong name there fails the base, which disables every Java feature at once. - F-15 "doris_be_test creates no JVM" had no enforcement left after the assertion became a GTEST_SKIP. A global test environment asserts it once, after every case, where the question is decidable; skipped when disabled cases are being run. - F-21 The new null guard in GetJniExceptionMsg checked two of the three statics it can dereference. throwable_to_stack_trace_id_ is assigned last, so a failure at the last step of Env::Init() left the guard passing and log_stack (default true) calling CallStaticObjectMethod with a null jmethodID. - F-22 The comment protecting ensure_jni_base() contradicted its own declared friend. The invariant that actually makes GetJNIEnvSlowPath safe - the bootstrap always completed the call_once first - is written down and pinned with a DCHECK. - F-17 The JindoFS duplicate-class exemptions outlived the copy loop this commit's predecessor deleted, permanently waving through ten class paths in iceberg and paimon. Removed. - F-18 The docker fixture looked for JuiceFS under output/be/lib/juicefs; this head deploys plugins/jni_fs/juicefs. Added, old globs kept for already-deployed CI clusters. - F-19 run-fs-benchmark.sh and run-task-executor-simulator.sh were rewritten for the new layout in the same commit that gave start_be.sh -Dlog4j2.configurationFile and did not get it, so both lost hadoop's INFO and WARN. - F-20 build.sh summarised the layout gate as "every plugin's dependency closure is complete". Rewritten to what it checks, with what it does not see named: plugins/jni_fs, and anything reached by a ServiceLoader. - F-08 The scope machinery had no test at all; it now has seven. F-10, F-26 The enableFileSystemCache javadoc argued from one call site: the scan-planning one runs with no doAs, so its filesystems are cached under the login user and are not covered by the release - now stated, along with the silent override of an operator's own disable.cache flag. F-25 Every default-path test had moved to the two-argument seam; two cases drive the production one-argument form again. NITS - F-27, F-28 Two stale claims: the aggregate WARN this commit's predecessor added, and "the same view call the scan uses" (it is not, and listPartitionPathsAsOf's own javadoc already said so). F-30 The shared-jar count and package were wrong - jindo-sdk carries 19 hadoop classes, juicefs 4, and none of juicefs's are in org.apache.hadoop.fs. F-31 A missing or empty plugins/jni_fs left no trace on the Java side, which is what "jfs:// stopped working after the upgrade" looks like from in there; both now log at INFO. F-32 PROTOCOL.md still said a plugin's classpath is the jars in its directory. F-34 The -Xmx boundary case asserted -Xmx8g, so every wrong threshold up to 8589934591 stayed green; the boundary is named exactly now. F-35 A comment credited a reordering with protecting "both metrics daemons", which no longer come through there. F-37 The wasb:// assertion proves resolution, not linking - said so rather than leaving false confidence. F-38 An in-place upgrade keeps conf/, so the new hadoop logging configuration never lands on exactly the deployments most likely to want it; start_be.sh now says so. F-39 The bthread branch of _create_hdfs_fs ran the JVM bootstrap on an anonymous std::thread with no thread context - named hdfs_fs_connect, with SCOPED_INIT_THREAD_CONTEXT. F-40 The healthy shape of /api/jni_plugin_status was the only one of four missing javaSupport and registryInitialized. F-43 hudi's commons-lang was scoped to the obs profile, but hive-apache-shade's ValidReadTxnList references the UNRELOCATED org.apache.commons.lang and the jar ships neither - moved to the main dependencies for hudi only; iceberg and paimon are correct as they are. F-44, F-45 Both are documented where the code is, since the claims they correct live in the release note: the FE still writes trinoconnector%g.log, and EXPIRATION_TIME is accepted but no longer acted on. ANSWERED RATHER THAN CHANGED - F-23 Adding kms next to sts would be ineffective - not added, as the finding itself recommends. - F-41 Does not reproduce. <optional> in jni-util.h is already in the same block as <atomic>, <cstdlib> and <string>; the only standalone one in the diff is in jni_plugin_status_action.cpp, separated from <fmt/format.h>, which is the correct grouping. VERIFICATION BE: the eight changed translation units and the two changed test files pass -fsyntax-only against the real compile flags (tests with -fno-access-control, as doris_be_test is built). No BE unit tests were executed. FE: BUILD SUCCESS across the touched modules, 0 checkstyle violations, every affected suite green. The three changed shell scripts pass bash -n, and start_be.sh's new block was run against four fixtures (default, custom shared path, a value written with ${DORIS_HOME}, and no directory at all). check_plugin_layout.py was run against a previously built output tree; the two families it still reports there are the ones the pom changes fix. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01JsphGANprUiPXGCPQMWVmn
1 parent 0c4ccf7 commit 11ff064

34 files changed

Lines changed: 1336 additions & 255 deletions

File tree

‎be/src/common/config.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1693,7 +1693,10 @@ DEFINE_String(jni_plugin_hadoop_conf_dir, "${DORIS_HOME}/plugins/hadoop_conf");
16931693
// classpath, AFTER the plugin's jars, so a plugin's own hadoop still wins; each plugin loads its
16941694
// own copy in its own classloader, so the isolation is unchanged. This is also the directory
16951695
// bin/start_be.sh puts on the system class path for the native libhdfs reader, which needs the
1696-
// same jars for the same schemes - one copy on disk serves both.
1696+
// same jars for the same schemes - one copy on disk serves both, and both honour this config:
1697+
// the script reads it out of be.conf by hand (the export loop there only picks up UPPERCASE
1698+
// keys), and JvmLauncher passes it to the JVM as -Ddoris.jni.fs.dir. Every subdirectory holding
1699+
// jars is taken, on both sides.
16971700
//
16981701
// Nothing has to be here: both filesystems are opt-in build flags (DISABLE_BUILD_JINDOFS=OFF,
16991702
// DISABLE_BUILD_JUICEFS=OFF), and a build without them leaves this directory absent.

‎be/src/common/metrics/jvm_metrics.cpp‎

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@
2828
namespace doris {
2929

3030
namespace {
31-
// The env these stats run on, taken from the JVM directly rather than through Jni::Env::Get().
31+
// The env these stats run on, taken from the JVM directly rather than through Jni::Env::Get(),
32+
// and primed as this thread's cached env for as long as the guard lives.
3233
//
3334
// Env::Get() is the gate for code that runs Java of Doris's own, and it refuses whenever the
3435
// plugin SPI could not be resolved. These stats are not that: JvmStats reaches only for
@@ -39,9 +40,19 @@ namespace {
3940
// Env::Get() took its fast path and never asked), while every later refresh() ran on the metrics
4041
// daemon, took the slow path, and got the cached base failure - 30 of those and JvmMetrics::update
4142
// logs "Jvm Stats CLOSE!" and sets every jvm_* gauge to 0. Published once, frozen, then zeroed.
42-
Status jvm_management_env(JNIEnv** env) {
43-
return Jni::JvmLauncher::attach_current_thread(env);
44-
}
43+
//
44+
// Why a guard around the whole body rather than one call at the top. Two reasons, and neither is
45+
// served by taking the env alone:
46+
//
47+
// * init() is called from JvmLauncher::_bootstrap(), INSIDE ensure_jvm()'s call_once. Anything
48+
// on this path that reaches ensure_jvm() - which is what both Jni::Env::Get()'s slow path and
49+
// JvmLauncher::attach_current_thread() do - re-enters that once flag and hangs the BE on the
50+
// first JVM it ever creates.
51+
// * every Jni::Local* this file allocates releases itself through Env::Get() (see
52+
// RefHelper<Local>::get_env), so an unprimed thread would put each of those destructors back
53+
// behind the same SPI gate: it logs "Can't destroy Jni Ref" and returns without deleting, once
54+
// per object, every 15 seconds, on exactly the deployment this indirection exists for.
55+
using ScopedManagementEnv = Jni::JvmLauncher::ScopedVmEnv;
4556
} // namespace
4657

4758
#define DEFINE_JVM_SIZE_BYTES_METRIC(name, type) \
@@ -225,8 +236,11 @@ void JvmMetrics::update() {
225236
}
226237

227238
Status JvmStats::init() {
239+
// First declaration in the scope, so it is the last thing destroyed: every JNI wrapper below
240+
// is released while this is still in force.
241+
ScopedManagementEnv scoped_env;
228242
JNIEnv* env = nullptr;
229-
RETURN_IF_ERROR(jvm_management_env(&env));
243+
RETURN_IF_ERROR(scoped_env.attach(&env));
230244

231245
RETURN_IF_ERROR(Jni::Util::find_class(env, "java/lang/management/ManagementFactory",
232246
&_managementFactoryClass));
@@ -325,8 +339,11 @@ Status JvmStats::refresh(JvmMetrics* jvm_metrics) const {
325339
return Status::InternalError("Jvm Stats not init complete.");
326340
}
327341

342+
// First declaration in the scope, so it is the last thing destroyed: every Jni::Local* below
343+
// is released while this is still in force. See ScopedManagementEnv at the top of this file.
344+
ScopedManagementEnv scoped_env;
328345
JNIEnv* env = nullptr;
329-
RETURN_IF_ERROR(jvm_management_env(&env));
346+
RETURN_IF_ERROR(scoped_env.attach(&env));
330347

331348
Jni::LocalObject memoryMXBeanObj;
332349
RETURN_IF_ERROR(_managementFactoryClass.call_static_object_method(env, _getMemoryMXBeanMethod)

‎be/src/io/fs/hdfs/hdfs_mgr.cpp‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,9 @@
3030
#include "io/hdfs_builder.h"
3131
#include "io/hdfs_util.h"
3232
#include "runtime/exec_env.h"
33+
#include "runtime/thread_context.h"
3334
#include "util/jvm_launcher.h"
35+
#include "util/thread.h"
3436

3537
namespace doris::io {
3638

@@ -223,6 +225,15 @@ Status HdfsMgr::_create_hdfs_fs(const THdfsParams& hdfs_params, const std::strin
223225
auto btx = bthread::butex_create();
224226
*(int*)btx = 0;
225227
std::thread t([&] {
228+
// Named and given a thread context like every other pthread the BE starts. This one is
229+
// where the JVM is most often created for the first time: _create_hdfs_fs_impl() calls
230+
// ensure_jvm(), and ensure_jvm() only switches to a pthread of its own when it is called
231+
// FROM a bthread - which it is not here, because this switch already happened. Without
232+
// these two lines the whole bootstrap - seconds of hadoop ServiceLoader scanning, and any
233+
// hang inside it - runs on an anonymous thread with no memory tracking, and an operator
234+
// looking at the stacks has nothing to grep for.
235+
SCOPED_INIT_THREAD_CONTEXT();
236+
Thread::set_self_name("hdfs_fs_connect");
226237
st = _create_hdfs_fs_impl(hdfs_params, fs_name, fs_handler);
227238
*(int*)btx = 1;
228239
bthread::butex_wake_all(btx);

‎be/src/service/http/action/jni_plugin_status_action.cpp‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,8 @@ void JniPluginStatusAction::handle(HttpRequest* req) {
8484

8585
if (!config::enable_java_support) {
8686
HttpChannel::send_reply(req, HttpStatus::OK,
87-
R"({"javaSupport":false,"plugins":[]})"
87+
R"({"javaSupport":false,"registryInitialized":false,)"
88+
R"("plugins":[]})"
8889
"\n");
8990
return;
9091
}
@@ -139,6 +140,14 @@ void JniPluginStatusAction::handle(HttpRequest* req) {
139140
json_string(st.to_string())));
140141
return;
141142
}
143+
// The healthy shape is rendered on the Java side, which knows neither of these two:
144+
// enable_java_support is a BE config, and "the registry is up" is what the branch above just
145+
// established. Spliced in rather than left out so that all four shapes this endpoint can emit
146+
// carry the same keys - monitoring that keys on javaSupport would otherwise read a perfectly
147+
// working BE as one whose Java support is unknown, which is the only state it never is here.
148+
if (status.size() > 2 && status.front() == '{' && status[1] != '}') {
149+
status.insert(1, R"("javaSupport":true,"registryInitialized":true,)");
150+
}
142151
HttpChannel::send_reply(req, HttpStatus::OK, status + "\n");
143152
}
144153

‎be/src/util/jni-util.cpp‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,18 @@ Status Env::GetJNIEnvSlowPath(JNIEnv** env) {
6666
// rejected caller that had already attached would stay bound to the JVM - one JVM thread
6767
// structure and one stack reservation each - for the rest of its life, on a deployment where
6868
// it can never run a line of Java. With the base broken and load arriving, that is every
69-
// scanner thread, every pipeline thread and both metrics daemons.
69+
// scanner thread and every pipeline thread. Not the metrics daemons: the jvm_* metrics are
70+
// published whenever a JVM exists, base or no base, so they take their env from
71+
// JvmLauncher::ScopedVmEnv and never come through here.
72+
//
73+
// Never the call that RUNS the resolution, only the one that reads its answer: ensure_jvm()
74+
// above returned OK, which means _bootstrap() ran to its end on some thread and its last step
75+
// was ensure_jni_base(). That is what keeps this line from re-entering a call_once from an
76+
// unattached thread, and it is the whole reason the order above is the order above - so it is
77+
// asserted rather than left to be rediscovered.
78+
DCHECK(Util::jni_base_outcome().has_value())
79+
<< "the JNI base has no outcome yet, so this call would run the resolution instead of "
80+
"reading it - see the ordering note above";
7081
RETURN_IF_ERROR(Util::ensure_jni_base());
7182

7283
// Finally the attach. On failure tls_env_ is left untouched, so the thread still looks
@@ -90,7 +101,14 @@ Status Env::GetJniExceptionMsg(JNIEnv* env, bool log_stack, const string& prefix
90101
// its own throwing would land here with no way to render the throwable, and a DCHECK is a
91102
// no-op in a release build. Degrade to a plain message rather than dereference null: this
92103
// function exists to turn a Java exception into a Status, and it can still do that much.
93-
if (jni_util_cl_ == nullptr || throwable_to_string_id_ == nullptr) {
104+
//
105+
// All THREE, not just the two this function uses first: init_throw_exception() assigns them
106+
// in order, so a failure at its last step leaves the class and toString set and
107+
// toStackTrace null, and the log_stack branch below - log_stack defaults to true - would then
108+
// call CallStaticObjectMethod with a null jmethodID, which is the very SIGSEGV this guard is
109+
// here to prevent.
110+
if (jni_util_cl_ == nullptr || throwable_to_string_id_ == nullptr ||
111+
throwable_to_stack_trace_id_ == nullptr) {
94112
env->ExceptionDescribe();
95113
env->ExceptionClear();
96114
return Status::JniError(

‎be/src/util/jni-util.h‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,10 @@ class Env {
146146
// Caches an env the bootstrap attached by itself, so that resolving the JNI base right
147147
// after the JVM comes up stays on Get()'s fast path instead of re-entering ensure_jvm().
148148
static void set_tls_env(JNIEnv* env) { tls_env_ = env; }
149+
// What Get() would hand out on this thread without asking anyone, nullptr when nothing is
150+
// cached. For JvmLauncher::ScopedVmEnv, which primes the cache for one scope and has to put
151+
// back what it found rather than assume there was nothing: on the bootstrap thread there is.
152+
static JNIEnv* tls_env() { return tls_env_; }
149153
friend class JvmLauncher;
150154

151155
private:
@@ -1233,10 +1237,21 @@ class Util {
12331237
// libhdfs. JvmLauncher runs it once at the end of its bootstrap and logs a failure; Env, the
12341238
// door every Java caller comes through, refuses to hand out a JNIEnv while it is failing.
12351239
//
1236-
// Only those two may call it, and JvmLauncher only from inside its bootstrap with the calling
1237-
// thread already attached and its env cached: the resolution asks Env::Get() for a
1238-
// JNIEnv, and reaching this from an unattached thread would go back through
1239-
// ensure_jvm() and deadlock on the once flag the bootstrap is already holding.
1240+
// Only those two may call it. JvmLauncher calls it from inside its bootstrap, with the
1241+
// calling thread already attached and its env cached, because the resolution asks Env::Get()
1242+
// for a JNIEnv and an unattached thread there would go back through ensure_jvm() and deadlock
1243+
// on the once flag the bootstrap is already holding.
1244+
//
1245+
// Env::GetJNIEnvSlowPath() calls it from an unattached thread on purpose, and the invariant
1246+
// that makes THAT safe is a different one, worth stating because nothing enforces it: by the
1247+
// time the slow path gets here, ensure_jvm()'s call_once has already run to completion on
1248+
// some thread, so the resolution below is never the one that runs - it is always the cached
1249+
// answer of the attempt the bootstrap made. Every early exit of _bootstrap() leaves
1250+
// jvm_status non-OK, and the slow path returns on that before reaching this line.
1251+
//
1252+
// Making the base lazy (that is, resolving it here for real rather than reading a cached
1253+
// answer) would put the deadlock straight back, so GetJNIEnvSlowPath() holds it down with a
1254+
// DCHECK: on that path an outcome must already exist before this is called.
12401255
static Status ensure_jni_base();
12411256
friend class JvmLauncher;
12421257
friend class Env;

‎be/src/util/jvm_launcher.cpp‎

Lines changed: 46 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -94,10 +94,12 @@ std::string scan_class_path() {
9494
collect_jars(doris_home + "/custom_lib", &class_path);
9595
// Named separately because it is the one thing libhdfs needs that does NOT live under lib/:
9696
// the third-party filesystem jars moved to plugins/jni_fs when the Java plugins started
97-
// reading them too (BE config jni_plugin_fs_dir, which this fallback cannot consult - it runs
98-
// before config is loaded in the cases it exists for). Leaving it out would silently drop
99-
// oss-hdfs:// and jfs:// for exactly the processes this branch serves.
100-
collect_jars(doris_home + "/plugins/jni_fs", &class_path);
97+
// reading them too. Leaving it out would silently drop oss-hdfs:// and jfs:// for exactly the
98+
// processes this branch serves. Read from the config rather than spelled out, because this
99+
// runs from _build_options() - well after config is loaded - and an operator who points
100+
// jni_plugin_fs_dir somewhere else must not end up with the Java plugins reading one
101+
// directory and the native reader another.
102+
collect_jars(config::jni_plugin_fs_dir, &class_path);
101103

102104
const std::string hadoop_conf_dir = env_value("HADOOP_CONF_DIR");
103105
if (!hadoop_conf_dir.empty()) {
@@ -387,9 +389,13 @@ Status JvmLauncher::_bootstrap() {
387389
// it up and whether or not the plugin SPI resolved. Publishing them anywhere but here would
388390
// mean a BE that has a JVM but reached it through libhdfs (an HDFS catalog served by the
389391
// native reader, no JNI table format and no Java UDF anywhere) exports no jvm_* series at all
390-
// and has its whole Java heap counted as untracked memory. Before reset_tls_env() below: the
391-
// constructor calls into JNI, and off the fast path it would re-enter ensure_jvm() and
392-
// deadlock on the once flag this bootstrap is holding.
392+
// and has its whole Java heap counted as untracked memory.
393+
//
394+
// Reached from inside this call_once, so it must not touch anything that leads back to
395+
// ensure_jvm(). What keeps it off that path is ScopedVmEnv, which JvmStats::init() takes and
396+
// holds across its whole body; the primed env above is not what makes this safe and neither
397+
// is the position before reset_tls_env(), because the metrics daemon reruns the same code
398+
// later on a thread that was never primed at all.
393399
DorisMetrics::instance()->init_jvm_metrics();
394400

395401
if (!base.ok()) {
@@ -474,6 +480,39 @@ Status JvmLauncher::attach_current_thread(JNIEnv** env) {
474480
return _attach_current_thread(env);
475481
}
476482

483+
JNIEnv* JvmLauncher::_tls_env() {
484+
return Env::tls_env();
485+
}
486+
487+
void JvmLauncher::_set_tls_env(JNIEnv* env) {
488+
Env::set_tls_env(env);
489+
}
490+
491+
Status JvmLauncher::ScopedVmEnv::attach(JNIEnv** env) {
492+
DCHECK(!_primed) << "ScopedVmEnv::attach() called twice on one guard";
493+
if (_vm == nullptr) {
494+
return Status::JniError(
495+
"This process has no JVM, so there is no JNIEnv to look at. ScopedVmEnv "
496+
"deliberately does not create one.");
497+
}
498+
JNIEnv* attached = nullptr;
499+
RETURN_IF_ERROR(_attach_current_thread(&attached));
500+
// Read before the write, and put back rather than cleared in the destructor: this runs on the
501+
// bootstrap thread too, which primed the cache itself before calling in and still needs it
502+
// afterwards to finish resolving the JNI base.
503+
_previous = _tls_env();
504+
_set_tls_env(attached);
505+
_primed = true;
506+
*env = attached;
507+
return Status::OK();
508+
}
509+
510+
JvmLauncher::ScopedVmEnv::~ScopedVmEnv() {
511+
if (_primed) {
512+
_set_tls_env(_previous);
513+
}
514+
}
515+
477516
Status JvmLauncher::_attach_current_thread(JNIEnv** env) {
478517
JNIEnv* thread_env = nullptr;
479518
jint rc = _vm->GetEnv(reinterpret_cast<void**>(&thread_env), JNI_VERSION_1_8);

‎be/src/util/jvm_launcher.h‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,40 @@ class JvmLauncher {
6262
// about the plugin SPI - Jni::Env::Get() is the entry point that checks that too.
6363
static Status attach_current_thread(JNIEnv** env);
6464

65+
// A JNIEnv on a thread that must NOT ask for a JVM to be created, for the whole of one
66+
// scope. Two things, and the one caller this exists for - JvmStats, the jvm_* metrics -
67+
// needs both:
68+
//
69+
// * it attaches WITHOUT ensure_jvm(). JvmStats::init() is reached from _bootstrap(), which
70+
// runs inside ensure_jvm()'s own call_once, so an ensure_jvm() there re-enters that once
71+
// flag and the process deadlocks on the very first JVM it creates.
72+
// * it primes Jni::Env::Get()'s thread-local cache for the life of the guard, putting back
73+
// whatever was there when it is destroyed. Everything the caller allocates is released by
74+
// a RAII wrapper that asks Env::Get() for an env of its own, and off the fast path that is
75+
// the plugin-SPI gate: on a BE whose doris-jni-spi.jar did not resolve it refuses, the
76+
// destructor logs and returns WITHOUT deleting the reference, and every metrics tick then
77+
// leaks one JNI local ref per object it touched.
78+
//
79+
// Deliberately not a general-purpose way around that gate. It is sound here because the
80+
// jvm_* metrics reach only for java.lang.management, which every JVM has and no plugin
81+
// supplies, and they are published whenever a JVM exists - base or no base. Anything that
82+
// runs Doris's own Java code must keep coming through Jni::Env::Get().
83+
class ScopedVmEnv {
84+
public:
85+
ScopedVmEnv() = default;
86+
~ScopedVmEnv();
87+
ScopedVmEnv(const ScopedVmEnv&) = delete;
88+
ScopedVmEnv& operator=(const ScopedVmEnv&) = delete;
89+
90+
// Fails when this process has no JVM yet: this guard looks at one, it never asks for one
91+
// to be made. Every call site is reached only after a JVM exists by construction.
92+
Status attach(JNIEnv** env);
93+
94+
private:
95+
JNIEnv* _previous = nullptr;
96+
bool _primed = false;
97+
};
98+
6599
// The VM of this process, nullptr until this process has one. Deliberately not "until
66100
// ensure_jvm() has succeeded": _bootstrap() has two error paths after JNI_CreateJavaVM
67101
// returns, so a failed ensure_jvm() can leave a live VM behind. "Does a JVM exist" is both
@@ -78,6 +112,11 @@ class JvmLauncher {
78112
// attach_current_thread() without the ensure_jvm(), for the bootstrap itself: it runs
79113
// inside that call_once and would deadlock on it.
80114
static Status _attach_current_thread(JNIEnv** env);
115+
// Jni::Env's thread-local env cache, reached from here because JvmLauncher is its friend
116+
// and ScopedVmEnv - a member of this class - inherits that access. Defined in the .cpp,
117+
// where Env is a complete type.
118+
static JNIEnv* _tls_env();
119+
static void _set_tls_env(JNIEnv* env);
81120
static Status _create_jvm();
82121
static std::vector<std::string> _build_options();
83122
static std::string _class_path_option();

0 commit comments

Comments
 (0)