Skip to content

Commit 7eaa891

Browse files
pzhdfyYour Name
authored andcommitted
[fix](java udf) fix possible deadlock when udf closed in bprc (#55302)
### What problem does this PR solve? Related PR: #30082 Problem Summary: We are using doris 3.0.4 with java udf, when running long time, some BE may got jvm deadlocks with UNKNOWN_owner_addr [deadlock in udf close] <img width="2314" height="882" alt="image" src="https://github.com/user-attachments/assets/c16b5d18-9900-45dc-8bb4-894180694a44" /> [deadlock in throwableToStackTrace, waiting a StringWriter , but this is impossible , because StringWriter was newed everytime] <img width="1724" height="720" alt="image" src="https://github.com/user-attachments/assets/b586f752-0ba8-43cc-89b4-352a7d9ab00c" /> <img width="618" height="144" alt="image" src="https://github.com/user-attachments/assets/20c2ad11-beef-4b0c-b2a2-343d9b0fd14c" /> In be.WARNING we found get JNIEnv failed when close udf <img width="1659" height="426" alt="image" src="https://github.com/user-attachments/assets/6cf54aa2-522a-407e-a53a-e17fdbc9162e" /> In be.out we found libhdfs error log [JNIEnv is got by libhdfs when in X86 ] Call to AttachCurrentThread failed with error: -1 getJNIEnv: getGlobalJNIEnv failed In be.WARNING there are some other err when using GetJniExceptionMsg <img width="1686" height="530" alt="image" src="https://github.com/user-attachments/assets/35ca7b4c-0a9d-4da3-8be2-9a07ccebb74b" /> we found all those err has common ground 1.all case occured in JavaFunctionCall::close 2.the stack all have bthread keyword after seaching the web , we found that JNI is not compatible with bthread https://blog.csdn.net/qq_46104835/article/details/139360911 https://gitee.com/baidu/BRPC/blob/master/docs/cn/server.md#pthread%E6%A8%A1%E5%BC%8F <img width="1101" height="211" alt="image" src="https://github.com/user-attachments/assets/e1da2f9a-9e37-47d9-b0a1-c1652da93e75" /> the bthread stack layout is not compatible for java, so we may meed 1. AttachCurrentThread[using by getJNIEnv] will check stack layout fail 2. AttachCurrentThread/getJNIEnv may success, but java method may execute failed, even deadlock, because the stack layout has incompatible address for jvm Then we switch bthread to pthread mode , every thing works fine.[add -usercode_in_pthread=true] We want to know how often bhread do the JavaFunctionCall::close, then we add metrics. only 1/10000 JavaFunctionCall::close running in bthread <img width="1886" height="358" alt="image" src="https://github.com/user-attachments/assets/34293bac-696b-41f9-b5eb-4daaab81fb7f" /> But why JavaFunctionCall::close occured in bthread[ after #16634, exec_plan_fragment is running in pthread instead of bthread] then we found a pr #30082 ExchangeSinkBuffer<Parent>::_send_rpc will set a send_callback with a weak_task_ctx <img width="1392" height="254" alt="image" src="https://github.com/user-attachments/assets/a1d28f18-c3c7-4953-baab-d1109f341aa1" /> So sometimes send_callback may using weak_task_ctx.lock() to get a shared_ptr to task_ctx, then task_ctx destructor my be called in send_callback[ send_callback is running in bthread] So We modify JavaFunctionCall::close when JavaFunctionCall::close running in bthread, we submit the jni operation to a pthread pool, and wait it finish because only 1/10000 JavaFunctionCall::close are running in bthread. the pthread pool size can set to a small number.
1 parent 841aa79 commit 7eaa891

4 files changed

Lines changed: 47 additions & 8 deletions

File tree

‎be/src/util/doris_metrics.cpp‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,8 @@ DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(num_io_bytes_read_total, MetricUnit::OPERAT
226226
DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(num_io_bytes_read_from_cache, MetricUnit::OPERATIONS);
227227
DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(num_io_bytes_read_from_remote, MetricUnit::OPERATIONS);
228228

229+
DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(udf_close_bthread_count, MetricUnit::OPERATIONS);
230+
229231
DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(query_ctx_cnt, MetricUnit::NOUNIT);
230232
DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(scanner_ctx_cnt, MetricUnit::NOUNIT);
231233
DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(scanner_cnt, MetricUnit::NOUNIT);
@@ -393,6 +395,8 @@ DorisMetrics::DorisMetrics() : _metric_registry(_s_registry_name) {
393395
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, num_io_bytes_read_from_cache);
394396
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, num_io_bytes_read_from_remote);
395397

398+
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, udf_close_bthread_count);
399+
396400
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, query_ctx_cnt);
397401
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, scanner_ctx_cnt);
398402
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, scanner_cnt);

‎be/src/util/doris_metrics.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,8 @@ class DorisMetrics {
244244
IntCounter* num_io_bytes_read_from_cache = nullptr;
245245
IntCounter* num_io_bytes_read_from_remote = nullptr;
246246

247+
IntCounter* udf_close_bthread_count = nullptr;
248+
247249
IntCounter* query_ctx_cnt = nullptr;
248250
IntCounter* scanner_ctx_cnt = nullptr;
249251
IntCounter* scanner_cnt = nullptr;

‎be/src/vec/functions/function_java_udf.cpp‎

Lines changed: 38 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,12 @@
1717

1818
#include "vec/functions/function_java_udf.h"
1919

20+
#include <bthread/bthread.h>
21+
2022
#include <memory>
2123
#include <string>
2224

25+
#include "common/exception.h"
2326
#include "jni.h"
2427
#include "runtime/user_function_cache.h"
2528
#include "util/jni-util.h"
@@ -32,9 +35,22 @@ const char* EXECUTOR_EVALUATE_SIGNATURE = "(Ljava/util/Map;Ljava/util/Map;)J";
3235
const char* EXECUTOR_CLOSE_SIGNATURE = "()V";
3336

3437
namespace doris::vectorized {
38+
std::unique_ptr<ThreadPool> JavaFunctionCall::close_workers;
39+
std::once_flag JavaFunctionCall::close_workers_init_once;
40+
3541
JavaFunctionCall::JavaFunctionCall(const TFunction& fn, const DataTypes& argument_types,
3642
const DataTypePtr& return_type)
37-
: fn_(fn), _argument_types(argument_types), _return_type(return_type) {}
43+
: fn_(fn), _argument_types(argument_types), _return_type(return_type) {
44+
std::call_once(close_workers_init_once, []() {
45+
auto build_st = ThreadPoolBuilder("UDFCloseWorkers")
46+
.set_min_threads(4)
47+
.set_max_threads(std::min(32, CpuInfo::num_cores()))
48+
.build(&close_workers);
49+
if (!build_st.ok()) {
50+
throw doris::Exception(ErrorCode::INTERNAL_ERROR, "Failed to build UDFCloseWorkers");
51+
}
52+
});
53+
}
3854

3955
Status JavaFunctionCall::open(FunctionContext* context, FunctionContext::FunctionStateScope scope) {
4056
JNIEnv* env = nullptr;
@@ -122,13 +138,27 @@ Status JavaFunctionCall::execute_impl(FunctionContext* context, Block& block,
122138

123139
Status JavaFunctionCall::close(FunctionContext* context,
124140
FunctionContext::FunctionStateScope scope) {
125-
auto* jni_ctx = reinterpret_cast<JniContext*>(
126-
context->get_function_state(FunctionContext::THREAD_LOCAL));
127-
// JNIContext own some resource and its release method depend on JavaFunctionCall
128-
// has to release the resource before JavaFunctionCall is deconstructed.
129-
if (jni_ctx) {
130-
RETURN_IF_ERROR(jni_ctx->close());
141+
auto close_func = [context]() {
142+
auto* jni_ctx = reinterpret_cast<JniContext*>(
143+
context->get_function_state(FunctionContext::THREAD_LOCAL));
144+
// JNIContext own some resource and its release method depend on JavaFunctionCall
145+
// has to release the resource before JavaFunctionCall is deconstructed.
146+
if (jni_ctx) {
147+
RETURN_IF_ERROR(jni_ctx->close());
148+
}
149+
return Status::OK();
150+
};
151+
152+
if (bthread_self() == 0) {
153+
return close_func();
154+
} else {
155+
DorisMetrics::instance()->udf_close_bthread_count->increment(1);
156+
// Use the close_workers pthread pool to execute the close function
157+
auto task = std::make_shared<std::packaged_task<Status()>>(std::move(close_func));
158+
auto task_future = task->get_future();
159+
RETURN_IF_ERROR(close_workers->submit_func([task]() { (*task)(); }));
160+
RETURN_IF_ERROR(task_future.get());
161+
return Status::OK();
131162
}
132-
return Status::OK();
133163
}
134164
} // namespace doris::vectorized

‎be/src/vec/functions/function_java_udf.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,9 @@ class JavaFunctionCall : public IFunctionBase {
110110
const DataTypes _argument_types;
111111
const DataTypePtr _return_type;
112112

113+
static std::unique_ptr<ThreadPool> close_workers;
114+
static std::once_flag close_workers_init_once;
115+
113116
struct JniContext {
114117
// Do not save parent directly, because parent is in VExpr, but jni context is in FunctionContext
115118
// The deconstruct sequence is not determined, it will core.

0 commit comments

Comments
 (0)