Repository navigation
branch-4.0: [fix](java udf) fix possible deadlock when udf closed in bprc #55302 - #57944
Merged
Merged
Conversation
### What problem does this PR solve? Related PR: #30082 Problem Summary: We are using doris 3.0.4 with java udf, when running long time, some BE may got jvm deadlocks with UNKNOWN_owner_addr [deadlock in udf close] <img width="2314" height="882" alt="image" src="https://github.com/user-attachments/assets/c16b5d18-9900-45dc-8bb4-894180694a44" /> [deadlock in throwableToStackTrace, waiting a StringWriter , but this is impossible , because StringWriter was newed everytime] <img width="1724" height="720" alt="image" src="https://github.com/user-attachments/assets/b586f752-0ba8-43cc-89b4-352a7d9ab00c" /> <img width="618" height="144" alt="image" src="https://github.com/user-attachments/assets/20c2ad11-beef-4b0c-b2a2-343d9b0fd14c" /> In be.WARNING we found get JNIEnv failed when close udf <img width="1659" height="426" alt="image" src="https://github.com/user-attachments/assets/6cf54aa2-522a-407e-a53a-e17fdbc9162e" /> In be.out we found libhdfs error log [JNIEnv is got by libhdfs when in X86 ] Call to AttachCurrentThread failed with error: -1 getJNIEnv: getGlobalJNIEnv failed In be.WARNING there are some other err when using GetJniExceptionMsg <img width="1686" height="530" alt="image" src="https://github.com/user-attachments/assets/35ca7b4c-0a9d-4da3-8be2-9a07ccebb74b" /> we found all those err has common ground 1.all case occured in JavaFunctionCall::close 2.the stack all have bthread keyword after seaching the web , we found that JNI is not compatible with bthread https://blog.csdn.net/qq_46104835/article/details/139360911 https://gitee.com/baidu/BRPC/blob/master/docs/cn/server.md#pthread%E6%A8%A1%E5%BC%8F <img width="1101" height="211" alt="image" src="https://github.com/user-attachments/assets/e1da2f9a-9e37-47d9-b0a1-c1652da93e75" /> the bthread stack layout is not compatible for java, so we may meed 1. AttachCurrentThread[using by getJNIEnv] will check stack layout fail 2. AttachCurrentThread/getJNIEnv may success, but java method may execute failed, even deadlock, because the stack layout has incompatible address for jvm Then we switch bthread to pthread mode , every thing works fine.[add -usercode_in_pthread=true] We want to know how often bhread do the JavaFunctionCall::close, then we add metrics. only 1/10000 JavaFunctionCall::close running in bthread <img width="1886" height="358" alt="image" src="https://github.com/user-attachments/assets/34293bac-696b-41f9-b5eb-4daaab81fb7f" /> But why JavaFunctionCall::close occured in bthread[ after #16634, exec_plan_fragment is running in pthread instead of bthread] then we found a pr #30082 ExchangeSinkBuffer<Parent>::_send_rpc will set a send_callback with a weak_task_ctx <img width="1392" height="254" alt="image" src="https://github.com/user-attachments/assets/a1d28f18-c3c7-4953-baab-d1109f341aa1" /> So sometimes send_callback may using weak_task_ctx.lock() to get a shared_ptr to task_ctx, then task_ctx destructor my be called in send_callback[ send_callback is running in bthread] So We modify JavaFunctionCall::close when JavaFunctionCall::close running in bthread, we submit the jni operation to a pthread pool, and wait it finish because only 1/10000 JavaFunctionCall::close are running in bthread. the pthread pool size can set to a small number.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
yiguolei
approved these changes
Nov 12, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #55302