Skip to content

branch-4.0: [fix](java udf) fix possible deadlock when udf closed in bprc #55302 - #57944

Merged
yiguolei merged 1 commit into
branch-4.0from
auto-pick-55302-branch-4.0
Nov 12, 2025
Merged

yiguolei merged 1 commit into
branch-4.0from
auto-pick-55302-branch-4.0

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Cherry-picked from #55302

### What problem does this PR solve?

Related PR: #30082

Problem Summary:

We are using doris 3.0.4 with java udf, when running long time, some BE
may got jvm deadlocks with UNKNOWN_owner_addr
[deadlock in udf close]
<img width="2314" height="882" alt="image"
src="https://github.com/user-attachments/assets/c16b5d18-9900-45dc-8bb4-894180694a44"
/>
[deadlock in throwableToStackTrace, waiting a StringWriter , but this is
impossible , because StringWriter was newed everytime]
<img width="1724" height="720" alt="image"
src="https://github.com/user-attachments/assets/b586f752-0ba8-43cc-89b4-352a7d9ab00c"
/>
<img width="618" height="144" alt="image"
src="https://github.com/user-attachments/assets/20c2ad11-beef-4b0c-b2a2-343d9b0fd14c"
/>


In be.WARNING we found  get JNIEnv failed when close udf
<img width="1659" height="426" alt="image"
src="https://github.com/user-attachments/assets/6cf54aa2-522a-407e-a53a-e17fdbc9162e"
/>
In be.out we found libhdfs error log [JNIEnv is got by libhdfs when in
X86 ]
Call to AttachCurrentThread failed with error: -1
getJNIEnv: getGlobalJNIEnv failed

In be.WARNING there are some other err  when using GetJniExceptionMsg
<img width="1686" height="530" alt="image"
src="https://github.com/user-attachments/assets/35ca7b4c-0a9d-4da3-8be2-9a07ccebb74b"
/>

we found all those err has common ground
1.all case occured  in JavaFunctionCall::close
2.the stack all have bthread keyword

after seaching the web , we found that JNI is not compatible with
bthread
https://blog.csdn.net/qq_46104835/article/details/139360911

https://gitee.com/baidu/BRPC/blob/master/docs/cn/server.md#pthread%E6%A8%A1%E5%BC%8F
<img width="1101" height="211" alt="image"
src="https://github.com/user-attachments/assets/e1da2f9a-9e37-47d9-b0a1-c1652da93e75"
/>

the bthread stack layout is not compatible for java, so we may meed
1. AttachCurrentThread[using by getJNIEnv] will check stack layout fail
2. AttachCurrentThread/getJNIEnv may success, but java method may
execute failed, even deadlock, because the stack layout has incompatible
address for jvm


Then we switch bthread to pthread mode , every thing works fine.[add
-usercode_in_pthread=true]

We want to know how often bhread do the JavaFunctionCall::close, then we
add metrics. only 1/10000 JavaFunctionCall::close running in bthread
<img width="1886" height="358" alt="image"
src="https://github.com/user-attachments/assets/34293bac-696b-41f9-b5eb-4daaab81fb7f"
/>

But why JavaFunctionCall::close occured in bthread[ after
#16634, exec_plan_fragment is
running in pthread instead of bthread]

then we found a pr
#30082

ExchangeSinkBuffer<Parent>::_send_rpc will set a send_callback with a
weak_task_ctx
<img width="1392" height="254" alt="image"
src="https://github.com/user-attachments/assets/a1d28f18-c3c7-4953-baab-d1109f341aa1"
/>
So sometimes send_callback may using weak_task_ctx.lock() to get a
shared_ptr to task_ctx, then task_ctx destructor my be called in
send_callback[ send_callback is running in bthread]

So We modify JavaFunctionCall::close
when JavaFunctionCall::close running in bthread, we submit the jni
operation to a pthread pool, and wait it finish
because only 1/10000 JavaFunctionCall::close are running in bthread. the
pthread pool size can set to a small number.
@github-actions
github-actions Bot requested a review from yiguolei as a code owner November 12, 2025 06:07
@Thearas

Thearas commented Nov 12, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Nov 12, 2025
@Thearas

Thearas commented Nov 12, 2025

Copy link
Copy Markdown
Contributor

run buildall

@yiguolei
yiguolei merged commit 3c3a396 into branch-4.0 Nov 12, 2025
10 of 12 checks passed
@github-actions
github-actions Bot deleted the auto-pick-55302-branch-4.0 branch November 12, 2025 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants