Repository navigation
Commit 7a74f84
authored
[fix](java udf) fix possible deadlock when udf closed in bprc (#55302)
### What problem does this PR solve?
Related PR: #30082
Problem Summary:
We are using doris 3.0.4 with java udf, when running long time, some BE
may got jvm deadlocks with UNKNOWN_owner_addr
[deadlock in udf close]
<img width="2314" height="882" alt="image"
src="https://github.com/user-attachments/assets/c16b5d18-9900-45dc-8bb4-894180694a44"
/>
[deadlock in throwableToStackTrace, waiting a StringWriter , but this is
impossible , because StringWriter was newed everytime]
<img width="1724" height="720" alt="image"
src="https://github.com/user-attachments/assets/b586f752-0ba8-43cc-89b4-352a7d9ab00c"
/>
<img width="618" height="144" alt="image"
src="https://github.com/user-attachments/assets/20c2ad11-beef-4b0c-b2a2-343d9b0fd14c"
/>
In be.WARNING we found get JNIEnv failed when close udf
<img width="1659" height="426" alt="image"
src="https://github.com/user-attachments/assets/6cf54aa2-522a-407e-a53a-e17fdbc9162e"
/>
In be.out we found libhdfs error log [JNIEnv is got by libhdfs when in
X86 ]
Call to AttachCurrentThread failed with error: -1
getJNIEnv: getGlobalJNIEnv failed
In be.WARNING there are some other err when using GetJniExceptionMsg
<img width="1686" height="530" alt="image"
src="https://github.com/user-attachments/assets/35ca7b4c-0a9d-4da3-8be2-9a07ccebb74b"
/>
we found all those err has common ground
1.all case occured in JavaFunctionCall::close
2.the stack all have bthread keyword
after seaching the web , we found that JNI is not compatible with
bthread
https://blog.csdn.net/qq_46104835/article/details/139360911
https://gitee.com/baidu/BRPC/blob/master/docs/cn/server.md#pthread%E6%A8%A1%E5%BC%8F
<img width="1101" height="211" alt="image"
src="https://github.com/user-attachments/assets/e1da2f9a-9e37-47d9-b0a1-c1652da93e75"
/>
the bthread stack layout is not compatible for java, so we may meed
1. AttachCurrentThread[using by getJNIEnv] will check stack layout fail
2. AttachCurrentThread/getJNIEnv may success, but java method may
execute failed, even deadlock, because the stack layout has incompatible
address for jvm
Then we switch bthread to pthread mode , every thing works fine.[add
-usercode_in_pthread=true]
We want to know how often bhread do the JavaFunctionCall::close, then we
add metrics. only 1/10000 JavaFunctionCall::close running in bthread
<img width="1886" height="358" alt="image"
src="https://github.com/user-attachments/assets/34293bac-696b-41f9-b5eb-4daaab81fb7f"
/>
But why JavaFunctionCall::close occured in bthread[ after
#16634, exec_plan_fragment is
running in pthread instead of bthread]
then we found a pr
#30082
ExchangeSinkBuffer<Parent>::_send_rpc will set a send_callback with a
weak_task_ctx
<img width="1392" height="254" alt="image"
src="https://github.com/user-attachments/assets/a1d28f18-c3c7-4953-baab-d1109f341aa1"
/>
So sometimes send_callback may using weak_task_ctx.lock() to get a
shared_ptr to task_ctx, then task_ctx destructor my be called in
send_callback[ send_callback is running in bthread]
So We modify JavaFunctionCall::close
when JavaFunctionCall::close running in bthread, we submit the jni
operation to a pthread pool, and wait it finish
because only 1/10000 JavaFunctionCall::close are running in bthread. the
pthread pool size can set to a small number.1 parent 97d8f23 commit 7a74f84
4 files changed
Lines changed: 47 additions & 8 deletions
File tree
- be/src
- util
- vec/functions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
| 228 | + | |
| 229 | + | |
228 | 230 | | |
229 | 231 | | |
230 | 232 | | |
| |||
388 | 390 | | |
389 | 391 | | |
390 | 392 | | |
| 393 | + | |
| 394 | + | |
391 | 395 | | |
392 | 396 | | |
393 | 397 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
| 247 | + | |
| 248 | + | |
247 | 249 | | |
248 | 250 | | |
249 | 251 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
| 25 | + | |
23 | 26 | | |
24 | 27 | | |
25 | 28 | | |
| |||
32 | 35 | | |
33 | 36 | | |
34 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
35 | 41 | | |
36 | 42 | | |
37 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
38 | 54 | | |
39 | 55 | | |
40 | 56 | | |
| |||
122 | 138 | | |
123 | 139 | | |
124 | 140 | | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
131 | 162 | | |
132 | | - | |
133 | 163 | | |
134 | 164 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
113 | 116 | | |
114 | 117 | | |
115 | 118 | | |
| |||
0 commit comments