[fix](java udf) fix possible deadlock when udf closed in bprc by pzhdfy · Pull Request #55302 · apache/doris

pzhdfy · 2025-08-26T07:51:17Z

What problem does this PR solve?

Related PR: #30082

Problem Summary:

We are using doris 3.0.4 with java udf, when running long time, some BE may got jvm deadlocks with UNKNOWN_owner_addr
[deadlock in udf close]

[deadlock in throwableToStackTrace, waiting a StringWriter , but this is impossible , because StringWriter was newed everytime]

In be.WARNING we found get JNIEnv failed when close udf

In be.out we found libhdfs error log [JNIEnv is got by libhdfs when in X86 ]
Call to AttachCurrentThread failed with error: -1
getJNIEnv: getGlobalJNIEnv failed

In be.WARNING there are some other err when using GetJniExceptionMsg

we found all those err has common ground
1.all case occured in JavaFunctionCall::close
2.the stack all have bthread keyword

after seaching the web , we found that JNI is not compatible with bthread
https://blog.csdn.net/qq_46104835/article/details/139360911
https://gitee.com/baidu/BRPC/blob/master/docs/cn/server.md#pthread%E6%A8%A1%E5%BC%8F

the bthread stack layout is not compatible for java, so we may meed

AttachCurrentThread[using by getJNIEnv] will check stack layout fail
AttachCurrentThread/getJNIEnv may success, but java method may execute failed, even deadlock, because the stack layout has incompatible address for jvm

Then we switch bthread to pthread mode , every thing works fine.[add -usercode_in_pthread=true]

We want to know how often bhread do the JavaFunctionCall::close, then we add metrics. only 1/10000 JavaFunctionCall::close running in bthread

But why JavaFunctionCall::close occured in bthread[ after https://github.com//issues/16634, exec_plan_fragment is running in pthread instead of bthread]

then we found a pr
#30082

ExchangeSinkBuffer::_send_rpc will set a send_callback with a weak_task_ctx

So sometimes send_callback may using weak_task_ctx.lock() to get a shared_ptr to task_ctx, then task_ctx destructor my be called in send_callback[ send_callback is running in bthread]

So We modify JavaFunctionCall::close
when JavaFunctionCall::close running in bthread, we submit the jni operation to a pthread pool, and wait it finish
because only 1/10000 JavaFunctionCall::close are running in bthread. the pthread pool size can set to a small number.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-08-26T07:51:23Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhangstar333 · 2025-08-26T09:46:10Z

run buildall

doris-robot · 2025-08-26T11:05:20Z

TPC-H: Total hot run time: 34074 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 53b6f23fd4c03c6cb9b25cf2455a90d165302f15, data reload: false

------ Round 1 ----------------------------------
q1	17595	5340	5191	5191
q2	1915	275	183	183
q3	10334	1268	684	684
q4	10238	1014	536	536
q5	7539	2465	2347	2347
q6	181	161	128	128
q7	907	738	596	596
q8	9290	1305	1119	1119
q9	6984	5137	5082	5082
q10	6955	2363	1957	1957
q11	486	290	270	270
q12	361	350	217	217
q13	17764	3696	3101	3101
q14	234	228	217	217
q15	557	482	480	480
q16	425	438	384	384
q17	598	883	366	366
q18	7574	7188	7098	7098
q19	1083	967	564	564
q20	344	342	214	214
q21	3898	3212	2362	2362
q22	1059	1034	978	978
Total cold run time: 106321 ms
Total hot run time: 34074 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5185	5164	5162	5162
q2	246	323	224	224
q3	2192	2667	2280	2280
q4	1379	1782	1348	1348
q5	4238	4555	4540	4540
q6	261	173	124	124
q7	2098	1970	1815	1815
q8	2686	2569	2648	2569
q9	7319	7434	7244	7244
q10	3116	3247	2880	2880
q11	634	511	492	492
q12	699	788	614	614
q13	3629	3962	3330	3330
q14	285	301	290	290
q15	517	484	477	477
q16	460	483	450	450
q17	1192	1608	1612	1608
q18	7724	7797	7617	7617
q19	829	827	833	827
q20	1911	1979	1856	1856
q21	4947	4392	4398	4392
q22	1102	1022	973	973
Total cold run time: 52649 ms
Total hot run time: 51112 ms

doris-robot · 2025-08-26T11:16:49Z

TPC-DS: Total hot run time: 184515 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 53b6f23fd4c03c6cb9b25cf2455a90d165302f15, data reload: false

query1	976	390	420	390
query2	6517	1725	1768	1725
query3	6762	221	212	212
query4	26321	24120	22944	22944
query5	4347	618	513	513
query6	313	218	204	204
query7	4645	499	287	287
query8	269	233	231	231
query9	8576	2893	2871	2871
query10	489	338	295	295
query11	15850	15004	14714	14714
query12	162	111	113	111
query13	1646	551	438	438
query14	8632	5784	5675	5675
query15	213	185	178	178
query16	7153	656	459	459
query17	1187	719	606	606
query18	1991	421	323	323
query19	194	196	166	166
query20	120	119	118	118
query21	214	124	105	105
query22	4108	4140	4087	4087
query23	33718	32893	33072	32893
query24	8069	2319	2321	2319
query25	526	482	393	393
query26	823	283	198	198
query27	2696	498	344	344
query28	4349	2264	2242	2242
query29	699	563	449	449
query30	283	234	214	214
query31	892	803	723	723
query32	86	76	73	73
query33	538	362	347	347
query34	789	837	501	501
query35	808	825	743	743
query36	975	1011	909	909
query37	124	108	88	88
query38	4150	4051	4033	4033
query39	1485	1408	1429	1408
query40	225	123	114	114
query41	58	57	53	53
query42	118	105	117	105
query43	521	491	498	491
query44	1333	855	842	842
query45	178	171	161	161
query46	852	1004	632	632
query47	1750	1803	1728	1728
query48	398	415	322	322
query49	698	471	391	391
query50	655	688	397	397
query51	4053	4215	4191	4191
query52	117	109	97	97
query53	236	256	191	191
query54	594	591	537	537
query55	90	88	88	88
query56	316	321	299	299
query57	1210	1210	1112	1112
query58	275	268	268	268
query59	2700	2813	2635	2635
query60	343	331	321	321
query61	122	125	125	125
query62	821	747	659	659
query63	218	195	190	190
query64	3492	1072	707	707
query65	4290	4205	4257	4205
query66	1041	414	327	327
query67	15381	15116	15131	15116
query68	8086	910	575	575
query69	495	321	282	282
query70	1255	1182	1144	1144
query71	481	336	327	327
query72	5703	4728	4849	4728
query73	832	662	361	361
query74	8899	9099	8697	8697
query75	3857	3075	2674	2674
query76	3633	1131	729	729
query77	797	399	321	321
query78	9741	9706	8865	8865
query79	2184	810	601	601
query80	637	542	485	485
query81	505	257	221	221
query82	445	141	110	110
query83	287	252	235	235
query84	300	104	95	95
query85	786	366	332	332
query86	344	352	297	297
query87	4287	4354	4151	4151
query88	2865	2214	2203	2203
query89	387	308	292	292
query90	1932	217	215	215
query91	143	137	112	112
query92	90	71	70	70
query93	1152	959	635	635
query94	674	391	281	281
query95	402	310	310	310
query96	481	599	275	275
query97	2666	2683	2567	2567
query98	241	222	211	211
query99	1436	1425	1281	1281
Total cold run time: 269803 ms
Total hot run time: 184515 ms

doris-robot · 2025-08-26T11:22:18Z

ClickBench: Total hot run time: 33.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 53b6f23fd4c03c6cb9b25cf2455a90d165302f15, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.24	0.08	0.07
query4	1.62	0.11	0.11
query5	0.43	0.44	0.39
query6	1.17	0.63	0.65
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.61	0.53	0.52
query10	0.59	0.57	0.59
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.61	0.62
query14	0.79	0.84	0.84
query15	0.89	0.86	0.86
query16	0.42	0.39	0.39
query17	1.05	1.06	1.05
query18	0.22	0.20	0.20
query19	1.94	1.84	1.80
query20	0.02	0.01	0.01
query21	15.39	0.94	0.57
query22	0.77	1.23	1.20
query23	14.74	1.42	0.65
query24	6.30	1.31	1.00
query25	0.54	0.10	0.15
query26	0.64	0.15	0.12
query27	0.06	0.05	0.04
query28	10.56	0.95	0.43
query29	12.53	3.94	3.27
query30	3.09	2.98	2.96
query31	2.83	0.59	0.37
query32	3.23	0.56	0.49
query33	2.99	3.19	3.04
query34	16.07	5.52	4.86
query35	4.90	4.89	4.98
query36	0.69	0.52	0.50
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.19	0.14	0.14
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 107 s
Total hot run time: 33.23 s

doris-robot · 2025-08-26T12:33:50Z

BE UT Coverage Report

Increment line coverage 3.45% (1/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	51.78% (17149/33117)
Line Coverage	37.26% (156261/419393)
Region Coverage	31.96% (119138/372808)
Branch Coverage	33.25% (52352/157472)

hello-stephen · 2025-08-26T13:27:45Z

BE Regression && UT Coverage Report

Increment line coverage 93.10% (27/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.66% (22990/32535)
Line Coverage	56.93% (238684/419283)
Region Coverage	52.20% (197482/378354)
Branch Coverage	53.97% (85522/158470)

github-actions · 2025-08-27T03:38:18Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-08-27T03:38:20Z

PR approved by anyone and no changes requested.

hello-stephen · 2025-08-27T05:10:00Z

BE Regression && UT Coverage Report

Increment line coverage 93.10% (27/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.66% (22990/32535)
Line Coverage	56.93% (238684/419283)
Region Coverage	52.20% (197482/378354)
Branch Coverage	53.97% (85522/158470)

hello-stephen · 2025-08-27T05:27:32Z

BE Regression && UT Coverage Report

Increment line coverage 93.10% (27/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.66% (22989/32535)
Line Coverage	56.93% (238698/419283)
Region Coverage	52.23% (197614/378354)
Branch Coverage	53.97% (85520/158470)

hello-stephen · 2025-08-27T05:47:05Z

BE Regression && UT Coverage Report

Increment line coverage 93.10% (27/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.66% (22989/32535)
Line Coverage	56.92% (238676/419283)
Region Coverage	52.21% (197531/378354)
Branch Coverage	53.96% (85508/158470)

hello-stephen · 2025-09-02T03:58:27Z

BE Regression && UT Coverage Report

Increment line coverage 93.10% (27/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.66% (22989/32535)
Line Coverage	56.92% (238676/419283)
Region Coverage	52.21% (197531/378354)
Branch Coverage	53.96% (85508/158470)

hello-stephen · 2025-09-02T04:07:04Z

BE Regression && UT Coverage Report

Increment line coverage 93.10% (27/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	70.66% (22990/32535)
Line Coverage	56.93% (238685/419283)
Region Coverage	52.20% (197508/378354)
Branch Coverage	53.96% (85512/158470)

zhangstar333 · 2025-09-28T08:55:18Z

run buildall

doris-robot · 2025-09-28T10:23:03Z

ClickBench: Total hot run time: 29.94 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c64463ebf644ae85a4baeed8d41ad49bdc793a7, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.06
query3	0.25	0.09	0.08
query4	1.61	0.13	0.12
query5	0.28	0.26	0.26
query6	1.20	0.66	0.65
query7	0.03	0.03	0.03
query8	0.05	0.04	0.05
query9	0.62	0.52	0.51
query10	0.59	0.57	0.58
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.63	0.62
query14	1.02	1.05	1.01
query15	0.85	0.89	0.85
query16	0.41	0.41	0.41
query17	1.04	1.07	1.04
query18	0.22	0.20	0.20
query19	1.90	1.79	1.89
query20	0.02	0.01	0.01
query21	15.41	0.90	0.57
query22	0.76	1.16	0.67
query23	15.01	1.38	0.64
query24	6.74	1.13	0.50
query25	0.44	0.14	0.06
query26	0.52	0.16	0.14
query27	0.07	0.05	0.05
query28	9.30	1.42	0.92
query29	12.60	3.89	3.20
query30	0.29	0.13	0.12
query31	2.86	0.59	0.38
query32	3.26	0.56	0.49
query33	3.02	3.16	3.04
query34	16.11	5.46	4.91
query35	4.92	4.86	4.88
query36	0.70	0.51	0.49
query37	0.10	0.08	0.08
query38	0.06	0.05	0.04
query39	0.03	0.03	0.03
query40	0.19	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 103.74 s
Total hot run time: 29.94 s

hello-stephen · 2025-09-28T11:43:46Z

BE UT Coverage Report

Increment line coverage 3.45% (1/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.41% (17634/33644)
Line Coverage	37.63% (160204/425713)
Region Coverage	32.18% (122221/379749)
Branch Coverage	33.53% (53587/159841)

hello-stephen · 2025-09-28T15:11:58Z

BE Regression && UT Coverage Report

Increment line coverage 93.10% (27/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.14% (23363/32842)
Line Coverage	57.52% (244188/424535)
Region Coverage	52.86% (203376/384771)
Branch Coverage	54.55% (87540/160490)

…#55302)" This reverts commit 7a74f84.

…57918) ### What problem does this PR solve? Problem Summary: this pr introuced #55302 _udf_close_workers thread pool not stop when exit ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

### What problem does this PR solve? Related PR: #30082 Problem Summary: We are using doris 3.0.4 with java udf, when running long time, some BE may got jvm deadlocks with UNKNOWN_owner_addr [deadlock in udf close] <img width="2314" height="882" alt="image" src="https://github.com/user-attachments/assets/c16b5d18-9900-45dc-8bb4-894180694a44" /> [deadlock in throwableToStackTrace, waiting a StringWriter , but this is impossible , because StringWriter was newed everytime] <img width="1724" height="720" alt="image" src="https://github.com/user-attachments/assets/b586f752-0ba8-43cc-89b4-352a7d9ab00c" /> <img width="618" height="144" alt="image" src="https://github.com/user-attachments/assets/20c2ad11-beef-4b0c-b2a2-343d9b0fd14c" /> In be.WARNING we found get JNIEnv failed when close udf <img width="1659" height="426" alt="image" src="https://github.com/user-attachments/assets/6cf54aa2-522a-407e-a53a-e17fdbc9162e" /> In be.out we found libhdfs error log [JNIEnv is got by libhdfs when in X86 ] Call to AttachCurrentThread failed with error: -1 getJNIEnv: getGlobalJNIEnv failed In be.WARNING there are some other err when using GetJniExceptionMsg <img width="1686" height="530" alt="image" src="https://github.com/user-attachments/assets/35ca7b4c-0a9d-4da3-8be2-9a07ccebb74b" /> we found all those err has common ground 1.all case occured in JavaFunctionCall::close 2.the stack all have bthread keyword after seaching the web , we found that JNI is not compatible with bthread https://blog.csdn.net/qq_46104835/article/details/139360911 https://gitee.com/baidu/BRPC/blob/master/docs/cn/server.md#pthread%E6%A8%A1%E5%BC%8F <img width="1101" height="211" alt="image" src="https://github.com/user-attachments/assets/e1da2f9a-9e37-47d9-b0a1-c1652da93e75" /> the bthread stack layout is not compatible for java, so we may meed 1. AttachCurrentThread[using by getJNIEnv] will check stack layout fail 2. AttachCurrentThread/getJNIEnv may success, but java method may execute failed, even deadlock, because the stack layout has incompatible address for jvm Then we switch bthread to pthread mode , every thing works fine.[add -usercode_in_pthread=true] We want to know how often bhread do the JavaFunctionCall::close, then we add metrics. only 1/10000 JavaFunctionCall::close running in bthread <img width="1886" height="358" alt="image" src="https://github.com/user-attachments/assets/34293bac-696b-41f9-b5eb-4daaab81fb7f" /> But why JavaFunctionCall::close occured in bthread[ after #16634, exec_plan_fragment is running in pthread instead of bthread] then we found a pr #30082 ExchangeSinkBuffer<Parent>::_send_rpc will set a send_callback with a weak_task_ctx <img width="1392" height="254" alt="image" src="https://github.com/user-attachments/assets/a1d28f18-c3c7-4953-baab-d1109f341aa1" /> So sometimes send_callback may using weak_task_ctx.lock() to get a shared_ptr to task_ctx, then task_ctx destructor my be called in send_callback[ send_callback is running in bthread] So We modify JavaFunctionCall::close when JavaFunctionCall::close running in bthread, we submit the jni operation to a pthread pool, and wait it finish because only 1/10000 JavaFunctionCall::close are running in bthread. the pthread pool size can set to a small number.

…bprc #55302 (#57944) Cherry-picked from #55302 Co-authored-by: Fangyuan Deng <[email protected]>

…57918) ### What problem does this PR solve? Problem Summary: this pr introuced #55302 _udf_close_workers thread pool not stop when exit ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…pache#57918) ### What problem does this PR solve? Problem Summary: this pr introuced apache#55302 _udf_close_workers thread pool not stop when exit ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…#55302) ### What problem does this PR solve? Related PR: apache#30082 Problem Summary: We are using doris 3.0.4 with java udf, when running long time, some BE may got jvm deadlocks with UNKNOWN_owner_addr [deadlock in udf close] <img width="2314" height="882" alt="image" src="https://github.com/user-attachments/assets/c16b5d18-9900-45dc-8bb4-894180694a44" /> [deadlock in throwableToStackTrace, waiting a StringWriter , but this is impossible , because StringWriter was newed everytime] <img width="1724" height="720" alt="image" src="https://github.com/user-attachments/assets/b586f752-0ba8-43cc-89b4-352a7d9ab00c" /> <img width="618" height="144" alt="image" src="https://github.com/user-attachments/assets/20c2ad11-beef-4b0c-b2a2-343d9b0fd14c" /> In be.WARNING we found get JNIEnv failed when close udf <img width="1659" height="426" alt="image" src="https://github.com/user-attachments/assets/6cf54aa2-522a-407e-a53a-e17fdbc9162e" /> In be.out we found libhdfs error log [JNIEnv is got by libhdfs when in X86 ] Call to AttachCurrentThread failed with error: -1 getJNIEnv: getGlobalJNIEnv failed In be.WARNING there are some other err when using GetJniExceptionMsg <img width="1686" height="530" alt="image" src="https://github.com/user-attachments/assets/35ca7b4c-0a9d-4da3-8be2-9a07ccebb74b" /> we found all those err has common ground 1.all case occured in JavaFunctionCall::close 2.the stack all have bthread keyword after seaching the web , we found that JNI is not compatible with bthread https://blog.csdn.net/qq_46104835/article/details/139360911 https://gitee.com/baidu/BRPC/blob/master/docs/cn/server.md#pthread%E6%A8%A1%E5%BC%8F <img width="1101" height="211" alt="image" src="https://github.com/user-attachments/assets/e1da2f9a-9e37-47d9-b0a1-c1652da93e75" /> the bthread stack layout is not compatible for java, so we may meed 1. AttachCurrentThread[using by getJNIEnv] will check stack layout fail 2. AttachCurrentThread/getJNIEnv may success, but java method may execute failed, even deadlock, because the stack layout has incompatible address for jvm Then we switch bthread to pthread mode , every thing works fine.[add -usercode_in_pthread=true] We want to know how often bhread do the JavaFunctionCall::close, then we add metrics. only 1/10000 JavaFunctionCall::close running in bthread <img width="1886" height="358" alt="image" src="https://github.com/user-attachments/assets/34293bac-696b-41f9-b5eb-4daaab81fb7f" /> But why JavaFunctionCall::close occured in bthread[ after apache#16634, exec_plan_fragment is running in pthread instead of bthread] then we found a pr apache#30082 ExchangeSinkBuffer<Parent>::_send_rpc will set a send_callback with a weak_task_ctx <img width="1392" height="254" alt="image" src="https://github.com/user-attachments/assets/a1d28f18-c3c7-4953-baab-d1109f341aa1" /> So sometimes send_callback may using weak_task_ctx.lock() to get a shared_ptr to task_ctx, then task_ctx destructor my be called in send_callback[ send_callback is running in bthread] So We modify JavaFunctionCall::close when JavaFunctionCall::close running in bthread, we submit the jni operation to a pthread pool, and wait it finish because only 1/10000 JavaFunctionCall::close are running in bthread. the pthread pool size can set to a small number.

pzhdfy force-pushed the fix_udf_deadlock branch 3 times, most recently from c132b8c to 53b6f23 Compare August 26, 2025 08:13

zclllyybb requested a review from zhangstar333 August 26, 2025 09:04

zhangstar333 approved these changes Aug 27, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2025

github-actions Bot added the reviewed label Aug 27, 2025

pzhdfy added 2 commits September 28, 2025 16:54

fix java udf deadlock

65fa2ea

fix style

1c64463

zhangstar333 force-pushed the fix_udf_deadlock branch from 53b6f23 to 1c64463 Compare September 28, 2025 08:54

Gabriel39 approved these changes Sep 29, 2025

View reviewed changes

zhangstar333 merged commit 7a74f84 into apache:master Sep 29, 2025

zhangstar333 added a commit that referenced this pull request Nov 11, 2025

Revert "[fix](java udf) fix possible deadlock when udf closed in bprc (…

7e78133

…#55302)" This reverts commit 7a74f84.

This was referenced Nov 11, 2025

Revert "[fix](java udf) fix possible deadlock when udf closed in bprc" #57916

Closed

[Bug](udf) fix the _udf_close_workers thread pool not stop when exit #57918

Merged

yiguolei added the dev/4.0.x label Nov 12, 2025

github-actions Bot mentioned this pull request Nov 12, 2025

branch-4.0: [fix](java udf) fix possible deadlock when udf closed in bprc #55302 #57944

Merged

yiguolei pushed a commit that referenced this pull request Nov 12, 2025

branch-4.0: [fix](java udf) fix possible deadlock when udf closed in …

3c3a396

…bprc #55302 (#57944) Cherry-picked from #55302 Co-authored-by: Fangyuan Deng <[email protected]>

yiguolei added dev/4.0.2-merged and removed dev/4.0.x labels Nov 12, 2025

Conversation

pzhdfy commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 26, 2025

Uh oh!

zhangstar333 commented Aug 26, 2025

Uh oh!

doris-robot commented Aug 26, 2025

Uh oh!

doris-robot commented Aug 26, 2025

Uh oh!

doris-robot commented Aug 26, 2025

Uh oh!

doris-robot commented Aug 26, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 26, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actions Bot commented Aug 27, 2025

Uh oh!

github-actions Bot commented Aug 27, 2025

Uh oh!

hello-stephen commented Aug 27, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 27, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 27, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 2, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 2, 2025

BE Regression && UT Coverage Report

Uh oh!

zhangstar333 commented Sep 28, 2025

Uh oh!

doris-robot commented Sep 28, 2025

Uh oh!

hello-stephen commented Sep 28, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 28, 2025

BE Regression && UT Coverage Report

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pzhdfy commented Aug 26, 2025 •

edited

Loading