Skip to content

[branch-2.0](txn) be dead exceeds 5min abort its txns (#22781, #28662, #35342) - #39317

Merged
dataroaring merged 6 commits into
apache:branch-2.0from
yujun777:branch-2.0-be-dead-5min-abort-txn
Aug 14, 2024
Merged

[branch-2.0](txn) be dead exceeds 5min abort its txns (#22781, #28662, #35342)#39317
dataroaring merged 6 commits into
apache:branch-2.0from
yujun777:branch-2.0-be-dead-5min-abort-txn

Conversation

@yujun777

@yujun777 yujun777 commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

cherry-pick: #22781, #28662, #35342

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777

yujun777 commented Aug 13, 2024

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot added area/load Issues or PRs related to all kinds of load area/nereids area/planner Issues or PRs related to the query planner kind/docs Categorizes issue or PR as related to documentation. kind/test labels Aug 13, 2024
@github-actions

github-actions Bot commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 49813 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3d07f6cddaf0f294e516e0682b5b3428e3a07a07, data reload: false

------ Round 1 ----------------------------------
q1	17651	4441	4358	4358
q2	2120	162	147	147
q3	10517	1914	1915	1914
q4	10364	1270	1331	1270
q5	8405	3905	3953	3905
q6	233	128	128	128
q7	2060	1627	1622	1622
q8	9308	2764	2745	2745
q9	10748	10351	10274	10274
q10	8635	3533	3528	3528
q11	412	255	252	252
q12	470	305	308	305
q13	18355	3966	4023	3966
q14	363	337	325	325
q15	523	460	456	456
q16	689	575	569	569
q17	1160	990	967	967
q18	7336	6868	6761	6761
q19	1714	1604	1492	1492
q20	521	301	315	301
q21	4431	4251	4126	4126
q22	491	402	404	402
Total cold run time: 116506 ms
Total hot run time: 49813 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4344	4352	4280	4280
q2	322	223	226	223
q3	4208	4142	4158	4142
q4	2751	2744	2736	2736
q5	7198	7145	7152	7145
q6	235	121	117	117
q7	3262	2872	2890	2872
q8	4402	4509	4510	4509
q9	16964	16717	16887	16717
q10	4275	4281	4281	4281
q11	771	698	697	697
q12	1032	847	834	834
q13	6958	3704	3725	3704
q14	444	420	427	420
q15	503	456	461	456
q16	737	674	678	674
q17	3939	3839	3793	3793
q18	8883	8659	8799	8659
q19	1718	1700	1644	1644
q20	2376	2120	2104	2104
q21	8559	8500	8404	8404
q22	999	1001	942	942
Total cold run time: 84880 ms
Total hot run time: 79353 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 201824 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3d07f6cddaf0f294e516e0682b5b3428e3a07a07, data reload: false

query1	922	394	411	394
query2	6550	2216	1976	1976
query3	6921	206	202	202
query4	20225	18015	18003	18003
query5	19742	6469	6481	6469
query6	283	224	229	224
query7	4170	308	305	305
query8	282	252	261	252
query9	3172	2721	2670	2670
query10	416	299	303	299
query11	11235	10758	10652	10652
query12	122	79	78	78
query13	5597	670	650	650
query14	17475	13849	13062	13062
query15	360	220	235	220
query16	6476	289	264	264
query17	1717	1443	895	895
query18	2313	422	434	422
query19	208	155	152	152
query20	82	77	78	77
query21	192	102	95	95
query22	5175	4995	4995	4995
query23	32901	31942	31981	31942
query24	7008	6494	6664	6494
query25	522	436	441	436
query26	609	166	163	163
query27	1997	297	304	297
query28	6177	2317	2286	2286
query29	2935	2735	2689	2689
query30	243	165	166	165
query31	913	749	747	747
query32	68	60	63	60
query33	405	253	247	247
query34	856	483	485	483
query35	1129	965	913	913
query36	1227	1271	1049	1049
query37	85	65	61	61
query38	3098	2930	2926	2926
query39	1378	1311	1330	1311
query40	204	99	99	99
query41	50	37	37	37
query42	99	85	92	85
query43	635	561	701	561
query44	1149	725	725	725
query45	246	233	229	229
query46	1230	976	967	967
query47	1946	1588	1621	1588
query48	989	697	692	692
query49	634	368	384	368
query50	848	556	606	556
query51	4736	4665	4658	4658
query52	101	80	82	80
query53	441	327	321	321
query54	2646	2454	2451	2451
query55	106	81	88	81
query56	221	227	204	204
query57	1171	1133	1174	1133
query58	217	204	201	201
query59	3554	3336	3193	3193
query60	215	218	205	205
query61	96	91	94	91
query62	800	511	448	448
query63	485	341	352	341
query64	2624	1500	1477	1477
query65	3641	3602	3567	3567
query66	819	371	377	371
query67	17697	15825	17289	15825
query68	8614	654	662	654
query69	571	359	339	339
query70	1649	1357	1396	1357
query71	402	312	321	312
query72	6544	3461	3485	3461
query73	732	321	318	318
query74	6304	5823	5755	5755
query75	4760	3746	3674	3674
query76	4911	1177	1182	1177
query77	676	258	253	253
query78	12636	12252	11568	11568
query79	10615	653	638	638
query80	1667	397	392	392
query81	499	235	231	231
query82	1439	95	102	95
query83	161	132	133	132
query84	268	72	70	70
query85	1133	325	317	317
query86	367	312	342	312
query87	3242	2995	3030	2995
query88	5216	2301	2331	2301
query89	481	307	306	306
query90	1828	208	208	208
query91	151	123	121	121
query92	61	60	52	52
query93	6841	605	572	572
query94	797	201	198	198
query95	1120	1059	1057	1057
query96	635	325	325	325
query97	6386	6354	6364	6354
query98	198	178	171	171
query99	2946	930	945	930
Total cold run time: 316636 ms
Total hot run time: 201824 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3d07f6cddaf0f294e516e0682b5b3428e3a07a07, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.04	0.05
query4	1.80	0.07	0.06
query5	0.55	0.53	0.52
query6	1.23	0.62	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.52	0.51	0.47
query10	0.54	0.52	0.55
query11	0.13	0.09	0.08
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.78	0.78	0.80
query15	0.80	0.77	0.77
query16	0.36	0.39	0.37
query17	1.02	1.01	1.03
query18	0.24	0.26	0.25
query19	1.95	1.76	1.86
query20	0.01	0.01	0.01
query21	15.46	0.57	0.56
query22	2.05	2.54	2.27
query23	17.11	0.94	0.96
query24	3.81	3.33	1.97
query25	0.39	0.12	0.05
query26	0.72	0.15	0.15
query27	0.04	0.04	0.04
query28	6.06	0.74	0.72
query29	12.64	2.25	2.10
query30	0.58	0.55	0.56
query31	2.80	0.38	0.37
query32	3.38	0.49	0.49
query33	3.06	3.05	3.03
query34	15.26	4.77	4.77
query35	4.83	4.82	4.82
query36	1.06	1.01	1.03
query37	0.06	0.05	0.05
query38	0.05	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 100.71 s
Total hot run time: 32.14 s

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3d07f6cddaf0f294e516e0682b5b3428e3a07a07 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.71% (8103/21487)
Line Coverage: 29.35% (66393/226205)
Region Coverage: 28.85% (34244/118708)
Branch Coverage: 24.73% (17600/71170)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3d07f6cddaf0f294e516e0682b5b3428e3a07a07_3d07f6cddaf0f294e516e0682b5b3428e3a07a07/report/index.html

@yujun777 yujun777 changed the title [improvement](txn) be dead exceeds 5min abort its txns (#22781, #28662, #35342) [branch-2.0](txn) be dead exceeds 5min abort its txns (#22781, #28662, #35342) Aug 13, 2024
@dataroaring
dataroaring merged commit eaffb69 into apache:branch-2.0 Aug 14, 2024
@yujun777 yujun777 mentioned this pull request Aug 23, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 23, 2024
@yujun777
[branch-2.0](thrift) fix TLoadTxnBeginRequest backend_id's field no (#…
5ec7f46 
…39821)

Make backend_id's field no the same with master branch.

For upgrading, change backend_id's field no is safe, because old fe can
torrent with TLoadTxnBeginRequest not setting backend id.

backend_id was introduce by #39317
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@yujun777 @sollhui @Baymine
[branch-2.0](txn) be dead exceeds 5min abort its txns (apache#22781, a…
9e321a9 
…pache#28662, apache#35342) (apache#39317)

cherry-pick:  apache#22781,  apache#28662, apache#35342

---------

Co-authored-by: HHoflittlefish777 <[email protected]>
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@yujun777 @Baymine
[branch-2.0](thrift) fix TLoadTxnBeginRequest backend_id's field no (a…
c45df48 
…pache#39821)

Make backend_id's field no the same with master branch.

For upgrading, change backend_id's field no is safe, because old fe can
torrent with TLoadTxnBeginRequest not setting backend id.

backend_id was introduce by apache#39317
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/load Issues or PRs related to all kinds of load area/nereids area/planner Issues or PRs related to the query planner kind/docs Categorizes issue or PR as related to documentation. kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yujun777 @doris-robot @dataroaring @sollhui