Skip to content

[Pick 2.0](segment iterator) fix shrink non-char column coredump - #36466

Merged
airborne12 merged 2 commits into
apache:branch-2.0from
airborne12:pick-2.0
Jun 18, 2024
Merged

[Pick 2.0](segment iterator) fix shrink non-char column coredump#36466
airborne12 merged 2 commits into
apache:branch-2.0from
airborne12:pick-2.0

Conversation

@airborne12

@airborne12 airborne12 commented Jun 18, 2024

Copy link
Copy Markdown
Member

Proposed changes

Pick from (#36275)

@airborne12
[Fix](segment iterator) fix shrink non-char column coredump (apache#3…
63d9add 
…6275)

If we execute a delete predicate in an inverted index table and then
query something from it, it will cause a core dump in
shrink_char_type_column_suffix_zero. This occurs because the delete
predicate column ID in _char_type_idx is incorrectly shifted to the
result column inserted by the inverted index result.

coredump stack like:
```
COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8)
*** Check failure stack trace: ***
F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)
*** Check failure stack trace: ***
F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8)

    at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126
    at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408
    at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090
    at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228
--Type <RET> for more, q to quit, c to continue without paging--c
    at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227
```
@doris-robot

doris-robot commented Jun 18, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12

airborne12 commented Jun 18, 2024

Copy link
Copy Markdown
Member Author

run buildall

@github-actions

github-actions Bot commented Jun 18, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actions Bot commented Jun 18, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Jun 18, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.79% (8094/21416)
Line Coverage: 29.44% (66127/224638)
Region Coverage: 28.91% (34087/117901)
Branch Coverage: 24.78% (17495/70610)
Coverage Report: http://coverage.selectdb-in.cc/coverage/542a9f89310af3344fcdd72d30291af05cb42d12_542a9f89310af3344fcdd72d30291af05cb42d12/report/index.html

@doris-robot

doris-robot commented Jun 18, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 49794 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 542a9f89310af3344fcdd72d30291af05cb42d12, data reload: false

------ Round 1 ----------------------------------
q1	17608	4468	4353	4353
q2	2056	156	146	146
q3	10457	1889	1933	1889
q4	10358	1237	1297	1237
q5	8871	3899	3924	3899
q6	236	129	128	128
q7	2015	1606	1581	1581
q8	9279	2765	2722	2722
q9	10637	10306	10163	10163
q10	8678	3522	3494	3494
q11	422	250	248	248
q12	480	304	298	298
q13	18358	3967	4036	3967
q14	353	330	322	322
q15	521	466	451	451
q16	695	588	581	581
q17	1141	973	1019	973
q18	7320	6908	6855	6855
q19	1787	1666	1613	1613
q20	523	296	294	294
q21	4481	4159	4131	4131
q22	517	454	449	449
Total cold run time: 116793 ms
Total hot run time: 49794 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4278	4310	4306	4306
q2	325	228	230	228
q3	4205	4144	4133	4133
q4	2782	2738	2750	2738
q5	7084	7036	7082	7036
q6	243	121	124	121
q7	3267	2796	2849	2796
q8	4358	4419	4449	4419
q9	16918	16725	16820	16725
q10	4234	4246	4280	4246
q11	759	682	688	682
q12	1036	857	856	856
q13	7098	3734	3769	3734
q14	450	434	430	430
q15	508	465	466	465
q16	742	704	678	678
q17	3731	3811	3800	3800
q18	8858	8824	8956	8824
q19	1741	1704	1680	1680
q20	2407	2194	2103	2103
q21	8515	8511	8476	8476
q22	1068	941	951	941
Total cold run time: 84607 ms
Total hot run time: 79417 ms

@doris-robot

doris-robot commented Jun 18, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 203078 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 542a9f89310af3344fcdd72d30291af05cb42d12, data reload: false

query1	935	424	382	382
query2	6557	2639	2687	2639
query3	6915	215	209	209
query4	20449	17916	18013	17916
query5	19740	6538	6513	6513
query6	292	219	232	219
query7	4160	302	307	302
query8	428	409	406	406
query9	3026	2625	2553	2553
query10	408	304	289	289
query11	11283	10624	10730	10624
query12	127	74	75	74
query13	5586	704	681	681
query14	17599	13360	13405	13360
query15	362	260	248	248
query16	6476	293	261	261
query17	1713	1442	889	889
query18	2311	411	402	402
query19	213	150	152	150
query20	81	78	76	76
query21	188	93	99	93
query22	5272	4984	4826	4826
query23	32628	31946	31865	31865
query24	7036	6649	6587	6587
query25	519	450	427	427
query26	535	161	162	161
query27	1861	291	295	291
query28	6216	2399	2351	2351
query29	2872	2721	2821	2721
query30	243	171	167	167
query31	918	755	752	752
query32	67	63	60	60
query33	409	253	254	253
query34	855	475	468	468
query35	1128	876	930	876
query36	1297	1232	1118	1118
query37	87	60	64	60
query38	3044	2964	2923	2923
query39	1354	1344	1322	1322
query40	204	98	94	94
query41	46	43	43	43
query42	86	84	82	82
query43	787	664	676	664
query44	1191	740	728	728
query45	247	239	235	235
query46	1237	986	969	969
query47	1890	1825	1676	1676
query48	1030	737	698	698
query49	631	370	386	370
query50	866	635	648	635
query51	4762	4633	4613	4613
query52	93	82	83	82
query53	447	325	330	325
query54	2703	2472	2445	2445
query55	96	81	89	81
query56	241	220	215	215
query57	1162	1156	1073	1073
query58	220	205	186	186
query59	4083	3987	4226	3987
query60	210	195	225	195
query61	98	98	96	96
query62	829	490	538	490
query63	500	339	334	334
query64	2340	1511	1466	1466
query65	3624	3576	3562	3562
query66	781	372	390	372
query67	15567	16975	15206	15206
query68	8496	640	652	640
query69	554	353	347	347
query70	1529	1387	1378	1378
query71	393	306	317	306
query72	6504	3486	3483	3483
query73	749	309	319	309
query74	6277	5824	5792	5792
query75	5034	3712	3710	3710
query76	5045	1155	1200	1155
query77	707	256	259	256
query78	12730	11515	11856	11515
query79	8616	637	650	637
query80	1436	402	405	402
query81	505	233	234	233
query82	1462	96	95	95
query83	184	138	133	133
query84	261	69	72	69
query85	876	320	316	316
query86	320	295	305	295
query87	3265	2990	3043	2990
query88	4909	2292	2289	2289
query89	389	280	289	280
query90	1937	208	211	208
query91	183	137	139	137
query92	56	57	57	57
query93	5690	607	577	577
query94	729	209	210	209
query95	1135	1072	1084	1072
query96	655	325	320	320
query97	6463	6403	6386	6386
query98	186	179	169	169
query99	2975	877	924	877
Total cold run time: 311411 ms
Total hot run time: 203078 ms

@doris-robot

doris-robot commented Jun 18, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 542a9f89310af3344fcdd72d30291af05cb42d12, data reload: false

query1	0.02	0.03	0.03
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.80	0.07	0.07
query5	0.54	0.54	0.52
query6	1.23	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.54	0.50	0.47
query10	0.52	0.55	0.54
query11	0.12	0.08	0.08
query12	0.12	0.08	0.09
query13	0.62	0.62	0.61
query14	0.79	0.78	0.79
query15	0.79	0.76	0.76
query16	0.36	0.39	0.36
query17	1.02	1.01	1.03
query18	0.23	0.27	0.25
query19	1.89	1.86	1.85
query20	0.02	0.01	0.01
query21	15.50	0.55	0.55
query22	2.12	2.60	1.63
query23	17.24	0.99	0.96
query24	7.00	0.89	1.11
query25	0.37	0.11	0.04
query26	0.71	0.17	0.15
query27	0.05	0.03	0.03
query28	6.24	0.75	0.71
query29	12.77	2.14	2.20
query30	0.57	0.59	0.54
query31	2.80	0.39	0.37
query32	3.42	0.49	0.50
query33	3.07	3.15	3.06
query34	15.23	4.81	4.78
query35	4.86	4.86	4.83
query36	1.07	1.00	1.01
query37	0.06	0.04	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.17	0.14	0.15
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.01
Total cold run time: 104.39 s
Total hot run time: 30.54 s

@doris-robot

doris-robot commented Jun 18, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 542a9f89310af3344fcdd72d30291af05cb42d12 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@airborne12
airborne12 merged commit c3d4998 into apache:branch-2.0 Jun 18, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@airborne12 caiconghui1
[Pick 2.0](segment iterator) fix shrink non-char column coredump (apa…
83e060f 
…che#36466)

## Proposed changes
Pick from (apache#36275)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@airborne12 @doris-robot