[third-party](faiss) Enable FAISS integration in Doris. by RoanHeNaN · Pull Request #49644 · apache/doris

RoanHeNaN · 2025-03-28T11:49:10Z

What problem does this PR solve?

Enable FAISS integration in Doris.

Dependency of faiss is OpenMP, BLAS and LAPACK.

OpenMP is distributed with gcc/llvm.
OpenBLAS could supply BLAS & LAPACK impl, so we introduced OpenBLAS.

If you are using ldb-toolchain, and version is before https://github.com/amosbird/ldb_toolchain_gen/releases/tag/v0.24, gcc should be used to compile openblas and faiss, since libopm.a is missing.

Build new thirdparty:

sh build-thirdparty.sh openblas
sh build-thirdparty.sh faiss

export ENABLE_BUILD_FAISS=ON to make doris link with faiss.

Release note

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-03-28T11:49:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

RoanHeNaN · 2025-03-28T11:49:22Z

run buildall

doris-robot · 2025-03-28T12:07:57Z

TPC-H: Total hot run time: 34226 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 834e85ef429d86508f0c79259e2f51476102f464, data reload: false

------ Round 1 ----------------------------------
q1	25827	5122	5072	5072
q2	2059	312	163	163
q3	10441	1227	671	671
q4	10223	1006	517	517
q5	7539	2337	2345	2337
q6	192	167	132	132
q7	900	752	611	611
q8	9327	1262	1136	1136
q9	6787	5099	5136	5099
q10	6874	2307	1889	1889
q11	482	276	267	267
q12	339	359	226	226
q13	17797	3650	3133	3133
q14	237	234	218	218
q15	528	494	492	492
q16	615	614	595	595
q17	592	854	352	352
q18	7650	7269	7111	7111
q19	1670	989	567	567
q20	322	334	202	202
q21	3864	3367	2459	2459
q22	1064	1005	977	977
Total cold run time: 115329 ms
Total hot run time: 34226 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5222	5081	5135	5081
q2	226	326	224	224
q3	2144	2687	2265	2265
q4	1446	1812	1421	1421
q5	4510	4443	4383	4383
q6	209	162	125	125
q7	1924	1877	1752	1752
q8	2586	2508	2439	2439
q9	7117	7063	7152	7063
q10	2957	3164	2779	2779
q11	585	494	481	481
q12	697	749	608	608
q13	3478	3884	3321	3321
q14	278	290	273	273
q15	518	463	482	463
q16	670	674	634	634
q17	1155	1578	1442	1442
q18	7723	7561	7348	7348
q19	845	844	852	844
q20	1973	2026	1931	1931
q21	5096	4829	4760	4760
q22	1087	1017	988	988
Total cold run time: 52446 ms
Total hot run time: 50625 ms

doris-robot · 2025-03-28T12:19:33Z

TPC-DS: Total hot run time: 186010 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 834e85ef429d86508f0c79259e2f51476102f464, data reload: false

query1	1012	481	489	481
query2	6535	1946	1904	1904
query3	6804	222	212	212
query4	26567	23351	23208	23208
query5	4419	669	469	469
query6	282	204	205	204
query7	4615	493	276	276
query8	306	269	222	222
query9	8606	2553	2528	2528
query10	476	300	283	283
query11	15363	15074	14852	14852
query12	159	110	111	110
query13	1660	520	390	390
query14	9271	6276	6396	6276
query15	214	190	172	172
query16	7148	654	464	464
query17	1207	727	580	580
query18	2006	416	314	314
query19	201	192	162	162
query20	130	119	120	119
query21	216	130	104	104
query22	4296	4441	4087	4087
query23	33734	32896	32883	32883
query24	7998	2411	2382	2382
query25	578	481	421	421
query26	1244	273	160	160
query27	2648	490	335	335
query28	4385	2402	2366	2366
query29	747	553	438	438
query30	288	218	191	191
query31	914	825	740	740
query32	72	61	64	61
query33	559	358	312	312
query34	778	851	511	511
query35	803	820	764	764
query36	966	1034	885	885
query37	116	104	76	76
query38	4119	4254	4087	4087
query39	1460	1396	1393	1393
query40	209	114	102	102
query41	57	55	55	55
query42	122	103	104	103
query43	505	515	473	473
query44	1308	808	808	808
query45	179	173	164	164
query46	851	1031	641	641
query47	1742	1799	1707	1707
query48	370	414	301	301
query49	790	516	409	409
query50	671	714	418	418
query51	4195	4201	4145	4145
query52	108	106	96	96
query53	227	259	177	177
query54	484	505	408	408
query55	82	79	86	79
query56	269	294	270	270
query57	1123	1139	1045	1045
query58	244	241	272	241
query59	2593	2713	2518	2518
query60	282	293	279	279
query61	138	159	126	126
query62	782	773	660	660
query63	221	189	187	187
query64	4289	1011	678	678
query65	4420	4245	4273	4245
query66	1149	424	308	308
query67	15706	15597	15404	15404
query68	8750	885	503	503
query69	466	322	267	267
query70	1219	1093	1088	1088
query71	467	289	278	278
query72	5577	4673	4571	4571
query73	686	567	341	341
query74	9001	8953	8937	8937
query75	4239	3267	2771	2771
query76	3690	1199	749	749
query77	793	371	302	302
query78	10152	10259	9297	9297
query79	2484	828	563	563
query80	818	519	442	442
query81	466	256	228	228
query82	587	128	94	94
query83	215	184	156	156
query84	295	97	75	75
query85	783	360	312	312
query86	336	300	304	300
query87	4411	4397	4280	4280
query88	3472	2281	2270	2270
query89	393	316	290	290
query90	1944	208	211	208
query91	137	138	109	109
query92	77	61	60	60
query93	1800	1045	574	574
query94	678	405	316	316
query95	353	276	266	266
query96	484	567	277	277
query97	3230	3210	3138	3138
query98	233	206	215	206
query99	1465	1387	1273	1273
Total cold run time: 275719 ms
Total hot run time: 186010 ms

doris-robot · 2025-03-28T12:24:52Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 834e85ef429d86508f0c79259e2f51476102f464, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.24	0.20	0.19
query4	1.59	0.20	0.11
query5	0.57	0.55	0.56
query6	1.18	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.53	0.53
query10	0.63	0.59	0.57
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.60
query14	2.67	2.68	2.69
query15	0.93	0.86	0.85
query16	0.41	0.37	0.37
query17	1.04	1.04	1.04
query18	0.22	0.19	0.20
query19	1.94	1.93	1.80
query20	0.02	0.01	0.01
query21	15.35	0.88	0.54
query22	0.74	1.35	0.71
query23	14.75	1.36	0.65
query24	6.94	1.70	0.41
query25	0.50	0.26	0.13
query26	0.67	0.15	0.13
query27	0.06	0.05	0.05
query28	9.46	0.87	0.41
query29	12.55	3.96	3.26
query30	0.25	0.09	0.06
query31	2.82	0.58	0.40
query32	3.23	0.54	0.48
query33	2.98	3.02	3.07
query34	15.74	5.10	4.46
query35	4.49	4.49	4.53
query36	0.68	0.50	0.49
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.02	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.83 s
Total hot run time: 30.62 s

hello-stephen · 2025-03-28T14:10:02Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	50.90% (13625/26770)
Line Coverage	40.27% (118287/293742)
Region Coverage	38.95% (60090/154282)
Branch Coverage	33.86% (30227/89260)

github-actions · 2025-03-29T01:17:58Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-29T01:18:00Z

PR approved by anyone and no changes requested.

RoanHeNaN · 2025-03-30T06:37:53Z

run buildall

doris-robot · 2025-03-30T07:14:16Z

TPC-H: Total hot run time: 34092 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47f93c8c57b7f873dc1fcdbdb33d7e1ab9c3a216, data reload: false

------ Round 1 ----------------------------------
q1	25678	5063	5141	5063
q2	2065	278	156	156
q3	10421	1254	666	666
q4	10223	1020	532	532
q5	7519	2357	2334	2334
q6	186	161	131	131
q7	890	748	617	617
q8	9318	1219	1110	1110
q9	6980	5094	5093	5093
q10	6803	2294	1886	1886
q11	465	266	259	259
q12	348	355	224	224
q13	17754	3700	3081	3081
q14	220	224	213	213
q15	537	514	484	484
q16	619	611	589	589
q17	574	847	351	351
q18	7382	7169	7159	7159
q19	1216	946	550	550
q20	325	332	197	197
q21	3923	2613	2459	2459
q22	1046	1050	938	938
Total cold run time: 114492 ms
Total hot run time: 34092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5080	5080	5077	5077
q2	235	324	233	233
q3	2170	2597	2240	2240
q4	1423	1854	1404	1404
q5	4324	4370	4416	4370
q6	214	171	128	128
q7	2008	1886	1780	1780
q8	2551	2626	2547	2547
q9	7291	7141	7016	7016
q10	3037	3149	2771	2771
q11	575	502	519	502
q12	698	761	616	616
q13	3551	3869	3431	3431
q14	276	308	305	305
q15	546	501	500	500
q16	669	689	650	650
q17	1157	1623	1393	1393
q18	7926	7687	7493	7493
q19	823	837	862	837
q20	1994	1968	1812	1812
q21	5303	4901	4731	4731
q22	1120	1088	1042	1042
Total cold run time: 52971 ms
Total hot run time: 50878 ms

doris-robot · 2025-03-30T07:26:11Z

TPC-DS: Total hot run time: 193054 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47f93c8c57b7f873dc1fcdbdb33d7e1ab9c3a216, data reload: false

query1	1406	1078	1021	1021
query2	6248	1927	1900	1900
query3	11121	4581	4698	4581
query4	25649	23137	23209	23137
query5	4777	653	480	480
query6	322	233	194	194
query7	3993	482	271	271
query8	298	254	235	235
query9	8519	2513	2547	2513
query10	476	331	298	298
query11	15527	15167	14761	14761
query12	156	112	107	107
query13	1552	554	376	376
query14	9582	6143	6196	6143
query15	197	177	163	163
query16	7615	655	475	475
query17	1134	728	590	590
query18	2035	409	321	321
query19	200	188	164	164
query20	126	134	118	118
query21	204	129	109	109
query22	4758	4719	4372	4372
query23	34266	33457	33603	33457
query24	8347	2406	2458	2406
query25	524	475	424	424
query26	1348	274	146	146
query27	2934	503	339	339
query28	4413	2429	2412	2412
query29	736	575	444	444
query30	279	230	199	199
query31	940	852	793	793
query32	76	61	65	61
query33	533	368	364	364
query34	784	874	511	511
query35	841	845	766	766
query36	972	1066	900	900
query37	120	100	74	74
query38	4159	4237	4135	4135
query39	1491	1471	1434	1434
query40	214	125	109	109
query41	55	55	56	55
query42	125	117	112	112
query43	516	510	490	490
query44	1335	850	828	828
query45	178	177	173	173
query46	851	1028	652	652
query47	1853	1893	1818	1818
query48	384	413	312	312
query49	737	556	435	435
query50	727	767	436	436
query51	4278	4248	4328	4248
query52	111	109	99	99
query53	229	260	180	180
query54	513	505	428	428
query55	82	82	83	82
query56	288	287	289	287
query57	1206	1232	1129	1129
query58	261	259	258	258
query59	2687	2931	2823	2823
query60	306	298	285	285
query61	188	166	133	133
query62	750	745	684	684
query63	229	185	186	185
query64	4101	1051	699	699
query65	4597	4359	4369	4359
query66	987	403	302	302
query67	16354	15600	15524	15524
query68	8338	888	508	508
query69	488	302	258	258
query70	1190	1129	1095	1095
query71	456	290	264	264
query72	5613	4837	4953	4837
query73	739	712	347	347
query74	9064	8985	8797	8797
query75	3912	3243	2710	2710
query76	3742	1227	762	762
query77	818	367	283	283
query78	10136	10341	9392	9392
query79	2110	822	565	565
query80	590	538	440	440
query81	473	258	215	215
query82	482	126	92	92
query83	173	179	158	158
query84	246	88	71	71
query85	786	360	319	319
query86	339	290	282	282
query87	4396	4552	4392	4392
query88	3675	2210	2222	2210
query89	400	313	271	271
query90	1950	211	208	208
query91	141	185	118	118
query92	73	59	57	57
query93	1327	1056	585	585
query94	655	417	316	316
query95	358	284	264	264
query96	482	562	275	275
query97	3157	3238	3170	3170
query98	222	214	204	204
query99	1445	1395	1270	1270
Total cold run time: 280776 ms
Total hot run time: 193054 ms

doris-robot · 2025-03-30T07:31:29Z

ClickBench: Total hot run time: 31.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47f93c8c57b7f873dc1fcdbdb33d7e1ab9c3a216, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.11	0.10
query3	0.24	0.19	0.19
query4	1.59	0.20	0.20
query5	0.62	0.57	0.58
query6	1.17	0.73	0.70
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.58	0.58	0.57
query11	0.16	0.10	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.60
query14	2.83	2.70	2.75
query15	0.92	0.85	0.85
query16	0.40	0.36	0.37
query17	1.03	1.07	1.04
query18	0.21	0.19	0.19
query19	1.90	1.94	1.80
query20	0.01	0.01	0.01
query21	15.38	0.86	0.56
query22	0.75	1.15	0.67
query23	14.92	1.36	0.64
query24	7.20	1.88	0.76
query25	0.50	0.13	0.06
query26	0.65	0.16	0.14
query27	0.05	0.04	0.05
query28	8.88	0.85	0.44
query29	12.57	4.01	3.33
query30	0.25	0.09	0.06
query31	2.81	0.58	0.38
query32	3.23	0.54	0.47
query33	3.06	3.04	3.16
query34	15.86	5.12	4.49
query35	4.59	4.61	4.54
query36	0.65	0.51	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.14	0.15
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.07 s
Total hot run time: 31.1 s

doris-robot · 2025-03-30T09:01:39Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	50.90% (13625/26770)
Line Coverage	40.27% (118314/293815)
Region Coverage	38.96% (60129/154336)
Branch Coverage	33.87% (30237/89278)

github-actions · 2025-03-31T10:04:10Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? Enable FAISS integration in Doris. Dependency of faiss is OpenMP, BLAS and LAPACK. OpenMP is distributed with gcc/llvm. OpenBLAS could supply BLAS & LAPACK impl, so we introduced OpenBLAS. If you are using ldb-toolchain, and version is before https://github.com/amosbird/ldb_toolchain_gen/releases/tag/v0.24, gcc should be used to compile openblas and faiss, since libopm.a is missing. Build new thirdparty: ``` sh build-thirdparty.sh openblas sh build-thirdparty.sh faiss ``` `export ENABLE_BUILD_FAISS=ON` to make doris link with faiss.

…he#49644)" This reverts commit 8740049.

This reverts commit 8740049.

…he#49644) This reverts commit 8740049.

RoanHeNaN changed the title ~~[third party](faiss) Enable doris link with faiss~~ [third-party](faiss) Enable doris link with faiss Mar 28, 2025

RoanHeNaN marked this pull request as ready for review March 28, 2025 12:04

yiguolei previously approved these changes Mar 29, 2025

View reviewed changes

github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 29, 2025

zclllyybb approved these changes Mar 29, 2025

View reviewed changes

RoanHeNaN added 3 commits March 30, 2025 14:33

D1

ee3126f

faiss

021c763

REF

47f93c8

RoanHeNaN dismissed yiguolei’s stale review via 47f93c8 March 30, 2025 06:37

RoanHeNaN force-pushed the feat-faiss-thirdparty branch from 834e85e to 47f93c8 Compare March 30, 2025 06:37

github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Mar 30, 2025

RoanHeNaN changed the title ~~[third-party](faiss) Enable doris link with faiss~~ [third-party](faiss) Enable FAISS integration in Doris. Mar 31, 2025

yiguolei approved these changes Mar 31, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2025

yiguolei merged commit 8740049 into apache:master Apr 1, 2025

RoanHeNaN added a commit to RoanHeNaN/doris that referenced this pull request Jul 28, 2025

Revert "[third-party](faiss) Enable FAISS integration in Doris. (apac…

3813266

…he#49644)" This reverts commit 8740049.

airborne12 pushed a commit that referenced this pull request Jul 28, 2025

Revert "[third-party](faiss) Enable FAISS integration in Doris. (#49644)

150bc0a

This reverts commit 8740049.

RoanHeNaN added a commit to RoanHeNaN/doris that referenced this pull request Aug 4, 2025

Revert "[third-party](faiss) Enable FAISS integration in Doris. (apac…

abda124

…he#49644) This reverts commit 8740049.

morningman mentioned this pull request Sep 29, 2026

[chore](thirdparty) Add OpenBLAS and datasketches-cpp to the third-party build #68617

Closed

3 of 16 tasks

Conversation

RoanHeNaN commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 28, 2025

Uh oh!

RoanHeNaN commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

hello-stephen commented Mar 28, 2025

BE UT Coverage Report

Uh oh!

github-actions Bot commented Mar 29, 2025

Uh oh!

github-actions Bot commented Mar 29, 2025

Uh oh!

RoanHeNaN commented Mar 30, 2025

Uh oh!

doris-robot commented Mar 30, 2025

Uh oh!

doris-robot commented Mar 30, 2025

Uh oh!

doris-robot commented Mar 30, 2025

Uh oh!

doris-robot commented Mar 30, 2025

BE UT Coverage Report

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

RoanHeNaN commented Mar 28, 2025 •

edited

Loading