Skip to content

[Opt](encoding) Add new encoding BinaryPlainPageV2 - #56846

Merged
dataroaring merged 14 commits into
apache:masterfrom
bobhan1:binary-plain-encoding-v2
Oct 22, 2025
Merged

dataroaring merged 14 commits into
apache:masterfrom
bobhan1:binary-plain-encoding-v2

Conversation

@bobhan1

@bobhan1 bobhan1 commented Oct 11, 2025 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

  1. Currently, plain binary are encoding as the following
|binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|...

This PR add a new plain binary encoding which have better storage occupation when binaries are small

|length1(varuint32)|binary1||length2(varuint32)|binary2|...

add a be conf binary_plain_encoding_default_impl to control.

  1. move the registration of column types' encoding to ExecEnv::_init

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
bobhan1 force-pushed the binary-plain-encoding-v2 branch 3 times, most recently from 24a0434 to 70f064f Compare October 11, 2025 08:41
@bobhan1
bobhan1 marked this pull request as ready for review October 11, 2025 08:44
@bobhan1
bobhan1 force-pushed the binary-plain-encoding-v2 branch from 70f064f to 329c4de Compare October 11, 2025 08:44
Comment thread gensrc/proto/segment_v2.proto Resolved
@dataroaring

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.38% (1635/1961)
Line Coverage 67.89% (28906/42578)
Region Coverage 68.15% (14250/20911)
Branch Coverage 58.43% (7586/12982)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 190704 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 329c4de638c6b03f7648e0b92d6467dcf824bc57, data reload: false

query1	1093	407	407	407
query2	6579	1699	1710	1699
query3	6765	230	221	221
query4	26903	23516	23434	23434
query5	4968	630	520	520
query6	322	248	231	231
query7	4676	504	307	307
query8	294	262	238	238
query9	8742	2592	2579	2579
query10	508	346	308	308
query11	15539	15750	14794	14794
query12	185	117	112	112
query13	1681	562	426	426
query14	12021	9260	9373	9260
query15	211	198	175	175
query16	7842	764	544	544
query17	1524	911	771	771
query18	2068	469	376	376
query19	233	235	203	203
query20	136	144	147	144
query21	227	160	117	117
query22	4667	4757	4554	4554
query23	34849	33775	33803	33775
query24	8254	2492	2496	2492
query25	682	524	493	493
query26	1308	290	167	167
query27	2972	521	402	402
query28	4920	2260	2226	2226
query29	820	635	515	515
query30	311	235	215	215
query31	961	902	748	748
query32	80	76	73	73
query33	623	389	376	376
query34	821	894	562	562
query35	862	874	801	801
query36	974	1039	902	902
query37	118	109	86	86
query38	3586	3518	3469	3469
query39	1506	1439	1414	1414
query40	237	126	121	121
query41	64	58	59	58
query42	125	112	112	112
query43	534	519	484	484
query44	1331	844	827	827
query45	183	179	177	177
query46	842	997	647	647
query47	1805	1810	1719	1719
query48	399	434	328	328
query49	761	492	403	403
query50	642	687	424	424
query51	3984	3843	3944	3843
query52	116	107	105	105
query53	235	256	202	202
query54	612	589	534	534
query55	91	91	86	86
query56	339	323	327	323
query57	1195	1190	1124	1124
query58	288	277	279	277
query59	2587	2697	2517	2517
query60	343	365	337	337
query61	173	174	160	160
query62	844	718	673	673
query63	231	192	195	192
query64	4337	1145	834	834
query65	4072	3967	3947	3947
query66	1103	436	324	324
query67	15761	15464	15138	15138
query68	7991	936	597	597
query69	468	315	342	315
query70	1376	1345	1273	1273
query71	490	344	319	319
query72	5911	4917	4977	4917
query73	597	579	371	371
query74	9174	9133	8996	8996
query75	4337	3338	2809	2809
query76	4731	1148	745	745
query77	940	411	311	311
query78	9624	9821	8901	8901
query79	3711	840	587	587
query80	731	619	501	501
query81	488	256	222	222
query82	724	165	132	132
query83	292	266	245	245
query84	304	118	94	94
query85	892	462	423	423
query86	350	311	283	283
query87	3821	3720	3671	3671
query88	2890	2256	2239	2239
query89	443	333	296	296
query90	1998	219	215	215
query91	163	164	138	138
query92	93	68	66	66
query93	2230	988	652	652
query94	690	450	341	341
query95	402	322	304	304
query96	485	570	290	290
query97	2963	3002	2857	2857
query98	250	217	211	211
query99	1457	1386	1331	1331
Total cold run time: 285081 ms
Total hot run time: 190704 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 329c4de638c6b03f7648e0b92d6467dcf824bc57, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.06	0.06
query3	0.26	0.09	0.08
query4	1.60	0.12	0.11
query5	0.28	0.27	0.25
query6	1.17	0.64	0.64
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.64	0.53	0.52
query10	0.59	0.60	0.59
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.65	0.62	0.62
query14	1.04	1.05	1.04
query15	0.86	0.84	0.88
query16	0.40	0.40	0.40
query17	1.02	1.04	1.03
query18	0.23	0.20	0.20
query19	1.95	1.84	1.85
query20	0.01	0.02	0.01
query21	15.44	0.95	0.57
query22	0.79	1.23	0.92
query23	14.71	1.42	0.64
query24	6.91	1.65	0.69
query25	0.51	0.26	0.12
query26	0.64	0.16	0.13
query27	0.06	0.05	0.05
query28	9.20	1.36	0.94
query29	12.56	3.92	3.27
query30	0.28	0.14	0.11
query31	2.83	0.61	0.39
query32	3.24	0.56	0.49
query33	3.10	3.08	3.07
query34	16.15	5.44	4.82
query35	4.93	4.93	4.87
query36	0.71	0.51	0.50
query37	0.10	0.07	0.08
query38	0.07	0.05	0.04
query39	0.04	0.02	0.03
query40	0.18	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.87 s
Total hot run time: 30.52 s

@bobhan1
bobhan1 force-pushed the binary-plain-encoding-v2 branch 4 times, most recently from a74cc37 to d1fe840 Compare October 16, 2025 11:25
@dataroaring

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.04% (29050/43335)
Region Coverage 67.30% (14365/21346)
Branch Coverage 57.69% (7638/13240)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 190075 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d1fe84082ce6f7343fd4a19da9baf83c75626bb0, data reload: false

query1	1065	436	401	401
query2	6568	1699	1661	1661
query3	6753	221	223	221
query4	27011	23715	23103	23103
query5	4931	655	500	500
query6	342	233	219	219
query7	4647	500	292	292
query8	310	286	250	250
query9	8722	2588	2597	2588
query10	531	344	294	294
query11	15430	15150	14868	14868
query12	186	117	113	113
query13	1674	562	446	446
query14	11265	9252	9465	9252
query15	213	191	179	179
query16	8434	726	515	515
query17	1190	781	662	662
query18	2138	457	350	350
query19	271	218	195	195
query20	169	139	146	139
query21	252	143	118	118
query22	4654	4584	4576	4576
query23	34391	34054	33684	33684
query24	8515	2502	2512	2502
query25	684	544	532	532
query26	1267	290	168	168
query27	2992	540	385	385
query28	4366	2227	2219	2219
query29	850	711	569	569
query30	334	258	223	223
query31	947	867	710	710
query32	85	73	71	71
query33	605	381	342	342
query34	827	838	534	534
query35	843	822	747	747
query36	952	1004	889	889
query37	129	113	92	92
query38	3499	3584	3461	3461
query39	1467	1403	1421	1403
query40	225	136	129	129
query41	65	63	110	63
query42	128	118	111	111
query43	488	502	473	473
query44	1385	845	851	845
query45	182	180	174	174
query46	860	1003	662	662
query47	1708	1788	1684	1684
query48	397	447	324	324
query49	771	500	422	422
query50	632	699	414	414
query51	3894	4011	4048	4011
query52	111	116	105	105
query53	243	284	204	204
query54	615	596	572	572
query55	88	85	89	85
query56	325	339	312	312
query57	1169	1191	1123	1123
query58	294	308	276	276
query59	2569	2632	2580	2580
query60	359	346	323	323
query61	165	174	151	151
query62	833	737	679	679
query63	237	196	201	196
query64	4392	1157	857	857
query65	4053	4003	3958	3958
query66	1086	438	330	330
query67	15565	15194	15256	15194
query68	9557	908	607	607
query69	484	354	290	290
query70	1413	1335	1212	1212
query71	500	352	319	319
query72	5530	4894	4923	4894
query73	720	575	370	370
query74	9295	8980	8658	8658
query75	4493	3357	2882	2882
query76	4015	1167	758	758
query77	1011	418	325	325
query78	9531	9782	8819	8819
query79	3900	826	571	571
query80	721	566	565	565
query81	507	261	234	234
query82	478	162	131	131
query83	291	269	256	256
query84	305	108	99	99
query85	901	467	434	434
query86	341	311	298	298
query87	3790	3738	3619	3619
query88	2924	2275	2270	2270
query89	457	335	292	292
query90	2022	222	222	222
query91	161	159	135	135
query92	76	67	65	65
query93	2449	1013	644	644
query94	697	462	341	341
query95	402	324	320	320
query96	491	598	289	289
query97	2973	3013	2925	2925
query98	233	219	220	219
query99	1502	1424	1282	1282
Total cold run time: 284518 ms
Total hot run time: 190075 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d1fe84082ce6f7343fd4a19da9baf83c75626bb0, data reload: false

query1	0.05	0.05	0.05
query2	0.12	0.06	0.06
query3	0.31	0.08	0.08
query4	1.62	0.08	0.09
query5	0.26	0.26	0.25
query6	1.17	0.64	0.64
query7	0.03	0.03	0.06
query8	0.07	0.05	0.05
query9	0.67	0.54	0.54
query10	0.58	0.58	0.59
query11	0.26	0.13	0.13
query12	0.25	0.14	0.14
query13	0.66	0.63	0.64
query14	1.04	1.05	1.05
query15	0.94	0.89	0.86
query16	0.39	0.39	0.40
query17	1.06	1.05	1.04
query18	0.24	0.22	0.23
query19	1.98	1.86	1.86
query20	0.02	0.02	0.01
query21	15.41	1.00	0.70
query22	0.93	1.06	0.85
query23	14.73	1.49	0.86
query24	5.10	0.63	0.34
query25	0.17	0.10	0.09
query26	0.56	0.21	0.18
query27	0.09	0.08	0.08
query28	11.13	1.63	1.04
query29	12.56	3.99	3.34
query30	0.33	0.12	0.09
query31	2.83	0.64	0.44
query32	3.24	0.60	0.51
query33	3.21	3.19	3.06
query34	16.70	5.47	4.78
query35	4.86	4.90	4.88
query36	0.66	0.53	0.53
query37	0.23	0.20	0.20
query38	0.20	0.18	0.17
query39	0.06	0.05	0.05
query40	0.21	0.19	0.18
query41	0.11	0.06	0.06
query42	0.08	0.06	0.06
query43	0.06	0.05	0.05
Total cold run time: 105.18 s
Total hot run time: 31.23 s

@bobhan1
bobhan1 force-pushed the binary-plain-encoding-v2 branch 2 times, most recently from 92e32d6 to 2409b58 Compare October 17, 2025 06:50
@bobhan1

bobhan1 commented Oct 17, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.07% (29062/43334)
Region Coverage 67.37% (14380/21346)
Branch Coverage 57.70% (7639/13240)

@bobhan1
bobhan1 force-pushed the binary-plain-encoding-v2 branch from 7503128 to 50d3f61 Compare October 17, 2025 07:39
@bobhan1

bobhan1 commented Oct 17, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.06% (29059/43334)
Region Coverage 67.37% (14381/21346)
Branch Coverage 57.70% (7640/13240)

@bobhan1
bobhan1 force-pushed the binary-plain-encoding-v2 branch from 2fb272f to 9dc5ca1 Compare October 21, 2025 07:26
@bobhan1
bobhan1 force-pushed the binary-plain-encoding-v2 branch from 5436de4 to b782827 Compare October 21, 2025 07:33
@bobhan1

bobhan1 commented Oct 21, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.03% (29053/43346)
Region Coverage 67.31% (14373/21352)
Branch Coverage 57.65% (7636/13246)

@bobhan1

bobhan1 commented Oct 21, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.05% (29063/43346)
Region Coverage 67.34% (14378/21352)
Branch Coverage 57.70% (7643/13246)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 82.83% (304/367) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.65% (17930/34055)
Line Coverage 37.88% (162629/429309)
Region Coverage 32.30% (124099/384190)
Branch Coverage 33.70% (54349/161280)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 87.73% (329/375) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.45% (23857/33388)
Line Coverage 57.81% (248049/429069)
Region Coverage 52.89% (205840/389154)
Branch Coverage 54.57% (88504/162179)

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 22, 2025
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@bobhan1 bobhan1 mentioned this pull request Oct 22, 2025
16 tasks
@dataroaring
dataroaring merged commit 8226a2a into apache:master Oct 22, 2025
11 of 12 checks passed
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Oct 24, 2025
1. Currently, plain binary are encoding as the following
```
|binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|...
```
This PR add a new plain binary encoding which have better storage
occupation when binaries are small
```
|length1(varuint32)|binary1||length2(varuint32)|binary2|...
```
add a be conf `binary_plain_encoding_default_impl` to control.

2. move the registration of column types' encoding to `ExecEnv::_init`
bobhan1 added a commit to bobhan1/doris that referenced this pull request Oct 28, 2025
1. Currently, plain binary are encoding as the following
```
|binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|...
```
This PR add a new plain binary encoding which have better storage
occupation when binaries are small
```
|length1(varuint32)|binary1||length2(varuint32)|binary2|...
```
add a be conf `binary_plain_encoding_default_impl` to control.

2. move the registration of column types' encoding to `ExecEnv::_init`
bobhan1 added a commit to bobhan1/doris that referenced this pull request Oct 30, 2025
1. Currently, plain binary are encoding as the following
```
|binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|...
```
This PR add a new plain binary encoding which have better storage
occupation when binaries are small
```
|length1(varuint32)|binary1||length2(varuint32)|binary2|...
```
add a be conf `binary_plain_encoding_default_impl` to control.

2. move the registration of column types' encoding to `ExecEnv::_init`
bobhan1 added a commit to bobhan1/doris that referenced this pull request Dec 17, 2025
1. Currently, plain binary are encoding as the following
```
|binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|...
```
This PR add a new plain binary encoding which have better storage
occupation when binaries are small
```
|length1(varuint32)|binary1||length2(varuint32)|binary2|...
```
add a be conf `binary_plain_encoding_default_impl` to control.

2. move the registration of column types' encoding to `ExecEnv::_init`
yiguolei pushed a commit that referenced this pull request Dec 19, 2025
…57220 #57327 #59075 (#57441)

### What problem does this PR solve?
pick:
- #56815
- #56846
- #57220
- #57327
- #59075

Clickbench result on alicloud c6.4xlarge machine, Intel(R) Xeon(R)
Platinum 8269CY CPU @ 2.50GHz

||(baseline) LZ4F, 64KB data page size, 48 bucket,
binary_plain_encoding_default_impl=v1,
integer_type_default_use_plain_encoding =false| zstd(3), 1MB data page
size, 48 bucket, binary_plain_encoding_default_impl=v1,
integer_type_default_use_plain_encoding =false | zstd(3), 1MB data page
size, 48 bucket, binary_plain_encoding_default_impl=v2,
integer_type_default_use_plain_encoding =true |
|---|---|---|---|
|data size|16.087 GB| 11.016 GB | 8.56 GB |
|load time|592s| 555s | 539s |
|cold run total| 214.22 s | 177.57 s | 131.4 s |
|hot run total| 36.03 s | 39.82 s | 39.05 s |
### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants