[Opt](storage) Add pre decoder for BinaryPlainPageV2 and BinaryDictPage by bobhan1 · Pull Request #57327 · apache/doris

bobhan1 · 2025-10-25T17:34:09Z

What problem does this PR solve?

Summary

This PR implements pre-decoders for BinaryPlainPageV2 and BinaryDictPage formats to enable O(1) random access seeking by converting page data at read time. The main improvements include:

✨ New BinaryPlainPageV2PreDecoder: Converts varint-encoded V2 format to offset-array V1 format for faster seeking
✨ New BinaryDictPagePreDecoder: Pre-decodes dictionary pages for efficient random access
🔧 Encoding Configuration: Adds enable_fuzzy_storage_encoding flag to control fuzzy encoding testing
🐛 Bug Fixes: Fixed issues in FileColumnIterator::_read_dict_data and dictionary page handling
🧪 Comprehensive Tests: Added unit tests covering all encoding types and pre-decoder scenarios
⚡ Performance Optimization: Reduced redundant memcpy operations in pre-decoder pipeline

Technical Details

Pre-Decoder Design

BinaryPlainPageV2PreDecoder (be/src/olap/rowset/segment_v2/binary_plain_page_v2_pre_decoder.h:41)

Converts V2 format (varint lengths) → V1 format (offset array)
Enables O(1) seeking without scanning entire page
Reduces memory copy overhead

BinaryDictPagePreDecoder (be/src/olap/rowset/segment_v2/binary_dict_page_pre_decoder.h)

Pre-decodes dictionary-encoded binary pages
Integrates with existing page cache infrastructure

Configuration Changes

Added enable_fuzzy_storage_encoding config flag (be/src/common/config.h:1677)
Default: Uses V1 format for stability and compatibility
V2 format remains available for fuzzy testing when enabled

Files Changed

Core Implementation: binary_plain_page_v2_pre_decoder.h, binary_dict_page_pre_decoder.h
Integration: encoding_info.cpp, column_reader.cpp, page_io.cpp
Tests: encoding_info_test.cpp, binary_plain_page_v2_test.cpp, binary_dict_page_test.cpp
Config: config.h, config.cpp, regression test configs

Test Plan

Unit tests for BinaryPlainPageV2PreDecoder with all data types
Unit tests for BinaryDictPagePreDecoder
EncodingInfo tests covering all type × encoding combinations
Fixed existing binary page tests
Regression tests with both V1 and V2 formats

Impact

Performance: Improves random access performance for binary columns
Compatibility: No breaking changes; V1 format used by default
Risk: Low - pre-decoding is transparent to upper layers

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-10-25T17:34:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2025-10-25T18:10:29Z

run buildall

bobhan1 · 2025-10-25T18:18:32Z

run buildall

bobhan1 · 2025-10-26T03:59:03Z

run buildall

doris-robot · 2025-10-26T05:50:05Z

ClickBench: Total hot run time: 27.38 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9342aa68dc5467f1e3f4b6dc83eecacb0d1023d5, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.26	0.27	0.25
query6	1.17	0.65	0.63
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.61	0.53	0.53
query10	0.58	0.57	0.57
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.63	0.61	0.61
query14	1.02	1.00	1.02
query15	0.86	0.84	0.85
query16	0.38	0.39	0.39
query17	1.01	1.02	1.01
query18	0.21	0.19	0.20
query19	1.89	1.83	1.81
query20	0.02	0.01	0.01
query21	15.44	0.17	0.13
query22	5.19	0.07	0.05
query23	15.68	0.26	0.10
query24	2.78	0.93	0.38
query25	0.08	0.06	0.05
query26	0.15	0.13	0.12
query27	0.06	0.05	0.06
query28	4.50	1.16	0.93
query29	12.58	3.99	3.29
query30	0.27	0.13	0.11
query31	2.82	0.58	0.37
query32	3.22	0.54	0.49
query33	3.02	3.03	3.03
query34	15.96	5.17	4.55
query35	4.57	4.57	4.62
query36	0.68	0.51	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.15	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 98.48 s
Total hot run time: 27.38 s

bobhan1 · 2025-10-27T08:39:26Z

run buildall

doris-robot · 2025-10-27T10:16:53Z

ClickBench: Total hot run time: 28.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 699dbb55d3326c603e73a192522896741be6a36d, data reload: false

query1	0.03	0.04	0.03
query2	0.08	0.04	0.04
query3	0.25	0.08	0.10
query4	1.60	0.11	0.11
query5	0.26	0.26	0.25
query6	1.23	0.67	0.67
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.62	0.53	0.52
query10	0.59	0.58	0.58
query11	0.15	0.10	0.11
query12	0.16	0.12	0.12
query13	0.63	0.60	0.61
query14	1.01	1.02	1.00
query15	0.86	0.82	0.83
query16	0.40	0.39	0.39
query17	1.02	1.06	1.01
query18	0.22	0.20	0.20
query19	1.89	1.84	1.85
query20	0.01	0.01	0.01
query21	15.43	0.18	0.14
query22	5.10	0.08	0.04
query23	15.64	0.25	0.11
query24	3.34	0.79	0.89
query25	0.09	0.06	0.06
query26	0.13	0.12	0.14
query27	0.07	0.05	0.05
query28	4.88	1.15	0.94
query29	12.57	3.96	3.28
query30	0.27	0.13	0.11
query31	2.82	0.58	0.39
query32	3.23	0.56	0.48
query33	3.02	3.11	3.18
query34	15.55	5.18	4.58
query35	4.54	4.58	4.56
query36	0.68	0.51	0.51
query37	0.11	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.05	0.04	0.03
Total cold run time: 99.03 s
Total hot run time: 28.04 s

bobhan1 · 2025-10-27T12:01:44Z

run buildall

doris-robot · 2025-10-27T13:37:35Z

ClickBench: Total hot run time: 27.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e881118311144b8ab5a0733e835f40161a977133, data reload: false

query1	0.06	0.05	0.04
query2	0.09	0.05	0.06
query3	0.26	0.09	0.09
query4	1.61	0.11	0.12
query5	0.26	0.26	0.26
query6	1.19	0.65	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.06
query9	0.65	0.52	0.52
query10	0.59	0.57	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.62	0.60	0.63
query14	1.00	1.03	0.99
query15	0.84	0.83	0.88
query16	0.40	0.40	0.39
query17	1.02	0.99	1.01
query18	0.22	0.20	0.19
query19	1.93	1.77	1.83
query20	0.02	0.02	0.01
query21	15.44	0.19	0.14
query22	5.14	0.06	0.05
query23	15.69	0.24	0.09
query24	1.98	0.91	0.29
query25	0.07	0.05	0.06
query26	0.15	0.14	0.14
query27	0.06	0.05	0.06
query28	3.32	1.16	0.94
query29	12.54	3.91	3.28
query30	0.29	0.14	0.12
query31	2.81	0.59	0.38
query32	3.24	0.55	0.46
query33	3.05	3.01	3.04
query34	15.98	5.18	4.54
query35	4.59	4.62	4.61
query36	0.68	0.51	0.50
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.02	0.02
query40	0.19	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.05	0.04	0.03
Total cold run time: 96.77 s
Total hot run time: 27.35 s

bobhan1 · 2025-10-28T02:27:36Z

run buildall

doris-robot · 2025-10-28T04:14:46Z

ClickBench: Total hot run time: 27.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72d5a719694f8f9fe74446f985180b3820253fa5, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.06	0.05
query3	0.27	0.09	0.08
query4	1.61	0.11	0.11
query5	0.29	0.26	0.24
query6	1.15	0.66	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.62	0.52	0.50
query10	0.58	0.59	0.59
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.63	0.60
query14	1.02	1.01	1.00
query15	0.85	0.83	0.85
query16	0.43	0.40	0.39
query17	1.05	1.06	1.03
query18	0.22	0.21	0.21
query19	1.93	1.85	1.83
query20	0.02	0.01	0.02
query21	15.44	0.18	0.13
query22	5.16	0.07	0.05
query23	15.66	0.26	0.11
query24	3.02	0.83	0.68
query25	0.10	0.06	0.08
query26	0.15	0.14	0.13
query27	0.07	0.05	0.05
query28	5.28	1.13	0.92
query29	12.60	3.97	3.28
query30	0.29	0.14	0.11
query31	2.82	0.59	0.38
query32	3.24	0.55	0.46
query33	3.02	3.10	3.02
query34	15.80	5.18	4.55
query35	4.54	4.53	4.57
query36	0.70	0.50	0.49
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.15
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 99.67 s
Total hot run time: 27.77 s

hello-stephen · 2025-10-28T05:43:38Z

BE UT Coverage Report

Increment line coverage 72.37% (110/152) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.76% (18052/34216)
Line Coverage	38.00% (163716/430871)
Region Coverage	32.37% (124752/385361)
Branch Coverage	33.74% (54557/161706)

hello-stephen · 2025-10-28T07:12:11Z

BE Regression && UT Coverage Report

Increment line coverage 76.32% (116/152) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.36% (23919/33521)
Line Coverage	57.74% (248673/430672)
Region Coverage	52.76% (205807/390076)
Branch Coverage	54.53% (88595/162479)

bobhan1 · 2025-10-29T02:37:46Z

run buildall

doris-robot · 2025-10-29T06:04:25Z

BE UT Coverage Report

Increment line coverage 71.05% (108/152) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.74% (18052/34226)
Line Coverage	37.98% (163710/431030)
Region Coverage	32.35% (124713/385547)
Branch Coverage	33.73% (54557/161764)

hello-stephen · 2025-10-29T06:42:10Z

BE Regression && UT Coverage Report

Increment line coverage 75.00% (114/152) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.37% (23930/33531)
Line Coverage	57.81% (249054/430838)
Region Coverage	52.94% (206625/390312)
Branch Coverage	54.65% (88824/162547)

bobhan1 · 2025-10-30T06:16:17Z

run buildall

doris-robot · 2025-10-30T08:34:06Z

ClickBench: Total hot run time: 29.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f292d0c15f47140f05759c77c1982c4890bb03b0, data reload: false

query1	0.07	0.06	0.07
query2	0.11	0.05	0.07
query3	0.26	0.09	0.10
query4	1.63	0.13	0.13
query5	0.31	0.28	0.27
query6	1.19	0.69	0.68
query7	0.04	0.03	0.02
query8	0.07	0.05	0.05
query9	0.67	0.58	0.58
query10	0.63	0.63	0.64
query11	0.18	0.13	0.13
query12	0.21	0.14	0.13
query13	0.64	0.61	0.62
query14	1.04	1.04	1.03
query15	0.92	0.90	0.90
query16	0.42	0.42	0.45
query17	1.12	1.22	1.17
query18	0.24	0.22	0.23
query19	2.04	1.89	1.89
query20	0.01	0.01	0.01
query21	15.40	0.21	0.15
query22	5.07	0.08	0.06
query23	15.62	0.32	0.12
query24	2.59	0.74	0.76
query25	0.08	0.07	0.07
query26	0.17	0.16	0.15
query27	0.07	0.07	0.07
query28	5.14	1.20	0.96
query29	12.68	4.78	3.80
query30	0.30	0.16	0.12
query31	2.84	0.69	0.42
query32	3.25	0.59	0.48
query33	3.20	3.11	3.19
query34	15.95	5.30	4.64
query35	4.63	4.63	4.64
query36	0.70	0.55	0.53
query37	0.11	0.07	0.07
query38	0.08	0.05	0.05
query39	0.04	0.03	0.03
query40	0.19	0.16	0.15
query41	0.10	0.04	0.04
query42	0.04	0.03	0.03
query43	0.05	0.05	0.04
Total cold run time: 100.1 s
Total hot run time: 29.53 s

hello-stephen · 2025-10-30T09:34:44Z

BE UT Coverage Report

Increment line coverage 74.34% (113/152) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.71% (18037/34219)
Line Coverage	37.97% (163733/431251)
Region Coverage	32.33% (124814/386004)
Branch Coverage	33.71% (54600/161946)

hello-stephen · 2025-10-30T14:51:13Z

BE Regression && UT Coverage Report

Increment line coverage 78.29% (119/152) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.46% (24028/33626)
Line Coverage	57.87% (249920/431829)
Region Coverage	52.82% (206773/391438)
Branch Coverage	54.63% (89053/163017)

dataroaring

LGTM

github-actions · 2025-11-01T01:54:45Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-01T01:54:47Z

PR approved by anyone and no changes requested.

…57220 #57327 #59075 (#57441) ### What problem does this PR solve? pick: - #56815 - #56846 - #57220 - #57327 - #59075 Clickbench result on alicloud c6.4xlarge machine, Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz ||(baseline) LZ4F, 64KB data page size, 48 bucket, binary_plain_encoding_default_impl=v1, integer_type_default_use_plain_encoding =false| zstd(3), 1MB data page size, 48 bucket, binary_plain_encoding_default_impl=v1, integer_type_default_use_plain_encoding =false | zstd(3), 1MB data page size, 48 bucket, binary_plain_encoding_default_impl=v2, integer_type_default_use_plain_encoding =true | |---|---|---|---| |data size|16.087 GB| 11.016 GB | 8.56 GB | |load time|592s| 555s | 539s | |cold run total| 214.22 s | 177.57 s | 131.4 s | |hot run total| 36.03 s | 39.82 s | 39.05 s | ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

bobhan1 changed the title ~~[Opt](storage) Add BinaryPlainPageV2PreDecoder~~ [Opt](storage) Add pre decoder for BinaryPlainPageV2 and BinaryDictPage Oct 27, 2025

bobhan1 force-pushed the plain-binary-v2-pre-decode branch from 159bbac to acf4d00 Compare October 27, 2025 07:32

bobhan1 marked this pull request as ready for review October 27, 2025 07:34

bobhan1 force-pushed the plain-binary-v2-pre-decode branch 2 times, most recently from 0c7b0e6 to 699dbb5 Compare October 27, 2025 08:31

bobhan1 force-pushed the plain-binary-v2-pre-decode branch 2 times, most recently from 9dd92c6 to e881118 Compare October 27, 2025 11:59

bobhan1 force-pushed the plain-binary-v2-pre-decode branch from e881118 to 72d5a71 Compare October 28, 2025 02:26

bobhan1 mentioned this pull request Oct 29, 2025

branch-4.0: [Opt](storage) Optimize storage occupation #56815 #56846 #57220 #57327 #59075 #57441

Merged

16 tasks

bobhan1 added 5 commits October 30, 2025 14:14

add BinaryPlainPageV2PreDecoder by claude

5de528b

fix encoding info

06f9c7b

format

0ca55bd

fix

a04c613

update

f53f783

bobhan1 added 11 commits October 30, 2025 14:14

use v2 by defaut

5b0f6f0

update

714271b

fix BinaryPlainPageV2Test

533d788

tmp fix binary dict ut

c294941

add binary dict page pre decoder and fix ut

c9592cf

add ut for EncodingInfo pre decoder for all types x encodings

edf4eab

format

769d592

reduce reduntant memcpy ini pre decoder

f857417

fix FileColumnIterator::_read_dict_data

575cd4c

use plain binary v1 by default and fuzzy storage format

3a81264

fix beut

f292d0c

bobhan1 force-pushed the plain-binary-v2-pre-decode branch from 8cd8e08 to f292d0c Compare October 30, 2025 06:14

dataroaring approved these changes Nov 1, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2025

github-actions Bot added the reviewed label Nov 1, 2025

gavinchou approved these changes Nov 1, 2025

View reviewed changes

dataroaring merged commit f11570b into apache:master Nov 3, 2025
12 checks passed

yiguolei added the dev/4.0.3-merged label Dec 19, 2025

Conversation

bobhan1 commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 25, 2025

Uh oh!

bobhan1 commented Oct 25, 2025

Uh oh!

bobhan1 commented Oct 25, 2025

Uh oh!

bobhan1 commented Oct 26, 2025

Uh oh!

doris-robot commented Oct 26, 2025

Uh oh!

bobhan1 commented Oct 27, 2025

Uh oh!

doris-robot commented Oct 27, 2025

Uh oh!

bobhan1 commented Oct 27, 2025

Uh oh!

doris-robot commented Oct 27, 2025

Uh oh!

bobhan1 commented Oct 28, 2025

Uh oh!

doris-robot commented Oct 28, 2025

Uh oh!

hello-stephen commented Oct 28, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 28, 2025

BE Regression && UT Coverage Report

Uh oh!

bobhan1 commented Oct 29, 2025

Uh oh!

doris-robot commented Oct 29, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 29, 2025

BE Regression && UT Coverage Report

Uh oh!

bobhan1 commented Oct 30, 2025

Uh oh!

doris-robot commented Oct 30, 2025

Uh oh!

hello-stephen commented Oct 30, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 30, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 1, 2025

Uh oh!

github-actions Bot commented Nov 1, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bobhan1 commented Oct 25, 2025 •

edited

Loading