Repository navigation
[Opt](encoding) Add new encoding BinaryPlainPageV2 - #56846
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
24a0434 to
70f064f
Compare
70f064f to
329c4de
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage `` 🎉 |
TPC-DS: Total hot run time: 190704 ms |
ClickBench: Total hot run time: 30.52 s |
a74cc37 to
d1fe840
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage `` 🎉 |
TPC-DS: Total hot run time: 190075 ms |
ClickBench: Total hot run time: 31.23 s |
92e32d6 to
2409b58
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
7503128 to
50d3f61
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
2fb272f to
9dc5ca1
Compare
5436de4 to
b782827
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage `` 🎉 |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
1. Currently, plain binary are encoding as the following ``` |binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|... ``` This PR add a new plain binary encoding which have better storage occupation when binaries are small ``` |length1(varuint32)|binary1||length2(varuint32)|binary2|... ``` add a be conf `binary_plain_encoding_default_impl` to control. 2. move the registration of column types' encoding to `ExecEnv::_init`
1. Currently, plain binary are encoding as the following ``` |binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|... ``` This PR add a new plain binary encoding which have better storage occupation when binaries are small ``` |length1(varuint32)|binary1||length2(varuint32)|binary2|... ``` add a be conf `binary_plain_encoding_default_impl` to control. 2. move the registration of column types' encoding to `ExecEnv::_init`
1. Currently, plain binary are encoding as the following ``` |binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|... ``` This PR add a new plain binary encoding which have better storage occupation when binaries are small ``` |length1(varuint32)|binary1||length2(varuint32)|binary2|... ``` add a be conf `binary_plain_encoding_default_impl` to control. 2. move the registration of column types' encoding to `ExecEnv::_init`
1. Currently, plain binary are encoding as the following ``` |binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|... ``` This PR add a new plain binary encoding which have better storage occupation when binaries are small ``` |length1(varuint32)|binary1||length2(varuint32)|binary2|... ``` add a be conf `binary_plain_encoding_default_impl` to control. 2. move the registration of column types' encoding to `ExecEnv::_init`
…57220 #57327 #59075 (#57441) ### What problem does this PR solve? pick: - #56815 - #56846 - #57220 - #57327 - #59075 Clickbench result on alicloud c6.4xlarge machine, Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz ||(baseline) LZ4F, 64KB data page size, 48 bucket, binary_plain_encoding_default_impl=v1, integer_type_default_use_plain_encoding =false| zstd(3), 1MB data page size, 48 bucket, binary_plain_encoding_default_impl=v1, integer_type_default_use_plain_encoding =false | zstd(3), 1MB data page size, 48 bucket, binary_plain_encoding_default_impl=v2, integer_type_default_use_plain_encoding =true | |---|---|---|---| |data size|16.087 GB| 11.016 GB | 8.56 GB | |load time|592s| 555s | 539s | |cold run total| 214.22 s | 177.57 s | 131.4 s | |hot run total| 36.03 s | 39.82 s | 39.05 s | ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
What problem does this PR solve?
This PR add a new plain binary encoding which have better storage occupation when binaries are small
add a be conf
binary_plain_encoding_default_implto control.ExecEnv::_initRelease note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)