Repository navigation
branch-4.0: [Opt](storage) Optimize storage occupation #56815 #56846 #57220 #57327 #59075 - #57441
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
2e37f12 to
95bd78b
Compare
|
run buildall |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
Add a system table `column_data_sizes` for querying columns' data page sizes: ```sql MySQL [email protected]:information_schema> show create table information_schema.column_data_sizes; +-------------------+------------------------------------------+ | Table | Create Table | +-------------------+------------------------------------------+ | column_data_sizes | CREATE TABLE `column_data_sizes` ( | | | `BACKEND_ID` bigint NULL, | | | `TABLE_ID` bigint NULL, | | | `INDEX_ID` bigint NULL, | | | `PARTITION_ID` bigint NULL, | | | `TABLET_ID` bigint NULL, | | | `ROWSET_ID` varchar(64) NULL, | | | `COLUMN_UNIQUE_ID` int NULL, | | | `COLUMN_NAME` varchar(64) NULL, | | | `COLUMN_TYPE` varchar(64) NULL, | | | `COMPRESSED_DATA_BYTES` bigint NULL, | | | `UNCOMPRESSED_DATA_BYTES` bigint NULL, | | | `RAW_DATA_BYTES` bigint NULL | | | ) ENGINE=SCHEMA; | +-------------------+------------------------------------------+ ``` - **COMPRESSED_DATA_BYTES**: total size of compressed data pages in bytes, for complex type(array/map/struct), it's the sum of all leaf scalar column writers' stats - **UNCOMPRESSED_DATA_BYTES**: total size of uncompressed data pages in bytes, for complex type(array/map/struct), it's the sum of all leaf scalar column writers' stats - **RAW_DATA_BYTES**: total size of input raw data in bytes(calculated via `XXXPageBuilder`'s `Status add(const uint8_t* vals, size_t* count)` ), for complex type(array/map/struct), it's the sum of all leaf scalar column writers' stats To query columns' data page sizes in a table: ```sql SELECT COLUMN_NAME, sum(COMPRESSED_DATA_BYTES) AS compressed_data_bytes, sum(UNCOMPRESSED_DATA_BYTES) AS uncompressed_data_bytes, sum(RAW_DATA_BYTES) as raw_data_bytes FROM information_schema.column_data_sizes WHERE table_id=${TABLE_ID} GROUP BY COLUMN_NAME, COLUMN_TYPE ORDER BY data_size desc; ``` Note: - variant column is not supported. - In cloud mode, only BE local's rowsets will be queried. None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
1. Currently, plain binary are encoding as the following ``` |binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|... ``` This PR add a new plain binary encoding which have better storage occupation when binaries are small ``` |length1(varuint32)|binary1||length2(varuint32)|binary2|... ``` add a be conf `binary_plain_encoding_default_impl` to control. 2. move the registration of column types' encoding to `ExecEnv::_init`
fix encoding info format fix update use v2 by defaut update fix BinaryPlainPageV2Test tmp fix binary dict ut add binary dict page pre decoder and fix ut add ut for EncodingInfo pre decoder for all types x encodings format reduce reduntant memcpy ini pre decoder fix FileColumnIterator::_read_dict_data
95bd78b to
0fb99cc
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
…ger and binary#59075 update add regression update related ut move to tablet schema due to memtable move remove old config remove log fix test fix cloud
|
run buildall |
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run p0 |
|
run buildall |
|
run feut |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run feut |
|
run nonConcurrent |
FE UT Coverage ReportIncrement line coverage |
What problem does this PR solve?
pick:
BinaryPlainPageV2#56846storage_formatcontrol default encoding for interger and binary #59075Clickbench result on alicloud c6.4xlarge machine, Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)