Skip to content

branch-4.0: [Opt](storage) Optimize storage occupation #56815 #56846 #57220 #57327 #59075 - #57441

Merged
yiguolei merged 8 commits into
apache:branch-4.0from
bobhan1:branch-4.0-pick-all-storage-opt
Dec 19, 2025
Merged

yiguolei merged 8 commits into
apache:branch-4.0from
bobhan1:branch-4.0-pick-all-storage-opt

Conversation

@bobhan1

@bobhan1 bobhan1 commented Oct 29, 2025 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

pick:

Clickbench result on alicloud c6.4xlarge machine, Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz

(baseline) LZ4F, 64KB data page size, 48 bucket, binary_plain_encoding_default_impl=v1, integer_type_default_use_plain_encoding =false zstd(3), 1MB data page size, 48 bucket, binary_plain_encoding_default_impl=v1, integer_type_default_use_plain_encoding =false zstd(3), 1MB data page size, 48 bucket, binary_plain_encoding_default_impl=v2, integer_type_default_use_plain_encoding =true
data size 16.087 GB 11.016 GB 8.56 GB
load time 592s 555s 539s
cold run total 214.22 s 177.57 s 131.4 s
hot run total 36.03 s 39.82 s 39.05 s

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
bobhan1 marked this pull request as ready for review October 29, 2025 05:06
@bobhan1
bobhan1 requested a review from yiguolei as a code owner October 29, 2025 05:06
@bobhan1

bobhan1 commented Oct 29, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@bobhan1
bobhan1 force-pushed the branch-4.0-pick-all-storage-opt branch from 2e37f12 to 95bd78b Compare October 30, 2025 06:17
@bobhan1

bobhan1 commented Oct 30, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring previously approved these changes Nov 1, 2025

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2025
@github-actions

github-actions Bot commented Nov 1, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Nov 1, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@gavinchou gavinchou changed the title branch-4.0: [Opt](storage) Optimize storage occupation branch-4.0: [Opt](storage) Optimize storage occupation #56815 #56846 #57220 #57327 Nov 5, 2025
Add a system table `column_data_sizes` for querying columns' data page
sizes:
```sql
MySQL [email protected]:information_schema> show create table information_schema.column_data_sizes;
+-------------------+------------------------------------------+
| Table             | Create Table                             |
+-------------------+------------------------------------------+
| column_data_sizes | CREATE TABLE `column_data_sizes` (       |
|                   |   `BACKEND_ID` bigint NULL,              |
|                   |   `TABLE_ID` bigint NULL,                |
|                   |   `INDEX_ID` bigint NULL,                |
|                   |   `PARTITION_ID` bigint NULL,            |
|                   |   `TABLET_ID` bigint NULL,               |
|                   |   `ROWSET_ID` varchar(64) NULL,          |
|                   |   `COLUMN_UNIQUE_ID` int NULL,           |
|                   |   `COLUMN_NAME` varchar(64) NULL,        |
|                   |   `COLUMN_TYPE` varchar(64) NULL,        |
|                   |   `COMPRESSED_DATA_BYTES` bigint NULL,   |
|                   |   `UNCOMPRESSED_DATA_BYTES` bigint NULL, |
|                   |   `RAW_DATA_BYTES` bigint NULL           |
|                   | ) ENGINE=SCHEMA;                         |
+-------------------+------------------------------------------+
```
- **COMPRESSED_DATA_BYTES**: total size of compressed data pages in
bytes, for complex type(array/map/struct), it's the sum of all leaf
scalar column writers' stats
- **UNCOMPRESSED_DATA_BYTES**: total size of uncompressed data pages in
bytes, for complex type(array/map/struct), it's the sum of all leaf
scalar column writers' stats
- **RAW_DATA_BYTES**: total size of input raw data in bytes(calculated
via `XXXPageBuilder`'s `Status add(const uint8_t* vals, size_t* count)`
), for complex type(array/map/struct), it's the sum of all leaf scalar
column writers' stats

To query columns' data page sizes in a table:
```sql
SELECT
    COLUMN_NAME,
    sum(COMPRESSED_DATA_BYTES) AS compressed_data_bytes,
    sum(UNCOMPRESSED_DATA_BYTES) AS uncompressed_data_bytes,
    sum(RAW_DATA_BYTES) as raw_data_bytes
FROM information_schema.column_data_sizes
WHERE table_id=${TABLE_ID}
GROUP BY COLUMN_NAME, COLUMN_TYPE
ORDER BY data_size desc;
```

Note:
- variant column is not supported.
- In cloud mode, only BE local's rowsets will be queried.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
1. Currently, plain binary are encoding as the following
```
|binary1|binary2|...|offset1(fixed uint32)|offset2(fixed uint32)|...
```
This PR add a new plain binary encoding which have better storage
occupation when binaries are small
```
|length1(varuint32)|binary1||length2(varuint32)|binary2|...
```
add a be conf `binary_plain_encoding_default_impl` to control.

2. move the registration of column types' encoding to `ExecEnv::_init`
fix encoding info

format

fix

update

use v2 by defaut

update

fix BinaryPlainPageV2Test

tmp fix binary dict ut

add binary dict page pre decoder and fix ut

add ut for EncodingInfo pre decoder for all types x encodings

format

reduce reduntant memcpy ini pre decoder

fix FileColumnIterator::_read_dict_data
@bobhan1
bobhan1 force-pushed the branch-4.0-pick-all-storage-opt branch from 95bd78b to 0fb99cc Compare December 17, 2025 08:55
@github-actions github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Dec 17, 2025
@bobhan1

bobhan1 commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.14% (1573/1915)
Line Coverage 67.14% (28062/41794)
Region Coverage 67.70% (13810/20399)
Branch Coverage 58.12% (7361/12666)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 55.68% (495/889) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.22% (18466/34697)
Line Coverage 38.86% (170110/437799)
Region Coverage 33.66% (131581/390859)
Branch Coverage 34.54% (56766/164372)

…ger and binary#59075

update

add regression

update related ut

move to tablet schema due to memtable move

remove old config

remove log

fix test

fix cloud
@bobhan1 bobhan1 changed the title branch-4.0: [Opt](storage) Optimize storage occupation #56815 #56846 #57220 #57327 branch-4.0: [Opt](storage) Optimize storage occupation #56815 #56846 #57220 #57327 #59075 Dec 17, 2025
@bobhan1

bobhan1 commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@bobhan1

bobhan1 commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.14% (1573/1915)
Line Coverage 67.17% (28075/41794)
Region Coverage 67.74% (13818/20399)
Branch Coverage 58.15% (7365/12666)

@doris-robot

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 53.92% (523/970) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.22% (18468/34701)
Line Coverage 38.90% (170316/437880)
Region Coverage 33.68% (131636/390884)
Branch Coverage 34.57% (56836/164392)

@bobhan1

bobhan1 commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.14% (1573/1915)
Line Coverage 67.19% (28082/41794)
Region Coverage 67.70% (13811/20399)
Branch Coverage 58.16% (7366/12666)

@doris-robot

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 53.92% (523/970) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.18% (18455/34701)
Line Coverage 38.86% (170182/437880)
Region Coverage 33.65% (131536/390884)
Branch Coverage 34.55% (56793/164392)

@bobhan1

bobhan1 commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

run p0

@bobhan1

bobhan1 commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@bobhan1

bobhan1 commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

run feut

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.14% (1573/1915)
Line Coverage 67.18% (28079/41794)
Region Coverage 67.71% (13813/20399)
Branch Coverage 58.14% (7364/12666)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 53.97% (523/969) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.15% (18455/34720)
Line Coverage 38.84% (170192/438173)
Region Coverage 33.59% (131384/391182)
Branch Coverage 34.52% (56790/164493)

@bobhan1

bobhan1 commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

run feut

@bobhan1

bobhan1 commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

run nonConcurrent

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 88.89% (16/18) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit 6560401 into apache:branch-4.0 Dec 19, 2025
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants