Skip to content

[enhancement](filecache) use rocksdb to persist cache block meta - #57072

Merged
hello-stephen merged 25 commits into
apache:masterfrom
freemandealer:ttl-enhancement
Nov 5, 2025
Merged

[enhancement](filecache) use rocksdb to persist cache block meta#57072
hello-stephen merged 25 commits into
apache:masterfrom
freemandealer:ttl-enhancement

Conversation

@freemandealer

@freemandealer freemandealer commented Oct 16, 2025

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer freemandealer changed the title Ttl enhancement [WIP][enhancement](filecache) use rocksdb to persist cache block meta Oct 17, 2025
@freemandealer freemandealer changed the title [WIP][enhancement](filecache) use rocksdb to persist cache block meta [enhancement](filecache) use rocksdb to persist cache block meta Oct 21, 2025
@freemandealer
freemandealer marked this pull request as ready for review October 21, 2025 02:03
dataroaring
dataroaring reviewed Oct 23, 2025
View reviewed changes
Comment thread be/src/io/cache/fs_file_cache_storage.cpp Resolved
gavinchou
gavinchou reviewed Oct 23, 2025
View reviewed changes
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
gavinchou
gavinchou reviewed Oct 23, 2025
View reviewed changes
Comment thread be/src/io/cache/cached_remote_file_reader.cpp Outdated
return std::make_pair(align_left, align_size);
}

std::optional<int64_t> CachedRemoteFileReader::_get_tablet_id() {

@gavinchou gavinchou Oct 23, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

内外表的文件路径怎么区分? 是调用这个函数之前判断的?

gavinchou
gavinchou reviewed Oct 28, 2025
View reviewed changes
Comment thread be/src/io/cache/cache_block_meta_store.h Resolved
gavinchou
gavinchou reviewed Oct 28, 2025
View reviewed changes
Comment thread be/src/io/cache/cache_block_meta_store.h Resolved
gavinchou
gavinchou reviewed Oct 28, 2025
View reviewed changes
Comment thread be/src/io/cache/cache_block_meta_store.h Resolved
gavinchou
gavinchou reviewed Oct 28, 2025
View reviewed changes
Comment thread be/src/io/cache/cache_block_meta_store.h Resolved
gavinchou
gavinchou reviewed Oct 28, 2025
View reviewed changes
Comment thread be/src/io/cache/cache_block_meta_store.h Resolved
Comment thread be/src/io/cache/cache_block_meta_store.h Resolved
Comment thread be/src/io/cache/cache_block_meta_store.h Resolved
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
gavinchou
gavinchou reviewed Oct 28, 2025
View reviewed changes
Comment thread be/src/io/cache/cache_block_meta_store.cpp Resolved
@freemandealer

freemandealer commented Oct 29, 2025

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

doris-robot commented Oct 29, 2025

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.66% (1656/2053)
Line Coverage 66.97% (29215/43625)
Region Coverage 67.38% (14500/21521)
Branch Coverage 57.65% (7705/13364)

@doris-robot

doris-robot commented Oct 29, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dabd384cbbbde6703aedcf21afbb54cd946766a3, data reload: false

query1	0.06	0.05	0.06
query2	0.11	0.05	0.06
query3	0.25	0.09	0.09
query4	1.61	0.12	0.12
query5	0.28	0.27	0.26
query6	1.19	0.66	0.67
query7	0.03	0.03	0.03
query8	0.06	0.06	0.05
query9	0.61	0.57	0.52
query10	0.59	0.59	0.59
query11	0.18	0.12	0.12
query12	0.16	0.13	0.14
query13	0.63	0.62	0.60
query14	1.02	1.04	1.02
query15	0.86	0.86	0.85
query16	0.43	0.42	0.41
query17	1.05	1.08	1.06
query18	0.23	0.23	0.20
query19	1.92	1.86	1.94
query20	0.02	0.02	0.01
query21	15.41	0.19	0.14
query22	5.09	0.07	0.05
query23	15.65	0.27	0.10
query24	3.03	0.61	1.01
query25	0.07	0.06	0.06
query26	0.15	0.14	0.14
query27	0.07	0.06	0.06
query28	5.42	1.17	0.95
query29	12.57	4.02	3.29
query30	0.28	0.14	0.12
query31	2.83	0.62	0.40
query32	3.24	0.57	0.48
query33	3.14	3.06	3.13
query34	15.74	5.16	4.60
query35	4.59	4.59	4.64
query36	0.71	0.53	0.51
query37	0.10	0.06	0.07
query38	0.08	0.05	0.05
query39	0.04	0.04	0.03
query40	0.18	0.16	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 99.87 s
Total hot run time: 28.24 s

@freemandealer

freemandealer commented Oct 30, 2025

Copy link
Copy Markdown
Member Author

run buildall

@freemandealer

freemandealer commented Oct 30, 2025

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

doris-robot commented Oct 30, 2025

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.63% (1673/2075)
Line Coverage 67.00% (29513/44052)
Region Coverage 67.46% (14656/21725)
Branch Coverage 57.74% (7793/13496)

@freemandealer

freemandealer commented Oct 30, 2025

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

doris-robot commented Oct 30, 2025

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.63% (1673/2075)
Line Coverage 66.94% (29488/44052)
Region Coverage 67.41% (14644/21725)
Branch Coverage 57.72% (7790/13496)

@doris-robot

doris-robot commented Oct 30, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 494bfa9b1f0958a5fe658cb88529ba7bce747037, data reload: false

query1	0.05	0.04	0.05
query2	0.12	0.07	0.07
query3	0.31	0.07	0.07
query4	1.60	0.10	0.09
query5	0.27	0.26	0.26
query6	1.17	0.67	0.65
query7	0.03	0.02	0.03
query8	0.08	0.06	0.06
query9	0.66	0.53	0.54
query10	0.60	0.60	0.58
query11	0.27	0.14	0.13
query12	0.26	0.14	0.14
query13	0.64	0.64	0.62
query14	1.03	1.04	1.04
query15	0.96	0.88	0.86
query16	0.40	0.39	0.38
query17	1.07	1.06	1.06
query18	0.24	0.22	0.23
query19	1.97	1.77	1.90
query20	0.02	0.02	0.02
query21	15.41	0.29	0.24
query22	5.01	0.10	0.09
query23	15.37	0.39	0.23
query24	2.96	0.48	0.30
query25	0.09	0.08	0.08
query26	0.19	0.18	0.18
query27	0.09	0.09	0.10
query28	3.63	1.27	1.08
query29	12.57	4.06	3.37
query30	0.34	0.12	0.10
query31	2.83	0.65	0.44
query32	3.25	0.61	0.50
query33	3.08	3.11	3.08
query34	16.52	5.24	4.51
query35	4.55	4.53	4.58
query36	0.65	0.53	0.52
query37	0.22	0.09	0.09
query38	0.19	0.06	0.07
query39	0.06	0.05	0.06
query40	0.21	0.18	0.18
query41	0.11	0.07	0.06
query42	0.07	0.04	0.04
query43	0.07	0.05	0.06
Total cold run time: 99.22 s
Total hot run time: 28.48 s

@doris-robot

doris-robot commented Oct 30, 2025

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 71.75% (518/722) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.72% (18067/34271)
Line Coverage 37.92% (163716/431790)
Region Coverage 32.26% (124695/386575)
Branch Coverage 33.62% (54486/162043)

@hello-stephen

hello-stephen commented Oct 30, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 80.62% (574/712) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.43% (24055/33675)
Line Coverage 57.85% (250132/432359)
Region Coverage 52.93% (207498/391996)
Branch Coverage 54.67% (89161/163102)

@freemandealer

freemandealer commented Nov 5, 2025

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

hello-stephen commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.71% (1674/2074)
Line Coverage 67.05% (29554/44080)
Region Coverage 67.59% (14703/21752)
Branch Coverage 57.82% (7817/13520)

@freemandealer

freemandealer commented Nov 5, 2025

Copy link
Copy Markdown
Member Author

run check_coverage

@doris-robot

doris-robot commented Nov 5, 2025

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 67.66% (569/841) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.74% (18114/34347)
Line Coverage 38.01% (164571/432955)
Region Coverage 32.30% (125288/387940)
Branch Coverage 33.73% (54827/162544)

@doris-robot

doris-robot commented Nov 5, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 189519 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24784dc9066a9ec213b7f7e4e4a326cbc72ca50d, data reload: false

query1	1069	417	396	396
query2	6567	1698	1695	1695
query3	6746	240	220	220
query4	26990	23488	23052	23052
query5	4391	645	499	499
query6	336	230	227	227
query7	4654	494	301	301
query8	310	245	249	245
query9	8705	2580	2597	2580
query10	506	342	307	307
query11	15280	15054	14818	14818
query12	179	121	123	121
query13	1694	579	452	452
query14	11137	9203	9206	9203
query15	205	185	171	171
query16	7685	691	514	514
query17	1476	751	632	632
query18	2028	415	315	315
query19	211	192	170	170
query20	140	150	135	135
query21	239	140	119	119
query22	4651	4732	4653	4653
query23	34763	33814	34105	33814
query24	8978	2591	2508	2508
query25	594	586	482	482
query26	1352	281	170	170
query27	3188	547	355	355
query28	4342	2304	2216	2216
query29	818	643	498	498
query30	299	232	202	202
query31	961	880	775	775
query32	87	85	68	68
query33	606	401	359	359
query34	863	899	545	545
query35	845	858	776	776
query36	1005	1035	941	941
query37	138	123	88	88
query38	3654	3585	3431	3431
query39	1509	1418	1415	1415
query40	227	127	114	114
query41	59	58	56	56
query42	121	112	111	111
query43	481	480	468	468
query44	1200	749	720	720
query45	182	174	176	174
query46	883	989	629	629
query47	1756	1849	1739	1739
query48	399	417	328	328
query49	780	513	414	414
query50	633	699	400	400
query51	3883	3980	3989	3980
query52	110	104	103	103
query53	232	260	196	196
query54	302	284	274	274
query55	84	91	82	82
query56	312	323	303	303
query57	1175	1212	1109	1109
query58	289	274	282	274
query59	2613	2719	2593	2593
query60	367	358	352	352
query61	198	187	190	187
query62	804	744	710	710
query63	229	198	193	193
query64	4539	1279	995	995
query65	4010	3937	3947	3937
query66	1103	468	352	352
query67	15598	15427	14890	14890
query68	8387	964	601	601
query69	480	335	289	289
query70	1322	1201	1279	1201
query71	510	349	313	313
query72	5929	5004	5025	5004
query73	678	603	361	361
query74	8853	9062	8676	8676
query75	4023	3374	2788	2788
query76	3795	1175	719	719
query77	802	417	312	312
query78	9667	9751	9016	9016
query79	2478	811	595	595
query80	638	559	506	506
query81	538	260	228	228
query82	435	159	138	138
query83	265	263	252	252
query84	261	120	91	91
query85	926	491	448	448
query86	388	301	305	301
query87	3647	3651	3625	3625
query88	3752	2259	2224	2224
query89	379	320	296	296
query90	1918	219	221	219
query91	169	173	137	137
query92	87	67	66	66
query93	1956	983	644	644
query94	697	447	341	341
query95	409	324	303	303
query96	484	592	287	287
query97	2972	2977	2865	2865
query98	245	219	210	210
query99	1654	1406	1312	1312
Total cold run time: 280514 ms
Total hot run time: 189519 ms

@doris-robot

doris-robot commented Nov 5, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24784dc9066a9ec213b7f7e4e4a326cbc72ca50d, data reload: false

query1	0.05	0.05	0.05
query2	0.12	0.06	0.06
query3	0.32	0.08	0.08
query4	1.60	0.09	0.09
query5	0.27	0.25	0.25
query6	1.18	0.66	0.64
query7	0.03	0.02	0.03
query8	0.07	0.06	0.06
query9	0.65	0.53	0.53
query10	0.60	0.59	0.59
query11	0.25	0.14	0.13
query12	0.26	0.14	0.14
query13	0.63	0.64	0.61
query14	1.02	1.03	1.05
query15	0.96	0.87	0.86
query16	0.40	0.38	0.39
query17	1.09	1.06	1.05
query18	0.25	0.23	0.23
query19	1.97	1.82	1.83
query20	0.02	0.01	0.01
query21	15.40	0.29	0.25
query22	4.97	0.11	0.09
query23	15.36	0.38	0.24
query24	3.00	0.46	0.30
query25	0.10	0.08	0.09
query26	0.19	0.18	0.18
query27	0.09	0.10	0.10
query28	3.60	1.27	1.07
query29	12.56	3.95	3.31
query30	0.33	0.12	0.11
query31	2.82	0.62	0.42
query32	3.23	0.63	0.52
query33	3.03	3.18	3.05
query34	16.96	5.18	4.44
query35	4.57	4.51	4.55
query36	0.65	0.52	0.51
query37	0.23	0.09	0.09
query38	0.20	0.06	0.05
query39	0.06	0.05	0.05
query40	0.20	0.17	0.17
query41	0.11	0.06	0.06
query42	0.07	0.04	0.05
query43	0.06	0.05	0.06
Total cold run time: 99.53 s
Total hot run time: 28.32 s

@hello-stephen

hello-stephen commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 75.21% (625/831) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.52% (24147/33763)
Line Coverage 57.92% (251210/433691)
Region Coverage 52.88% (208091/393523)
Branch Coverage 54.73% (89556/163641)

@freemandealer

freemandealer commented Nov 5, 2025

Copy link
Copy Markdown
Member Author

run feut

@github-actions

github-actions Bot commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 5, 2025
@github-actions

github-actions Bot commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
hello-stephen merged commit 30406d6 into apache:master Nov 5, 2025
12 checks passed
freemandealer added a commit to freemandealer/doris that referenced this pull request Nov 14, 2025
@freemandealer
[enhancement](filecache) use rocksdb to persist cache block meta (apa…
7ba218d 
…che#57072)
freemandealer added a commit to freemandealer/doris that referenced this pull request Nov 17, 2025
@freemandealer
[enhancement](filecache) use rocksdb to persist cache block meta (apa…
38b7669 
…che#57072)
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 18, 2025
@freemandealer @wyxxxcat
[enhancement](filecache) use rocksdb to persist cache block meta (apa…
741419c 
…che#57072)
@gavinchou

gavinchou commented Nov 25, 2025

Copy link
Copy Markdown
Contributor

run coverage

@gavinchou

gavinchou commented Nov 25, 2025

Copy link
Copy Markdown
Contributor

run check_coverage

freemandealer added a commit to freemandealer/doris that referenced this pull request Dec 9, 2025
@freemandealer
[enhancement](filecache) use rocksdb to persist cache block meta (apa…
97f9162 
…che#57072)
freemandealer added a commit to freemandealer/doris that referenced this pull request Dec 24, 2025
@freemandealer
[enhancement](filecache) use rocksdb to persist cache block meta (apa…
401993c 
…che#57072)
@freemandealer freemandealer mentioned this pull request Mar 20, 2026
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Mar 21, 2026
@freemandealer
branch-4.1:[enhancement](filecache) filecache meta persist PRs compil…
73bbd4d 
…ation (#61581)

pick the following PRs/Commits:

  - #57072 -> 30406d6
  - #57783 -> 4cbae8a
  - #57922 -> 0918952
  - #58082 -> 9f9a3e4
  - #59259 -> 9f2c6ed
  - #59269 -> e010d8d
  - #59311 -> d336eed
  - #59314 -> f7aa74e
  - #59645 -> c4c3d14
  - #60252 -> b1585e1
  - #60269 -> 26f44b5

 - #59645 -> c4c3d14
 * #60252 -> b1585e1
 * #60269 -> 26f44b5
 * #59269 -> e010d8d

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Signed-off-by: zhengyu <[email protected]>
Signed-off-by: freemandealer <[email protected]>
@deardeng deardeng mentioned this pull request Oct 15, 2026
Merged
3 of 16 tasks
deardeng added a commit that referenced this pull request Oct 16, 2026
@deardeng
[fix](be) Restore file-level cache metadata loading during startup (#…
e9b3373 
…69116)

Related PR: #57072, #65658

Problem Summary: During BE startup, get_or_set, the fully-covered
lookup, and probe attempt to restore an absent file before background
metadata loading completes. Their FileCacheKey leaves offset
uninitialized, while the storage implementation now performs an exact
RocksDB lookup for one block. A file with multiple persisted blocks can
therefore miss existing cache data or restore only one block. Query
metadata by tablet and file hash prefix, restore all blocks, and assert
that the file hash is absent when entering synchronous restoration. Keep
caller trigger conditions, key encoding, and existing tablet-range
queries unchanged.

Fix reuse of persistent file cache blocks while BE startup metadata
loading is in progress.
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Nov 6, 2026
@deardeng @bobhan1
[fix](be) Restore file-level cache metadata loading during startup (a…
986ddeb 
…pache#69116)

Related PR: apache#57072, apache#65658

Problem Summary: During BE startup, get_or_set, the fully-covered
lookup, and probe attempt to restore an absent file before background
metadata loading completes. Their FileCacheKey leaves offset
uninitialized, while the storage implementation now performs an exact
RocksDB lookup for one block. A file with multiple persisted blocks can
therefore miss existing cache data or restore only one block. Query
metadata by tablet and file hash prefix, restore all blocks, and assert
that the file hash is absent when entering synchronous restoration. Keep
caller trigger conditions, key encoding, and existing tablet-range
queries unchanged.

Fix reuse of persistent file cache blocks while BE startup metadata
loading is in progress.

(cherry picked from commit e9b3373)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring left review comments
@gavinchou gavinchou gavinchou approved these changes
@hello-stephen hello-stephen hello-stephen approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@freemandealer @Thearas @doris-robot @hello-stephen @gavinchou @dataroaring