Parquet footer cache by grantholly-clickhouse · Pull Request #89750 · ClickHouse/ClickHouse

grantholly-clickhouse · 2025-11-07T23:16:01Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Added a new SLRU cache for Parquet metadata to improve reading performance by removing the need to re-download files just to read metadata. The cache can be dropped with SYSTEM DROP PARQUET METADATA CACHE. Resolves #89102.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Details

This adds a new SLRU cache for parquet metadata (#89102). This aims to improve parquet reading performance by removing the need to re-download a file just to read the metadata. I tried to closely follow the pattern of the Iceberg file metadata cache as was suggested in the related issue.

All the configuration for both caches falls under parquet_metadata_cache_*.

…che on and off

…query context

…ress the v3 reader

…and factoring out extractFilePathAndETagFromReadBuffer

…ng separate options for v3 caches

src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp

src/Processors/Formats/Impl/Parquet/ReadManager.cpp

grantholly-clickhouse · 2026-02-05T18:50:03Z

src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp

            if (metadata_cache && metadata.has_value())
            {
-                String file_name = metadata->getFileName();
+                String file_name = metadata->getPath();


What is the important difference here?

Full path on a blob_storage opposed to a last file part (after the last slash) in a file path. We can have potential problems with files which have one file name but lay in different folders

…se/ClickHouse into parquet_footer_cache

divanik · 2026-02-06T12:36:33Z

src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp

            std::lock_guard lock(reader_mutex);
            reader.emplace();
            reader->reader.prefetcher.init(in, read_options, parser_shared_resources);
+            reader->reader.file_metadata = getFileMetadata(reader->reader.prefetcher);


I wonder if this line of code was added (I refactored the block of code which was done before, so I don't know).

That line is correct. This is where we need to look up the file metadata.

divanik · 2026-02-09T11:44:13Z

The PR seems ok to merge, but there is some strange stuff with CI, creating new commit and restarting CI not to spend time on dealing with CI infrastructure

…t_footer_cache

divanik · 2026-02-09T12:29:16Z

Sorry, I merged dirty master to the PR, already fixed the problem

divanik

Small question if the change is related

src/Access/Common/AccessType.h

…footer_cache" This reverts commit 5243ef7, reversing changes made to 6e3a5ad.

Revert "Merge pull request #89750 - parquet_footer_cache"

…se/parquet_footer_cache" This reverts commit 223e53c.

divanik · 2026-02-13T14:01:46Z

The problem is clear: firstly we took info about use_parquet_v3_reader from the context, after that we took it from the format_settings. For some reason we have different values in two different sources (that's obviously mess, but out of the scope of this PR). @grantholly-clickhouse, can you try to reproduce the problem, adding a test, fixing and opening the new PR (I suggest not dealing with rerevert, it can be dangerous)

grantholly-clickhouse added 30 commits November 3, 2025 15:23

adding new default settings values

956714a

adding serve settings

f5ff646

adding settings to change history and declaring bool value to turn ca…

a961a37

…che on and off

first draft of caching class

338f3f5

adding basic instrumentation around cache hits and misses

417c3e4

using a new hash function for hashing cache keys

25b8e7e

adding new parquet metadata cache in context header

0f90c17

adding implementation for setting up the parquet cache in the shared …

40f6f3d

…query context

only hashing once with the concat of file name and the etag field

454447c

beginning to integrate the metadata cache into the parquet block input

08a8312

fixing settings check for cache usage by bringing in Setting

5e27811

wiring the v3 parquet reader to the cache and updating its constructor

f9ea34d

first attempt at wiring up cache to parquet reader, still need to add…

6bcb0f7

…ress the v3 reader

adding a separate cache for the V3 parquet reader and integrating it …

2a0203b

…and factoring out extractFilePathAndETagFromReadBuffer

separating v2 and v3 caches

965b54a

adding cache to Arrow schema reader

4d33bff

initializing schema readers with caches correctly

27f0eae

using both v2 and v3 caches in parquet block input format constructors

6ee876d

adding 'drop cache' statements support mirroring iceberg implementation

382228f

fixing typo in call to cache clearing methods

f6d4ebe

forgot to add permissions check for drop cache

c2035ea

adding server settings and docs for drop cache statement

1a68168

collapsing drop cache statements into a single one

4b8e356

removing v3 drop cache statement from docs

6aa6792

forgot to remove the v3 drop cache command from ASTSsystemQuery

1cf7ff5

fixing copy paste error

4f007ae

moving all settings under parquet_metadata_cache_* and no longer havi…

bc81218

…ng separate options for v3 caches

more cleanup

d334c8b

more cleanup

396e53a

more cleanup removing v3 settings

12d746a

divanik added 4 commits February 5, 2026 18:04

Address review comments

e9b4f4b

Remove unrelated diff

160db7e

Remove unrelated diff

c30469a

Fix style

8dc2a74

divanik reviewed Feb 5, 2026

View reviewed changes

src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp Outdated Show resolved Hide resolved

src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp Outdated Show resolved Hide resolved

src/Processors/Formats/Impl/Parquet/ReadManager.cpp Show resolved Hide resolved

grantholly-clickhouse commented Feb 5, 2026

View reviewed changes

grantholly-clickhouse and others added 3 commits February 5, 2026 23:03

more style fixes

988d5c6

Not launch the test on parallel replicas

2027b78

Merge branch 'parquet_footer_cache' of github.com:grantholly-clickhou…

0f87eb8

…se/ClickHouse into parquet_footer_cache

divanik reviewed Feb 6, 2026

View reviewed changes

grantholly-clickhouse added 2 commits February 6, 2026 10:22

changing logs to LOG_TRACE

aedb2c3

merging in upstream master

d4179cb

divanik force-pushed the parquet_footer_cache branch from b4886cc to d4179cb Compare February 9, 2026 12:28

Merge branch 'master' of github.com:ClickHouse/ClickHouse into parque…

5c33f11

…t_footer_cache

divanik approved these changes Feb 9, 2026

View reviewed changes

divanik reviewed Feb 9, 2026

View reviewed changes

src/Access/Common/AccessType.h Outdated Show resolved Hide resolved

divanik and others added 2 commits February 9, 2026 13:38

Remove unrelated change

7e86306

updating show privileges reference for SYSTEM CLEAR privileges

f589706

divanik added this pull request to the merge queue Feb 10, 2026

Merged via the queue into ClickHouse:master with commit 5243ef7 Feb 10, 2026
127 of 134 checks passed

robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Feb 10, 2026

alexey-milovidov added a commit that referenced this pull request Feb 13, 2026

Revert "Merge pull request #89750 from grantholly-clickhouse/parquet_…

223e53c

…footer_cache" This reverts commit 5243ef7, reversing changes made to 6e3a5ad.

alexey-milovidov mentioned this pull request Feb 13, 2026

Revert "Merge pull request #89750 - parquet_footer_cache" #96795

Merged

1 task

alexey-milovidov added a commit that referenced this pull request Feb 13, 2026

Merge pull request #96795 from ClickHouse/revert-parquet-footer-cache

e91859b

Revert "Merge pull request #89750 - parquet_footer_cache"

grantholly-clickhouse added a commit to grantholly-clickhouse/ClickHouse that referenced this pull request Feb 13, 2026

Reapply "Merge pull request ClickHouse#89750 from grantholly-clickhou…

696588e

…se/parquet_footer_cache" This reverts commit 223e53c.

grantholly-clickhouse mentioned this pull request Mar 5, 2026

Parquet metadata cache v2 #98140

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parquet footer cache#89750

Parquet footer cache#89750
divanik merged 138 commits intoClickHouse:masterfrom
grantholly-clickhouse:parquet_footer_cache

grantholly-clickhouse commented Nov 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grantholly-clickhouse Feb 5, 2026

Uh oh!

divanik Feb 6, 2026

Uh oh!

divanik Feb 6, 2026

Uh oh!

grantholly-clickhouse Feb 6, 2026

Uh oh!

divanik commented Feb 9, 2026

Uh oh!

divanik commented Feb 9, 2026

Uh oh!

divanik left a comment

Uh oh!

Uh oh!

Uh oh!

divanik commented Feb 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

grantholly-clickhouse commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Documentation entry for user-facing changes

Details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grantholly-clickhouse Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

divanik Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

divanik Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

grantholly-clickhouse Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

divanik commented Feb 9, 2026

Uh oh!

divanik commented Feb 9, 2026

Uh oh!

divanik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

divanik commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

grantholly-clickhouse commented Nov 7, 2025 •

edited

Loading

divanik commented Feb 13, 2026 •

edited

Loading