Skip to content

Commit ea9057f

Browse files
committed
[fix](scan) Preserve legacy cache policy and isolate footer profiles
### What problem does this PR solve? Related PR: #69723 Problem Summary: ORC and Parquet registered the same flat counter names under different format parents, so the first reader acquired both formats' footer statistics. Qualify the ORC names and verify both registration orders. Restore the pre-existing ORC FileMetaCache key policy and the original native Parquet key builder, removing the newly introduced shared stable-key API. Preserve two still-valid operator cache ownership and shard-limit tests in a maintained suite. ### Release note Keep ORC and Parquet footer statistics attributable to their respective formats. ### Check List (For Author) - Test: 311 related BE ASAN unit tests; restored operator cases retested after cleanup; clang-format 16; build hygiene; changed-line clang-tidy - Behavior changed: ORC footer counters use format-qualified names; ORC cache identity retains the legacy policy and native Parquet identity is unchanged - Does this need documentation: Yes (review guide updated in this commit)
1 parent ec07842 commit ea9057f

7 files changed

Lines changed: 233 additions & 54 deletions

File tree

‎be/src/format_v2/orc/orc_reader.cpp‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1172,10 +1172,12 @@ void OrcReader::_init_profile() {
11721172
_profile, "DictionaryStripesFiltered", TUnit::UNIT, orc_profile, 1);
11731173
_orc_profile.dictionary_values_evaluated = ADD_CHILD_COUNTER_WITH_LEVEL(
11741174
_profile, "DictionaryValuesEvaluated", TUnit::UNIT, orc_profile, 1);
1175+
// Counter names are global to the scanner profile even when their parents differ. Keep ORC
1176+
// footer deltas separate from Parquet when both formats share the same scanner profile.
11751177
_orc_profile.file_footer_read_calls = ADD_CHILD_COUNTER_WITH_LEVEL(
1176-
_profile, "FileFooterReadCalls", TUnit::UNIT, orc_profile, 1);
1178+
_profile, "OrcFileFooterReadCalls", TUnit::UNIT, orc_profile, 1);
11771179
_orc_profile.file_footer_cache_hits = ADD_CHILD_COUNTER_WITH_LEVEL(
1178-
_profile, "FileFooterHitCache", TUnit::UNIT, orc_profile, 1);
1180+
_profile, "OrcFileFooterHitCache", TUnit::UNIT, orc_profile, 1);
11791181
_orc_profile.reader_call =
11801182
ADD_CHILD_COUNTER_WITH_LEVEL(_profile, "ReaderCall", TUnit::UNIT, orc_profile, 1);
11811183
_orc_profile.reader_inclusive_latency_us = ADD_CHILD_COUNTER_WITH_LEVEL(
@@ -1302,13 +1304,11 @@ Status OrcReader::init(RuntimeState* state) {
13021304
options.setReaderMetrics(&_state->reader_metrics);
13031305

13041306
auto* meta_cache = ExecEnv::GetInstance()->file_meta_cache();
1305-
auto cache_key = FileMetaCache::get_stable_key(
1306-
_file_description->fs_name, _file_reader->path().native(), _file_description->mtime,
1307-
_file_reader->mtime(), _file_description->file_size,
1308-
static_cast<int64_t>(_file_reader->size()), _file_description->is_immutable);
1307+
// Preserve the V1 identity policy, including its file-size fallback when mtime is absent.
1308+
auto cache_key = FileMetaCache::get_key(_file_reader, *_file_description);
13091309
// The shared cache also contains Parquet objects and legacy ORC strings under other keys.
13101310
// Keep serialized V2 tails in their own domain so a hit always has the expected value type.
1311-
const bool cache_enabled = !cache_key.empty() && meta_cache != nullptr && meta_cache->enabled();
1311+
const bool cache_enabled = meta_cache != nullptr && meta_cache->enabled();
13121312
cache_key.append("\0orc-v2-tail", 12);
13131313
ObjLRUCache::CacheHandle cache_handle;
13141314
const bool cache_hit = cache_enabled && meta_cache->lookup(cache_key, &cache_handle);

‎be/src/format_v2/parquet/parquet_file_context.cpp‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,8 +129,14 @@ std::string build_native_file_cache_key(std::string_view fs_name, std::string_vi
129129
int64_t description_mtime, int64_t reader_mtime,
130130
int64_t description_file_size, int64_t reader_file_size,
131131
bool is_immutable) {
132-
return FileMetaCache::get_stable_key(fs_name, path, description_mtime, reader_mtime,
133-
description_file_size, reader_file_size, is_immutable);
132+
const int64_t mtime = description_mtime != 0 ? description_mtime : reader_mtime;
133+
if (mtime == 0 && !is_immutable) {
134+
// Unknown version is not a stable identity: an overwrite can preserve path and size while
135+
// changing both footer semantics and page bytes.
136+
return {};
137+
}
138+
const int64_t file_size = description_file_size >= 0 ? description_file_size : reader_file_size;
139+
return fmt::format("{}::{}::mtime={}::size={}", fs_name, path, mtime, file_size);
134140
}
135141

136142
bool is_serialized_index_range_safe(size_t file_size, int64_t offset, int64_t length) {

‎be/src/io/fs/file_meta_cache.cpp‎

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -17,24 +17,8 @@
1717

1818
#include "io/fs/file_meta_cache.h"
1919

20-
#include <fmt/format.h>
21-
2220
namespace doris {
2321

24-
std::string FileMetaCache::get_stable_key(std::string_view fs_name, std::string_view path,
25-
int64_t description_mtime, int64_t reader_mtime,
26-
int64_t description_file_size, int64_t reader_file_size,
27-
bool is_immutable) {
28-
const int64_t mtime = description_mtime != 0 ? description_mtime : reader_mtime;
29-
// Mutable files can be overwritten without changing their size. An unknown version cannot
30-
// safely identify cached footer semantics; immutable table-format files do not have that risk.
31-
if (mtime == 0 && !is_immutable) {
32-
return {};
33-
}
34-
const int64_t file_size = description_file_size >= 0 ? description_file_size : reader_file_size;
35-
return fmt::format("{}::{}::mtime={}::size={}", fs_name, path, mtime, file_size);
36-
}
37-
3822
std::string FileMetaCache::get_key(const std::string file_name, int64_t modification_time,
3923
int64_t file_size) {
4024
std::string meta_cache_key;

‎be/src/io/fs/file_meta_cache.h‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,6 @@
1717

1818
#pragma once
1919

20-
#include <string_view>
21-
2220
#include "io/file_factory.h"
2321
#include "io/fs/file_reader_writer_fwd.h"
2422
#include "util/obj_lru_cache.h"
@@ -43,11 +41,6 @@ class FileMetaCache {
4341
static std::string get_key(io::FileReaderSPtr file_reader,
4442
const io::FileDescription& _file_description);
4543

46-
static std::string get_stable_key(std::string_view fs_name, std::string_view path,
47-
int64_t description_mtime, int64_t reader_mtime,
48-
int64_t description_file_size, int64_t reader_file_size,
49-
bool is_immutable);
50-
5144
bool lookup(const std::string& key, ObjLRUCache::CacheHandle* handle) {
5245
return _cache.lookup({key}, handle);
5346
}
Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "exec/operator/file_scan_operator.h"
19+
20+
#include <gtest/gtest.h>
21+
22+
#include <cstdint>
23+
#include <limits>
24+
#include <list>
25+
#include <map>
26+
#include <memory>
27+
#include <string>
28+
#include <utility>
29+
#include <vector>
30+
31+
#include "common/object_pool.h"
32+
#include "exec/pipeline/dependency.h"
33+
#include "exec/scan/file_scanner_v2.h"
34+
#include "exec/scan/scanner.h"
35+
#include "exec/scan/scanner_scheduler.h"
36+
#include "gen_cpp/PlanNodes_types.h"
37+
#include "runtime/descriptors.h"
38+
#include "testutil/mock/mock_runtime_state.h"
39+
40+
namespace doris::pipeline {
41+
namespace {
42+
43+
// One tuple with no slots: all a file scan node needs to be initialized and prepared.
44+
Status create_one_tuple_descriptors(ObjectPool* pool, DescriptorTbl** descriptors) {
45+
TDescriptorTable thrift_descriptors;
46+
TTupleDescriptor tuple_descriptor;
47+
tuple_descriptor.id = 0;
48+
tuple_descriptor.byteSize = 0;
49+
tuple_descriptor.numNullBytes = 0;
50+
thrift_descriptors.tupleDescriptors.push_back(tuple_descriptor);
51+
return DescriptorTbl::create(pool, thrift_descriptors, descriptors);
52+
}
53+
54+
// A file scan node over that tuple.
55+
TPlanNode file_scan_plan_node() {
56+
TPlanNode plan_node;
57+
plan_node.node_id = 0;
58+
plan_node.node_type = TPlanNodeType::FILE_SCAN_NODE;
59+
plan_node.num_children = 0;
60+
plan_node.limit = -1;
61+
plan_node.row_tuples.push_back(0);
62+
plan_node.file_scan_node.tuple_id = 0;
63+
plan_node.__isset.file_scan_node = true;
64+
return plan_node;
65+
}
66+
67+
TFileScanRangeParams scan_level_format(const std::string& table_format) {
68+
TFileScanRangeParams params;
69+
if (!table_format.empty()) {
70+
TTableFormatFileDesc fmt;
71+
fmt.__set_table_format_type(table_format);
72+
params.__set_table_format_params(fmt);
73+
}
74+
return params;
75+
}
76+
77+
TFileRangeDesc jni_range(const std::string& table_format) {
78+
TTableFormatFileDesc fmt;
79+
fmt.__set_table_format_type(table_format);
80+
TFileRangeDesc range;
81+
range.__set_format_type(TFileFormatType::FORMAT_JNI);
82+
range.__set_table_format_params(fmt);
83+
return range;
84+
}
85+
86+
void expect_scanners_share_node_cache(FileScanLocalState* local_state, MockRuntimeState* state,
87+
FileScanOperatorX* node) {
88+
std::list<ScannerSPtr> scanners;
89+
ASSERT_TRUE(local_state->_init_scanners(&scanners).ok());
90+
ASSERT_FALSE(scanners.empty());
91+
for (const auto& scanner : scanners) {
92+
const auto* file_scanner = dynamic_cast<FileScannerV2*>(scanner.get());
93+
ASSERT_NE(file_scanner, nullptr);
94+
EXPECT_EQ(file_scanner->_kv_cache, node->_kv_cache.get());
95+
}
96+
}
97+
98+
} // namespace
99+
100+
// All instances of a node share cached log-tail and delete-file data. A per-instance cache
101+
// would reload the same entry when splits referencing it are assigned to different instances.
102+
TEST(FileScanOperatorCacheTest, EveryInstanceOfTheNodeReadsThroughTheNodesCache) {
103+
ObjectPool pool;
104+
DescriptorTbl* descriptors = nullptr;
105+
ASSERT_TRUE(create_one_tuple_descriptors(&pool, &descriptors).ok());
106+
MockRuntimeState state;
107+
state.set_desc_tbl(descriptors);
108+
109+
const TPlanNode plan_node = file_scan_plan_node();
110+
FileScanOperatorX node(&pool, plan_node, 0, *descriptors, /*parallel_tasks=*/1);
111+
ASSERT_TRUE(node.init(plan_node, &state).ok());
112+
ASSERT_TRUE(node.prepare(&state).ok());
113+
ASSERT_NE(node._kv_cache, nullptr);
114+
115+
TFileRangeDesc log_range = jni_range("fluss");
116+
log_range.table_format_params.__set_fluss_params({{"fluss.range_type", "LOG"}});
117+
TFileScanRange file_scan_range;
118+
file_scan_range.__set_params(scan_level_format("fluss"));
119+
// A query reads no source tuple; tuple 0 there would make this scan a load.
120+
file_scan_range.params.__set_src_tuple_id(1);
121+
file_scan_range.ranges.push_back(log_range);
122+
TScanRangeParams scan_range;
123+
scan_range.scan_range.ext_scan_range.__set_file_scan_range(file_scan_range);
124+
scan_range.scan_range.__isset.ext_scan_range = true;
125+
const std::vector<TScanRangeParams> scan_ranges {scan_range};
126+
127+
RuntimeProfile profile("EveryInstanceOfTheNodeReadsThroughTheNodesCache");
128+
const std::map<int, std::pair<std::shared_ptr<BasicSharedState>,
129+
std::vector<std::shared_ptr<Dependency>>>>
130+
shared_state_map;
131+
for (int instance = 0; instance < 2; ++instance) {
132+
SCOPED_TRACE(instance);
133+
auto local_state = FileScanLocalState::create_unique(&state, &node);
134+
LocalStateInfo info {.parent_profile = &profile,
135+
.scan_ranges = scan_ranges,
136+
.shared_state = nullptr,
137+
.shared_state_map = shared_state_map,
138+
.task_idx = instance};
139+
ASSERT_TRUE(local_state->init(&state, info).ok());
140+
ASSERT_NO_FATAL_FAILURE(expect_scanners_share_node_cache(local_state.get(), &state, &node));
141+
}
142+
}
143+
144+
// An unlimited scanner setting can overflow an int when multiplied by the instance count.
145+
// Shard count must instead be capped by remote scan threads, preserving concurrency across instances.
146+
TEST(FileScanOperatorCacheTest, CacheShardsOfAnUnlimitedScannerSettingAreCappedNotOverflowed) {
147+
ObjectPool pool;
148+
DescriptorTbl* descriptors = nullptr;
149+
ASSERT_TRUE(create_one_tuple_descriptors(&pool, &descriptors).ok());
150+
MockRuntimeState state;
151+
state.set_desc_tbl(descriptors);
152+
state._query_options.__set_parallel_instance(2);
153+
state._query_options.__set_max_file_scanners_concurrency(std::numeric_limits<int32_t>::max());
154+
155+
const TPlanNode plan_node = file_scan_plan_node();
156+
FileScanOperatorX node(&pool, plan_node, 0, *descriptors, /*parallel_tasks=*/2);
157+
ASSERT_TRUE(node.init(plan_node, &state).ok());
158+
ASSERT_TRUE(node.prepare(&state).ok());
159+
EXPECT_EQ(node._kv_cache->_num_shards,
160+
static_cast<uint32_t>(ScannerScheduler::default_remote_scan_thread_num()));
161+
}
162+
163+
} // namespace doris::pipeline

‎be/test/format_v2/orc/orc_reader_test.cpp‎

Lines changed: 42 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -7181,6 +7181,31 @@ TEST_F(NewOrcReaderTest, ProfileKeepsPruningCountersBelowOrcReader) {
71817181
EXPECT_TRUE(children.at("OrcReader").contains("OrcFilteredRowsByGroup"));
71827182
}
71837183

7184+
void expect_footer_counter_isolation(RuntimeProfile* profile, format::orc::OrcReader* orc_reader,
7185+
const format::parquet::ParquetProfile& parquet_profile) {
7186+
auto* orc_reads = orc_reader->_orc_profile.file_footer_read_calls;
7187+
auto* orc_hits = orc_reader->_orc_profile.file_footer_cache_hits;
7188+
EXPECT_NE(orc_reads, parquet_profile.file_footer_read_calls);
7189+
EXPECT_NE(orc_hits, parquet_profile.file_footer_hit_cache);
7190+
// A shared scanner profile must keep both formats' deltas regardless of registration order.
7191+
COUNTER_UPDATE(orc_reads, 3);
7192+
COUNTER_UPDATE(orc_hits, 5);
7193+
COUNTER_UPDATE(parquet_profile.file_footer_read_calls, 7);
7194+
COUNTER_UPDATE(parquet_profile.file_footer_hit_cache, 11);
7195+
EXPECT_EQ(orc_reads->value(), 3);
7196+
EXPECT_EQ(orc_hits->value(), 5);
7197+
EXPECT_EQ(parquet_profile.file_footer_read_calls->value(), 7);
7198+
EXPECT_EQ(parquet_profile.file_footer_hit_cache->value(), 11);
7199+
TRuntimeProfileTree tree;
7200+
profile->to_thrift(&tree, 3);
7201+
ASSERT_FALSE(tree.nodes.empty());
7202+
const auto& children = tree.nodes.front().child_counters_map;
7203+
EXPECT_TRUE(children.at("OrcReader").contains("OrcFileFooterReadCalls"));
7204+
EXPECT_TRUE(children.at("OrcReader").contains("OrcFileFooterHitCache"));
7205+
EXPECT_TRUE(children.at("ParquetReader").contains("FileFooterReadCalls"));
7206+
EXPECT_TRUE(children.at("ParquetReader").contains("FileFooterHitCache"));
7207+
}
7208+
71847209
TEST_F(NewOrcReaderTest, ProfileCountersStayIsolatedWhenParquetInitializesFirst) {
71857210
RuntimeProfile profile("parquet_then_orc_profile");
71867211
format::parquet::ParquetProfile parquet_profile;
@@ -7193,6 +7218,7 @@ TEST_F(NewOrcReaderTest, ProfileCountersStayIsolatedWhenParquetInitializesFirst)
71937218
ASSERT_NE(profile.get_counter("RowGroupsFiltered"), nullptr);
71947219
EXPECT_NE(profile.get_counter("OrcRowGroupsFiltered"),
71957220
profile.get_counter("RowGroupsFiltered"));
7221+
expect_footer_counter_isolation(&profile, reader.get(), parquet_profile);
71967222
}
71977223

71987224
TEST_F(NewOrcReaderTest, ProfileCountersStayIsolatedWhenOrcInitializesFirst) {
@@ -7207,6 +7233,7 @@ TEST_F(NewOrcReaderTest, ProfileCountersStayIsolatedWhenOrcInitializesFirst) {
72077233
ASSERT_NE(profile.get_counter("RowGroupsFiltered"), nullptr);
72087234
EXPECT_NE(profile.get_counter("OrcRowGroupsFiltered"),
72097235
profile.get_counter("RowGroupsFiltered"));
7236+
expect_footer_counter_isolation(&profile, reader.get(), parquet_profile);
72107237
}
72117238

72127239
TEST_F(NewOrcReaderTest, DisableOrcFilterByMinMaxKeepsRowGroupProfileZero) {
@@ -8027,7 +8054,7 @@ TEST_F(NewOrcReaderTest, SargVarcharCastToStringConjunctPrunesStripes) {
80278054
EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 200);
80288055
}
80298056

8030-
TEST_F(NewOrcReaderTest, FooterCacheAvoidsWarmReadsAndSeparatesFileVersions) {
8057+
TEST_F(NewOrcReaderTest, FooterCachePreservesLegacyFileIdentityAndAvoidsWarmReads) {
80318058
FileMetaCache cache(1024);
80328059
auto* env = ExecEnv::GetInstance();
80338060
auto* previous = env->_file_meta_cache;
@@ -8041,38 +8068,33 @@ TEST_F(NewOrcReaderTest, FooterCacheAvoidsWarmReadsAndSeparatesFileVersions) {
80418068
sp->set_call_back(
80428069
"LocalFileReader::pread", [&](auto&&) { ++reads; }, &guard);
80438070
RuntimeState state {TQueryOptions(), TQueryGlobals()};
8044-
const auto open = [&](int64_t version, const std::string& fs_name, bool immutable = false) {
8071+
const auto open = [&](int64_t version) {
80458072
reads = 0;
80468073
RuntimeProfile profile("footer-cache");
80478074
auto reader = create_reader(&profile);
80488075
reader->_file_description->mtime = version;
8049-
reader->_file_description->fs_name = fs_name;
8050-
reader->_file_description->is_immutable = immutable;
80518076
auto status = reader->init(&state);
80528077
EXPECT_TRUE(status.ok()) << status;
80538078
EXPECT_EQ(reader->get_total_rows(), ROW_COUNT);
80548079
EXPECT_TRUE(reader->close().ok());
8055-
const auto read_calls = profile.get_counter("FileFooterReadCalls")->value();
8056-
const auto cache_hits = profile.get_counter("FileFooterHitCache")->value();
8080+
const auto read_calls = profile.get_counter("OrcFileFooterReadCalls")->value();
8081+
const auto cache_hits = profile.get_counter("OrcFileFooterHitCache")->value();
80578082
EXPECT_TRUE(reader->close().ok());
8058-
EXPECT_EQ(profile.get_counter("FileFooterReadCalls")->value(), read_calls);
8059-
EXPECT_EQ(profile.get_counter("FileFooterHitCache")->value(), cache_hits);
8083+
EXPECT_EQ(profile.get_counter("OrcFileFooterReadCalls")->value(), read_calls);
8084+
EXPECT_EQ(profile.get_counter("OrcFileFooterHitCache")->value(), cache_hits);
80608085
return reads;
80618086
};
8062-
EXPECT_GT(open(1, "fs-one"), 0);
8063-
EXPECT_EQ(open(1, "fs-one"), 0);
8064-
EXPECT_GT(open(2, "fs-one"), 0);
8065-
EXPECT_EQ(open(2, "fs-one"), 0);
8066-
EXPECT_GT(open(2, "fs-two"), 0);
8067-
EXPECT_EQ(open(2, "fs-two"), 0);
8068-
EXPECT_GT(open(0, "mutable"), 0);
8069-
EXPECT_GT(open(0, "mutable"), 0);
8070-
EXPECT_GT(open(0, "immutable", true), 0);
8071-
EXPECT_EQ(open(0, "immutable", true), 0);
8087+
EXPECT_GT(open(1), 0);
8088+
EXPECT_EQ(open(1), 0);
8089+
EXPECT_GT(open(2), 0);
8090+
EXPECT_EQ(open(2), 0);
8091+
// V1 caches versionless files by path and size; migration preserves that existing behavior.
8092+
EXPECT_GT(open(0), 0);
8093+
EXPECT_EQ(open(0), 0);
80728094
FileMetaCache disabled(0);
80738095
env->_file_meta_cache = &disabled;
8074-
EXPECT_GT(open(1, "fs-one"), 0);
8075-
EXPECT_GT(open(1, "fs-one"), 0);
8096+
EXPECT_GT(open(1), 0);
8097+
EXPECT_GT(open(1), 0);
80768098
}
80778099

80788100
TEST_F(NewOrcReaderTest, DictionaryFiltersEncodedRowsBeforeExpandingLongStrings) {

‎docs/file-scanner-v2-code-review-guide.md‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,17 @@ This guide contains the detailed checklists referenced by
44
`be/src/format_v2/AGENTS.md`. Read the common checklist for every FileReader review, then apply the
55
format-specific checklist when reviewing Parquet or ORC.
66

7+
## Retiring an Existing Reader
8+
9+
- Compare supported inputs, results, errors, lifecycle, profile ownership and performance behavior
10+
with the old caller chain. Check actual FE payloads, including omitted optional metadata, before
11+
treating a failing synthetic request as a migration regression.
12+
- Inventory tests by the invariant they protect. Keep shared/operator and active V2 coverage even
13+
when its fixture also contains obsolete routing tests or lives in a retired reader directory.
14+
- Keep unrelated cache-policy changes and other behavior improvements separate from retirement.
15+
Restore the supported legacy contract at the responsible layer and test interactions with
16+
shared state, such as both format initialization orders on one scanner profile.
17+
718
## Common FileReader: Indexes and Predicate Filtering
819

920
- Inventory the reader's actual pruning capabilities before evaluating a change: metadata or
@@ -313,8 +324,8 @@ format-specific checklist when reviewing Parquet or ORC.
313324
should filter encoded rows before expanding rejected strings, including with lazy materialization
314325
disabled. Reset match bits on every Stripe, preserve NULL semantics and pruning fences, and keep
315326
physical row positions distinct from compact predicate-column positions.
316-
- Verify warm ORC opens reuse serialized tails only for stable file identities. Filesystem identity,
317-
version and size must participate in the key; mutable files without a version bypass the cache.
327+
- Verify warm ORC opens reuse serialized tails with the existing V1 `FileMetaCache` key policy,
328+
and keep cached V2 metadata representations isolated from other payload types.
318329
- Review SARG cost for large IN lists, deep trees, many Runtime Filters, repeated literal conversion,
319330
Stripe-statistics reads, and SDK index initialization. Build once per reader/Split setup and keep
320331
expensive work out of batch loops.

0 commit comments

Comments
 (0)