Skip to content

Commit e493c61

Browse files
authored
[improvement](be) Prioritize Parquet V2 row groups for TopN scans (apache#69646)
### What problem does this PR solve? Problem Summary: Parquet V2 currently visits row groups in physical file order. For a small `ORDER BY ... LIMIT`, the best values can be near the end of a file. Earlier groups are then decoded before the TopN sink publishes a useful bound. Even after a bound arrives, the deferred row-group path does not recheck the footer unless a new scan request is published; TopN updates its bound in place. `VTopNPred` also lacks the ZoneMap interface needed to reject whole row groups. For example, consider four groups containing consecutive integer keys `[300, 399]`, `[200, 299]`, `[100, 199]`, `[0, 99]` and `ORDER BY k ASC LIMIT 3`. Reading the last group first produces a bound of `2`; the other three groups can be rejected using their footer minimum before dictionary/index probes, prefetch, or data decoding. The descending case uses the footer maximum. ### Optimization approach - Add conservative ZoneMap evaluation to `VTopNPred`. Snapshot the current runtime bound at each evaluation, retain equality for secondary sort keys, honor NULL ordering, and fall back when types, ranges, or floating-point NaN information are unsuitable. - Prioritize selected row groups by the leading TopN slot's sanitized footer minimum/maximum. Resolve the file-local slot through the request's column mapping. Keep execution indices separate from the physical plans so row IDs, delete coordinates, and Condition Cache boundaries remain valid. - Recheck the latest TopN bound at every row-group boundary before expensive metadata probes and I/O. Files without statistics retain the existing path. Reordering respects row-ID fetch order and the request's existing metadata-pruning boundary; missing or unusable statistics never exclude a group by themselves. ### Result and evaluation contract The optimization must preserve the final TopN result, including secondary-key ties and NULL placement. It may avoid evaluating expressions or materializing columns for rows that a valid TopN bound proves cannot contribute to that result. Preserving potential exceptions from such excluded rows is not a requirement: a DATE overflow in a discarded row group, for example, does not require reading that group. Existing TopN/runtime-filter pushdown eligibility and file-to-table localization remain unchanged. No executed error is caught, ignored, or converted to success. A conversion that is actually needed still follows the existing strict/non-strict rules and propagates its failure. This distinction also applies to unprojected equality-delete keys and set-operation inputs reduced through existing runtime filters. It does not permit skipping candidate rows, bypassing deletes for retained rows, re-evaluating a volatile leading key as a different value, or trusting incompatible statistics. Earlier follow-ups added FE expression whitelists and table-mapping restrictions to preserve errors in rows outside the result. Those restrictions are removed; they unnecessarily reduced existing optimization eligibility. The production change is confined to the TopN ZoneMap capability and Parquet scheduling. Existing metadata-pruning boundaries and ordered row-ID fetches remain respected. Upgrade compatibility for legacy FE requests using Parquet V1 is outside the scope of this PR; the follow-up dedicated to that case has been reverted. ### Eligibility and limits - FE must already generate a TopN runtime filter for the plan and leading-key type. Its existing selectivity check remains `max(estimated_scan_rows, 1) * topn_filter_ratio > LIMIT + OFFSET` (default ratio: `0.5`); existing window, CTE, nested-TopN and join-side rules remain unchanged. - This row-group scheduling optimization applies to Parquet V2. After localization, the leading-key filter must reference a direct, non-repeated primitive file column with a compatible type. Arbitrary computed keys such as `abs(k)` are not mapped to footer bounds. Projected or secondary-key expressions do not need a new non-throwing proof. - Reordering requires at least two selected row groups, at least one usable leading-key range, no ordered row-ID fetch, and all localized conjuncts inside the existing metadata-pruning-safe boundary. Ordering is within the current file/split, without coordinating a global order across files or scanners. Current-bound footer rechecks can still apply without reordering when their own eligibility holds. - ASC/DESC, NULL ordering, OFFSET and multiple sort keys retain their existing semantics. Leading-key equality is retained for secondary keys. Missing/untrusted statistics, incompatible types, or an unavailable bound retain candidates; they cannot by themselves reject a group. - Benefits depend on reaching promising groups before the others are read and on having separable ranges. Overlapping ranges, many leading-key ties, one group per split, or late feedback can substantially reduce the gain. No new production setting is introduced. ### Measured optimization effect The included opt-in `ParquetScanTest.TopNRowGroupOrderMicrobench` exercises the actual Parquet V2 reader plus a bounded TopK heap and per-block updates of the real runtime predicate. Its workload corresponds to: ```sql SELECT k FROM parquet_source ORDER BY k ASC LIMIT 10; SELECT k FROM parquet_source ORDER BY k DESC LIMIT 10; ``` Fixture: one required INT32 column, 128 row groups x 4,096 rows (524,288 rows), disjoint key ranges, and the best group physically last for each direction. Files use uncompressed PLAIN DataPageV2 with dictionary and PageIndex disabled, isolating footer pruning. Every iteration verifies all ten final values. | Per reader query | Before | After | | --- | ---: | ---: | | Row groups decoded | 128 | 1 | | RawRowsRead | 524,288 | 4,096 (-99.22%) | | RowGroupsFilteredByMinMax | 0 | 127 | | Actual local read bytes | 2,149,888 | 65,602 (-96.95%) | | ASC median latency | 10.698 ms | 0.669 ms (~16.0x) | | DESC median latency | 8.590 ms | 0.660 ms (~13.0x) | Method: Clang 20 RELEASE builds, warm local filesystem cache, the same pinned CPU, nine paired samples alternating baseline/candidate order, and 32 reader queries per sample after warmup. These paired timings were collected before the eligibility cleanup; the direct one-column workload exercises the same scheduling path. Fresh reader correctness tests still verify best-first reads and late-bound group rejection. The baseline restores `vtopn_pred.{h,cpp}` and `parquet_scan.{h,cpp}` from the pre-optimization base `8d2bd1e14b6d95d923d43e00857da326639cc466` while retaining the identical benchmark harness and request plumbing; the benchmark calls ParquetReader directly. All paired checksums match. Work counters are identical across all samples. ASC candidate samples range from 0.647 to 1.097 ms and DESC from 0.642 to 0.745 ms; the shared host has timing outliers. These are reader microbenchmark results; complete SQL gains depend on file splits, parallelism, feedback latency, and storage. ### Validation and reproduction - Fresh verification of the revised implementation: 497 BE tests passed (one opt-in microbenchmark skipped in the ordinary run), and all 31 FE tests passed across the four suites below. The three aggregation query shapes also pass the final translated `TOPN OPT:` EXPLAIN assertion. The output-evolution optimization test and FE filter-generation test first failed on the preceding implementation, then passed after removal of the unnecessary fences. - Reader integration covers ASC/DESC, top-level/nested DATE-to-TIMESTAMP_NS output evolution, strict cast on/off, strict identity DATE materialization, and an unprojected Iceberg equality-delete key. Excluded groups are skipped. The native and hidden-delete cases include ordinary-scan and reversed-TopN controls that still raise the original conversion error when the invalid group is required; existing output-evolution tests also retain their overflow checks. The tests verify all surviving keys/converted values and row counts with real per-block TopN feedback. - FE Checkstyle (zero violations), clang-format 16, BE build hygiene, and changed-line clang-tidy for every affected C++ file passed. - Existing reader tests cover exact TopK results, physical row IDs, localized slot mapping, missing statistics, metadata-pruning boundaries, ordered row-ID fetches, NULL placement, and equality boundaries. - FE coverage checks that potentially throwing secondary expressions do not disable existing TopN runtime-filter generation. The existing `nereids_tpch_p0/tpch/topn-filter.groovy` also exposed an actual regression: the blanket visitor stop removed `TOPN OPT` for `GROUP BY k ORDER BY k LIMIT 1`. Restore the original visitor traversal and cover grouped keys, aggregation with HAVING and a join, and grouping sets with `push_topn_to_agg=false`. These cases run full FE planning and check `TOPN OPT:` in the translated EXPLAIN fragments. The same unnecessary expression whitelist also disabled `TOPN OPT` for a nested STRUCT order key with an `IS NOT NULL` predicate on a sibling field, as exposed by `nereids_rules_p0/column_pruning/topn_filter_nested_column_pruning.groovy`. Additional translated-plan checks cover TIMESTAMPTZ, INT and DATE fields in ASC/DESC, including the complete and predicate-only nested access paths. Both existing regression suites and their assertions are unchanged. FE production logic matches the pre-PR base; only an explanatory traversal comment remains in the FE production diff. - The earlier SQL suite asserting that all excluded-row errors must be preserved is removed. Complete SQL performance is not claimed; performance evidence comes from the reader microbenchmark below. Correctness suites: ```bash ./run-be-ut.sh -j8 --run \ --filter='*Parquet*:*ColumnMapper*:*TableReader*:*Iceberg*:*EvolutionCast*:*RuntimePredicate*' ./run-fe-ut.sh --run \ org.apache.doris.nereids.postprocess.TopNRuntimeFilterTest,org.apache.doris.nereids.rules.rewrite.DateArithmeticFilterPushDownTest,org.apache.doris.nereids.postprocess.PushDownFilterThroughProjectTest,org.apache.doris.nereids.util.ExpressionUtilsTest ``` Opt-in benchmark (use a RELEASE UT build): ```bash DORIS_TOPN_ROW_GROUP_BENCHMARK=1 BUILD_TYPE_UT=RELEASE \ ./run-be-ut.sh -j8 --run --filter=ParquetScanTest.TopNRowGroupOrderMicrobench ``` For paired comparison, build and save baseline/candidate UT executables with the same harness, then use the existing runner: ```bash DORIS_TOPN_ROW_GROUP_BENCHMARK=1 \ DORIS_BE_UT_EXECUTABLE=build-support/run-query-engine-paired-benchmarks.py \ QUERY_ENGINE_BENCH_BASELINE=./baseline/doris_be_test \ QUERY_ENGINE_BENCH_CANDIDATE=./candidate/doris_be_test \ QUERY_ENGINE_BENCH_CPU=0 \ QUERY_ENGINE_BENCH_OUTPUT=./paired-results \ BUILD_TYPE_UT=RELEASE ./run-be-ut.sh -j8 --run \ --filter=ParquetScanTest.TopNRowGroupOrderMicrobench ``` Select an available idle CPU for `QUERY_ENGINE_BENCH_CPU`. The runner emits paired timings/checksums; `PARQUET_TOPN_WORK` lines report per-query raw rows, groups filtered by min/max, and actual local read bytes. ### Release note Improve Parquet V2 TopN scans by reading promising row groups first and applying the latest runtime bound before opening subsequent groups. ### Check List (For Author) - Test - [x] Unit Test - [x] Manual test (reader microbenchmark and commands above) - Behavior changed: - [x] Yes. Eligible Parquet V2 TopN scans may read row groups in a different order and skip more groups. SQL ordering remains defined by the TopN operator. Expressions and conversions in excluded groups may not execute, so an exception from an excluded row may no longer be observed. - Does this need documentation? - [x] No. Internal scan optimization with no new user-facing option. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 69d0094 commit e493c61

8 files changed

Lines changed: 867 additions & 5 deletions

File tree

‎be/src/exprs/vtopn_pred.cpp‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,42 @@ bool topn_string_matches(int comparison, RawComparisonOp op) {
9797

9898
} // namespace
9999

100+
bool VTopNPred::can_evaluate_zonemap_filter() const {
101+
// Capability must remain available before the first bound; readers cache it during open.
102+
return _predicate != nullptr && _children.size() == 1 && _children[0]->is_slot_ref();
103+
}
104+
105+
ZoneMapFilterResult VTopNPred::evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const {
106+
if (!can_evaluate_zonemap_filter()) {
107+
return unsupported_zonemap_filter(ctx);
108+
}
109+
const Field bound = _predicate->get_value();
110+
if (bound.is_null()) {
111+
return ZoneMapFilterResult::kMayMatch;
112+
}
113+
const auto* slot = assert_cast<const VSlotRef*>(_children[0].get());
114+
const auto zone = ctx.zone_map(slot->column_id());
115+
const auto type = ctx.data_type(slot->column_id());
116+
if (zone == nullptr || !expr_zonemap::data_types_compatible(type, slot->data_type())) {
117+
return unsupported_zonemap_filter(ctx);
118+
}
119+
if (_predicate->nulls_first() && zone->has_null) {
120+
return ZoneMapFilterResult::kMayMatch;
121+
}
122+
if (!zone->has_not_null) {
123+
return ZoneMapFilterResult::kNoMatch;
124+
}
125+
if (ctx.floating_nan_count_unknown(slot->column_id()) || bound.is_nan() ||
126+
!expr_zonemap::range_stats_usable_for_zonemap(*zone, type) ||
127+
!expr_zonemap::field_types_compatible(bound.get_type(),
128+
remove_nullable(type)->get_primitive_type())) {
129+
return unsupported_zonemap_filter(ctx);
130+
}
131+
// Keep equality: with ORDER BY a, b a row tied on a can still improve b.
132+
const bool excluded = _predicate->is_asc() ? bound < zone->min_value : zone->max_value < bound;
133+
return excluded ? ZoneMapFilterResult::kNoMatch : ZoneMapFilterResult::kMayMatch;
134+
}
135+
100136
bool VTopNPred::can_execute_on_raw_fixed_values(const DataTypePtr& data_type, int column_id) const {
101137
if (_predicate == nullptr || data_type == nullptr || _children.size() != 1) {
102138
return false;

‎be/src/exprs/vtopn_pred.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,12 @@ class VTopNPred : public VExpr {
6363
}
6464

6565
int source_node_id() const { return _source_node_id; }
66+
bool is_asc() const { return _predicate->is_asc(); }
67+
bool nulls_first() const { return _predicate->nulls_first(); }
68+
69+
bool can_evaluate_zonemap_filter() const override;
70+
ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override;
71+
6672
Status clone_node(VExprSPtr* cloned_expr) const override {
6773
DORIS_CHECK(cloned_expr != nullptr);
6874
*cloned_expr = VTopNPred::create_shared(clone_texpr_node(), _source_node_id, nullptr);

‎be/src/format_v2/parquet/parquet_scan.cpp‎

Lines changed: 137 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include <iterator>
2121
#include <limits>
2222
#include <memory>
23+
#include <numeric>
2324
#include <optional>
2425
#include <ranges>
2526
#include <set>
@@ -43,6 +44,7 @@
4344
#include "exprs/vcompound_pred.h"
4445
#include "exprs/vectorized_fn_call.h"
4546
#include "exprs/vexpr_context.h"
47+
#include "exprs/vtopn_pred.h"
4648
#include "format_v2/parquet/parquet_column_schema.h"
4749
#include "format_v2/parquet/parquet_file_context.h"
4850
#include "format_v2/parquet/parquet_statistics.h"
@@ -1230,6 +1232,9 @@ void ParquetScanScheduler::set_condition_cache_context(std::shared_ptr<Condition
12301232

12311233
void ParquetScanScheduler::reset() {
12321234
_next_row_group_plan_idx = 0;
1235+
_row_group_order.clear();
1236+
_row_group_order_prepared = false;
1237+
_topn_footer_recheck = false;
12331238
_raw_rows_read = 0;
12341239
_predicate_schedule_request = nullptr;
12351240
_predicate_schedule = {};
@@ -1409,21 +1414,147 @@ std::vector<format::LocalColumnIndex> ParquetScanScheduler::adaptive_predicate_p
14091414
return result;
14101415
}
14111416

1417+
namespace {
1418+
1419+
struct TopNOrderingColumn {
1420+
const VTopNPred* predicate;
1421+
const ParquetColumnSchema* schema;
1422+
};
1423+
1424+
std::optional<TopNOrderingColumn> find_topn_ordering_column(
1425+
const std::vector<std::unique_ptr<ParquetColumnSchema>>& file_schema,
1426+
const format::FileScanRequest& request) {
1427+
const size_t safe_count =
1428+
std::min(request.metadata_pruning_safe_conjunct_count, request.conjuncts.size());
1429+
const VTopNPred* topn = nullptr;
1430+
for (size_t idx = 0; idx < safe_count; ++idx) {
1431+
const auto& root = request.conjuncts[idx]->root();
1432+
if (root->is_topn_filter() && root->can_evaluate_zonemap_filter()) {
1433+
topn = assert_cast<const VTopNPred*>(root.get());
1434+
break;
1435+
}
1436+
}
1437+
if (topn == nullptr) {
1438+
return std::nullopt;
1439+
}
1440+
const auto* slot = assert_cast<const VSlotRef*>(topn->get_child(0).get());
1441+
const auto position = std::ranges::find_if(request.local_positions, [&](const auto& entry) {
1442+
return entry.second.value() == slot->column_id();
1443+
});
1444+
if (position == request.local_positions.end()) {
1445+
return std::nullopt;
1446+
}
1447+
const auto column = std::ranges::find_if(file_schema, [&](const auto& schema) {
1448+
return schema->local_id == position->first.value();
1449+
});
1450+
if (column == file_schema.end() || (*column)->kind != ParquetColumnSchemaKind::PRIMITIVE ||
1451+
(*column)->max_repetition_level != 0 ||
1452+
!expr_zonemap::data_types_compatible((*column)->type, slot->data_type())) {
1453+
return std::nullopt;
1454+
}
1455+
return TopNOrderingColumn {.predicate = topn, .schema = column->get()};
1456+
}
1457+
1458+
} // namespace
1459+
1460+
void ParquetScanScheduler::prepare_topn_row_group_order(
1461+
const ParquetFileContext& file_context,
1462+
const std::vector<std::unique_ptr<ParquetColumnSchema>>& file_schema,
1463+
const format::FileScanRequest& request) {
1464+
_row_group_order_prepared = true;
1465+
const auto binding = find_topn_ordering_column(file_schema, request);
1466+
if (!binding.has_value()) {
1467+
return;
1468+
}
1469+
const auto* topn = binding->predicate;
1470+
const auto& schema = *binding->schema;
1471+
const size_t safe_count =
1472+
std::min(request.metadata_pruning_safe_conjunct_count, request.conjuncts.size());
1473+
const auto& metadata = file_context.native_metadata->to_thrift();
1474+
_topn_footer_recheck = std::ranges::any_of(_scan_plan->row_groups, [&](const auto& plan) {
1475+
const auto& group = metadata.row_groups[plan.row_group_id];
1476+
DORIS_CHECK(schema.leaf_column_id >= 0 && schema.leaf_column_id < group.columns.size());
1477+
const auto& chunk = group.columns[schema.leaf_column_id];
1478+
return chunk.__isset.meta_data && chunk.meta_data.__isset.statistics;
1479+
});
1480+
// Scheduling need not evaluate conversions in groups excluded by TopN. Reuse the request's
1481+
// existing metadata-pruning boundary; row-ID fetches still require increasing file positions.
1482+
if (!_topn_footer_recheck || request.row_ids.has_value() ||
1483+
safe_count != request.conjuncts.size() || _scan_plan->row_groups.size() < 2) {
1484+
return;
1485+
}
1486+
struct Priority {
1487+
std::optional<Field> best;
1488+
bool may_have_null = false;
1489+
};
1490+
std::vector<Priority> priorities;
1491+
priorities.reserve(_scan_plan->row_groups.size());
1492+
const bool trusted_nulls =
1493+
native::parquet_reader_compat(metadata.__isset.created_by ? metadata.created_by : "")
1494+
.null_count_trusted;
1495+
bool usable = false;
1496+
for (const auto& plan : _scan_plan->row_groups) {
1497+
const auto& group = metadata.row_groups[plan.row_group_id];
1498+
DORIS_CHECK(schema.leaf_column_id >= 0 && schema.leaf_column_id < group.columns.size());
1499+
const auto& chunk = group.columns[schema.leaf_column_id];
1500+
Priority priority;
1501+
if (chunk.__isset.meta_data && chunk.meta_data.__isset.statistics) {
1502+
const auto safe_statistics = detail::sanitize_native_footer_statistics(
1503+
schema.type_descriptor, chunk.meta_data.statistics,
1504+
detail::has_supported_type_defined_order(metadata, schema.leaf_column_id));
1505+
const auto stats = ParquetStatisticsUtils::TransformColumnStatistics(
1506+
schema, &safe_statistics, chunk.meta_data.num_values, _timezone, trusted_nulls);
1507+
if (stats.has_min_max) {
1508+
priority.best = topn->is_asc() ? stats.min_value : stats.max_value;
1509+
usable = true;
1510+
}
1511+
priority.may_have_null = stats.has_null;
1512+
}
1513+
priorities.push_back(std::move(priority));
1514+
}
1515+
if (!usable) {
1516+
return;
1517+
}
1518+
_row_group_order.resize(priorities.size());
1519+
std::iota(_row_group_order.begin(), _row_group_order.end(), 0);
1520+
std::stable_sort(_row_group_order.begin(), _row_group_order.end(), [&](size_t lhs, size_t rhs) {
1521+
const auto& left = priorities[lhs];
1522+
const auto& right = priorities[rhs];
1523+
if (topn->nulls_first() && left.may_have_null != right.may_have_null) {
1524+
return left.may_have_null;
1525+
}
1526+
if (left.best.has_value() != right.best.has_value()) {
1527+
return left.best.has_value();
1528+
}
1529+
if (!left.best.has_value()) {
1530+
return false;
1531+
}
1532+
return topn->is_asc() ? *left.best < *right.best : *right.best < *left.best;
1533+
});
1534+
}
1535+
14121536
Status ParquetScanScheduler::open_next_row_group(
14131537
ParquetFileContext& file_context,
14141538
const std::vector<std::unique_ptr<ParquetColumnSchema>>& file_schema,
14151539
const format::FileScanRequest& request, bool* has_row_group) {
14161540
*has_row_group = false;
14171541
RowGroupReadPlan* selected_plan = nullptr;
14181542
DORIS_CHECK(_scan_plan != nullptr);
1543+
if (!_row_group_order_prepared) {
1544+
prepare_topn_row_group_order(file_context, file_schema, request);
1545+
}
14191546
if (_requested_leaf_ids_need_refresh) {
14201547
// A runtime-filter refresh changes the immutable projection baseline exactly once; later
14211548
// row groups only apply their full-Variant fallback delta to this rebuilt set.
14221549
_scan_plan->requested_leaf_column_ids = request_leaf_column_ids(file_schema, request);
14231550
_requested_leaf_ids_need_refresh = false;
14241551
}
14251552
while (_next_row_group_plan_idx < _scan_plan->row_groups.size()) {
1426-
RowGroupReadPlan& candidate_plan = _scan_plan->row_groups[_next_row_group_plan_idx++];
1553+
const size_t plan_idx = _row_group_order.empty()
1554+
? _next_row_group_plan_idx
1555+
: _row_group_order[_next_row_group_plan_idx];
1556+
++_next_row_group_plan_idx;
1557+
RowGroupReadPlan& candidate_plan = _scan_plan->row_groups[plan_idx];
14271558
// Probe only the row group about to execute. This keeps LIMIT/cancellation latency
14281559
// independent of the number of later remote row groups while preserving eager footer
14291560
// statistics pruning during open.
@@ -1440,6 +1571,11 @@ Status ParquetScanScheduler::open_next_row_group(
14401571
const auto& row_group = file_context.native_metadata->to_thrift()
14411572
.row_groups[candidate_plan.row_group_id];
14421573
prepare_row_group_physical_projection(row_group, file_schema, request, &candidate_plan);
1574+
}
1575+
// TopN changes its bound in place, without publishing a new FileScanRequest. For example,
1576+
// reading [0, 99] first can tighten ASC LIMIT 3 to 2 and exclude a later group [100, 199].
1577+
// Recheck before dictionary/index probes or prefetch, using the current monotonic bound.
1578+
if (_remaining_plans_need_replanning || _topn_footer_recheck) {
14431579
const auto row_group_leaf_ids = row_group_leaf_column_ids(
14441580
file_schema, request, candidate_plan, _scan_plan->requested_leaf_column_ids);
14451581
const auto& requested_leaf_ids = row_group_leaf_ids.has_value()

‎be/src/format_v2/parquet/parquet_scan.h‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -239,6 +239,10 @@ class ParquetScanScheduler {
239239
static constexpr size_t PROFILE_FLUSH_BATCH_INTERVAL = 16;
240240

241241
void reset_current_row_group();
242+
void prepare_topn_row_group_order(
243+
const ParquetFileContext& file_context,
244+
const std::vector<std::unique_ptr<ParquetColumnSchema>>& file_schema,
245+
const format::FileScanRequest& request);
242246
void activate_pending_scan_request_at_row_group_boundary();
243247
void flush_current_reader_profiles();
244248
bool finish_current_reader_batch_profiles();
@@ -287,6 +291,10 @@ class ParquetScanScheduler {
287291

288292
std::shared_ptr<RowGroupScanPlan> _scan_plan; // shared with the reader's aggregate path
289293
size_t _next_row_group_plan_idx = 0; // index of the next row group to process
294+
// Execution order is separate from physical plans: Condition Cache uses their file-row order.
295+
std::vector<size_t> _row_group_order;
296+
bool _row_group_order_prepared = false;
297+
bool _topn_footer_recheck = false;
290298

291299
bool _has_current_row_group = false;
292300
// Readers retain pointers into this immutable row-group map, so it must outlive both maps below.

0 commit comments

Comments
 (0)