Skip to content

Commit a9260ff

Browse files
csun5285claude
andcommitted
[fix](be) Do not push storage TopN to a MOR table read as DUP
### What problem does this PR solve? Issue Number: None Related PR: #62222 Problem Summary: A merge-on-read UNIQUE table listed in read_mor_as_dup_tables is read without merging, so OlapScanLocalState::_storage_no_merge() is true for it. Since #62222 the scanner alone decides whether VCollectIterator takes the storage TopN path, and it pushes read_orderby_key_limit for every key TopN scan where _storage_no_merge() holds. TabletReader::_init_orderby_keys_param() prepares the key columns that VCollectIterator::_topn_next() compares only for DUP_KEYS and merge-on-write UNIQUE_KEYS tablets, so for a MOR table read as DUP read_orderby_key_columns stays null and the DORIS_CHECK at the top of _topn_next() aborts the BE. FE pushes the sort info to the scan for any table whose ORDER BY is a key prefix, so a plain query is enough: -- mor_tbl: UNIQUE KEY(k1, k2), enable_unique_key_merge_on_write = false SET read_mor_as_dup_tables = 'mor_tbl'; SELECT * FROM mor_tbl ORDER BY k1, k2 LIMIT 4; -- ERROR 1105 (HY000): RpcException, msg: send fragments failed. -- be.out: Check failed: _reader->_reader_context.read_orderby_key_columns != nullptr The scanner now keeps segment LIMIT off for a key TopN on a MOR table read as DUP. The scan returns every row and the TopN operator above sorts them, the same as any key TopN that is not pushed to storage. A plain LIMIT on such a table is still pushed to SegmentIterator. TabletReader::_init_orderby_keys_param() now also returns INTERNAL_ERROR when storage TopN is requested but no order-by key columns are set up, for example on a merge-on-read tablet or with an empty key prefix. If the scanner pushes TopN where storage cannot run it, the query fails at reader init instead of aborting the BE in _topn_next(). ### Release note Fix a BE crash on ORDER BY <key> LIMIT queries over a merge-on-read unique table listed in read_mor_as_dup_tables. ### Check List (For Author) - Test: Regression test, Unit Test, Manual test - unique_with_mor_p0/test_read_mor_as_dup_key_topn: aborts the BE before this change, passes after. - unique_with_mor_p0/test_read_mor_as_dup and all 19 suites in query_p0/limit pass. - TabletReaderTest: storage TopN without order-by key columns fails at init. - With the scanner change reverted by hand, the query above fails with this INTERNAL_ERROR and the BE stays up. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 81eeced commit a9260ff

5 files changed

Lines changed: 167 additions & 3 deletions

File tree

‎be/src/exec/scan/olap_scanner.cpp‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -475,6 +475,9 @@ Status OlapScanner::_init_tablet_reader_params(
475475
const bool no_runtime_filters = _total_rf_num == 0;
476476
const bool segment_limit_enabled = _state->enable_segment_limit_pushdown();
477477
const bool storage_no_merge = olap_scan_local_state->_storage_no_merge();
478+
// Storage TopN compares key columns that TabletReader prepares only for DUP and
479+
// merge-on-write tablets, so a MOR table read as DUP cannot use it.
480+
const bool storage_topn_supported = storage_no_merge && !read_mor_as_dup;
478481

479482
if (_limit > 0 && no_runtime_filters && segment_limit_enabled && storage_no_merge) {
480483
for (const auto& conjunct : _conjuncts) {
@@ -486,9 +489,10 @@ Status OlapScanner::_init_tablet_reader_params(
486489

487490
// Segment LIMIT has only two legal states: completely disabled, or enabled after every
488491
// row-filtering conjunct has become a storage predicate or SegmentIterator common expr.
489-
const bool can_push_down_segment_limit = _limit > 0 && no_runtime_filters &&
490-
_conjuncts.empty() && segment_limit_enabled &&
491-
storage_no_merge;
492+
// A key TopN also needs storage TopN support, otherwise nothing is pushed.
493+
const bool can_push_down_segment_limit =
494+
_limit > 0 && no_runtime_filters && _conjuncts.empty() && segment_limit_enabled &&
495+
storage_no_merge && (!has_key_topn || storage_topn_supported);
492496
if (can_push_down_segment_limit) {
493497
if (has_key_topn) {
494498
_tablet_reader_params.read_orderby_key = true;

‎be/src/storage/tablet/tablet_reader.cpp‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,16 @@ Status TabletReader::_init_orderby_keys_param(const ReaderParams& read_params) {
379379
}
380380
}
381381

382+
// Storage TopN sorts rows by these columns, and only DUP and merge-on-write tablets set
383+
// them up. Without them, fail the scan here instead of aborting later in VCollectIterator.
384+
if (read_params.read_orderby_key_limit > 0 && _orderby_key_columns.empty()) {
385+
return Status::Error<ErrorCode::INTERNAL_ERROR>(
386+
"storage TopN needs order-by key columns, but none are set up: keys_type={}, "
387+
"enable_unique_key_merge_on_write={}, read_orderby_key_num_prefix_columns={}",
388+
_tablet_schema->keys_type(), _tablet->enable_unique_key_merge_on_write(),
389+
read_params.read_orderby_key_num_prefix_columns);
390+
}
391+
382392
return Status::OK();
383393
}
384394

‎be/test/storage/tablet_reader_test.cpp‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@
3131
#include "storage/delete/delete_handler.h"
3232
#include "storage/rowset/rowset_meta.h"
3333
#include "storage/schema.h"
34+
#include "storage/storage_engine.h"
35+
#include "storage/tablet/tablet.h"
36+
#include "storage/tablet/tablet_meta.h"
3437
#include "storage/tablet/tablet_schema.h"
3538

3639
namespace doris {
@@ -70,6 +73,25 @@ class TabletReaderTest : public testing::Test {
7073
ASSERT_TRUE(handler.init({rs_meta}, /*version=*/100, *read_schema, &dropped_columns).ok());
7174
(*read_schema)->append_dropped_columns(std::move(dropped_columns));
7275
}
76+
77+
// Give `reader` a tablet with the given keys type. Its only key column is k1.
78+
static void init_tablet(StorageEngine& engine, TabletReader& reader, KeysType keys_type,
79+
bool enable_mow) {
80+
auto tablet_meta = std::make_shared<TabletMeta>();
81+
tablet_meta->set_enable_unique_key_merge_on_write(enable_mow);
82+
reader._tablet = std::make_shared<Tablet>(engine, std::move(tablet_meta), nullptr);
83+
reader._tablet_schema = create_schema({{"k1", 10}, {"v1", 12}}, keys_type);
84+
reader._read_schema = std::make_shared<ReadSchema>(reader._tablet_schema->columns());
85+
}
86+
87+
// Set up the order-by key columns for a storage TopN that sorts by the first
88+
// `num_prefix_columns` key columns.
89+
static Status init_storage_topn(TabletReader& reader, size_t num_prefix_columns) {
90+
TabletReader::ReaderParams params;
91+
params.read_orderby_key_num_prefix_columns = num_prefix_columns;
92+
params.read_orderby_key_limit = 10;
93+
return reader._init_orderby_keys_param(params);
94+
}
7395
};
7496

7597
// The delete columns (resolved by the delete handler to read-schema ordinals) are mapped to their
@@ -211,4 +233,43 @@ TEST_F(TabletReaderTest, delete_predicate_keeps_dropped_same_name_column_identit
211233
EXPECT_EQ(1, delete_zone_maps.count(2));
212234
}
213235
}
236+
237+
// DUP and merge-on-write tablets give storage TopN the leading key columns to sort by.
238+
TEST_F(TabletReaderTest, storage_topn_sorts_by_leading_key_columns) {
239+
StorageEngine engine({});
240+
TabletReader dup_reader;
241+
init_tablet(engine, dup_reader, KeysType::DUP_KEYS, /*enable_mow=*/false);
242+
auto st = init_storage_topn(dup_reader, 1);
243+
ASSERT_TRUE(st.ok()) << st;
244+
EXPECT_EQ(std::vector<uint32_t>({0}), dup_reader._orderby_key_columns);
245+
246+
TabletReader mow_reader;
247+
init_tablet(engine, mow_reader, KeysType::UNIQUE_KEYS, /*enable_mow=*/true);
248+
st = init_storage_topn(mow_reader, 1);
249+
ASSERT_TRUE(st.ok()) << st;
250+
EXPECT_EQ(std::vector<uint32_t>({0}), mow_reader._orderby_key_columns);
251+
}
252+
253+
// Storage TopN without order-by key columns fails at init, instead of aborting the BE
254+
// later in VCollectIterator.
255+
TEST_F(TabletReaderTest, storage_topn_without_orderby_key_columns_fails) {
256+
StorageEngine engine({});
257+
// A merge-on-read tablet, such as one read as DUP, sets up no order-by key columns.
258+
TabletReader mor_reader;
259+
init_tablet(engine, mor_reader, KeysType::UNIQUE_KEYS, /*enable_mow=*/false);
260+
auto st = init_storage_topn(mor_reader, 1);
261+
EXPECT_TRUE(st.is<ErrorCode::INTERNAL_ERROR>()) << st;
262+
263+
// An empty key prefix leaves nothing to sort by.
264+
TabletReader dup_reader;
265+
init_tablet(engine, dup_reader, KeysType::DUP_KEYS, /*enable_mow=*/false);
266+
st = init_storage_topn(dup_reader, 0);
267+
EXPECT_TRUE(st.is<ErrorCode::INTERNAL_ERROR>()) << st;
268+
269+
// Without TopN, a merge-on-read tablet needs no order-by key columns.
270+
TabletReader plain_reader;
271+
init_tablet(engine, plain_reader, KeysType::UNIQUE_KEYS, /*enable_mow=*/false);
272+
st = plain_reader._init_orderby_keys_param(TabletReader::ReaderParams {});
273+
EXPECT_TRUE(st.ok()) << st;
274+
}
214275
} // namespace doris
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !merged --
3+
1 1 12
4+
2 2 20
5+
3 3 30
6+
7+
-- !dup_asc --
8+
1 1 10
9+
1 1 11
10+
1 1 12
11+
2 2 20
12+
13+
-- !dup_desc --
14+
7 7 70
15+
6 6 60
16+
5 5 50
17+
18+
-- !dup_prefix --
19+
1 1 10
20+
1 1 11
21+
1 1 12
22+
2 2 20
23+
24+
-- !dup_filter --
25+
1 1 11
26+
1 1 12
27+
2 2 20
28+
29+
-- !dup_offset --
30+
2 2 20
31+
3 3 30
32+
33+
-- !dup_limit --
34+
5
35+
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
// ORDER BY a key prefix with LIMIT on a MOR table listed in read_mor_as_dup_tables.
19+
// The table is read as DUP, so every version of a key is a row and the TopN sees all of them.
20+
suite("test_read_mor_as_dup_key_topn") {
21+
sql "DROP TABLE IF EXISTS test_read_mor_as_dup_key_topn"
22+
sql """
23+
CREATE TABLE test_read_mor_as_dup_key_topn (
24+
k1 INT NOT NULL,
25+
k2 INT NOT NULL,
26+
v INT NULL
27+
) ENGINE=OLAP
28+
UNIQUE KEY(k1, k2)
29+
DISTRIBUTED BY HASH(k1) BUCKETS 1
30+
PROPERTIES (
31+
"enable_unique_key_merge_on_write" = "false",
32+
"disable_auto_compaction" = "true",
33+
"replication_num" = "1"
34+
)
35+
"""
36+
// Three loads make three overlapping rowsets; key (1, 1) has three versions.
37+
sql "INSERT INTO test_read_mor_as_dup_key_topn VALUES (1, 1, 10), (2, 2, 20), (5, 5, 50)"
38+
sql "INSERT INTO test_read_mor_as_dup_key_topn VALUES (1, 1, 11), (3, 3, 30), (6, 6, 60)"
39+
sql "INSERT INTO test_read_mor_as_dup_key_topn VALUES (1, 1, 12), (4, 4, 40), (7, 7, 70)"
40+
41+
// Merged read: one row per key.
42+
qt_merged "SELECT * FROM test_read_mor_as_dup_key_topn ORDER BY k1, k2 LIMIT 3"
43+
44+
sql "SET read_mor_as_dup_tables = 'test_read_mor_as_dup_key_topn'"
45+
// The crash needs segment LIMIT pushdown, which fuzzy sessions may turn off.
46+
sql "SET enable_segment_limit_pushdown = true"
47+
order_qt_dup_asc "SELECT * FROM test_read_mor_as_dup_key_topn ORDER BY k1, k2 LIMIT 4"
48+
qt_dup_desc "SELECT * FROM test_read_mor_as_dup_key_topn ORDER BY k1 DESC, k2 DESC LIMIT 3"
49+
order_qt_dup_prefix "SELECT * FROM test_read_mor_as_dup_key_topn ORDER BY k1 LIMIT 4"
50+
order_qt_dup_filter "SELECT * FROM test_read_mor_as_dup_key_topn WHERE v > 10 ORDER BY k1, k2 LIMIT 3"
51+
qt_dup_offset "SELECT * FROM test_read_mor_as_dup_key_topn ORDER BY k1, k2 LIMIT 2 OFFSET 3"
52+
// A plain LIMIT without ORDER BY still works on the table read as DUP.
53+
order_qt_dup_limit "SELECT count(*) FROM (SELECT * FROM test_read_mor_as_dup_key_topn LIMIT 5) t"
54+
}

0 commit comments

Comments
 (0)