Skip to content

Commit 40e1463

Browse files
authored
[fix](be) Validate SNII count fast-path document frequency (apache#66877)
### What problem does this PR solve? Issue Number: None Related PR: apache#66052 Problem Summary: SNII's count-only shortcut trusted the persisted dictionary document frequency without checking it against Core document statistics. A corrupted 10-row segment whose posting contains two documents but whose dictionary entry records df=100 could fabricate a 100-id bitmap in both nullable and non-nullable paths, producing an inflated count after segment masking. Validate df against both total and indexed document counts before fabricating the count-shaped bitmap and report the cross-section mismatch as index corruption. Add CRC-valid nullable and non-nullable fixtures that rewrite only df while preserving real two-document postings. ### Release note Fix SNII count-only queries to reject corrupted document frequencies outside the segment document domain.
1 parent ab529c9 commit 40e1463

7 files changed

Lines changed: 400 additions & 33 deletions

File tree

‎be/src/storage/index/snii/query/count_query.h‎

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -47,22 +47,27 @@ Status count_only_term_df(const reader::LogicalIndexReader& idx, std::string_vie
4747
// `count` row ids DISJOINT from `nulls` (the first `count` non-null row ids,
4848
// all < count + |nulls|). Why disjoint: the MATCH machinery unconditionally
4949
// subtracts the segment null bitmap from every index result
50-
// (FunctionMatchBase -> InvertedIndexResultBitmap::mask_out_null). Real
51-
// postings never contain null docs -- the writer adds NO tokens for a null doc
52-
// (scalar add_nulls) and a NULL array row is stored as an empty range (zero
53-
// tokens) -- so that subtraction is a no-op on true results and df already IS
54-
// the exact match count regardless of nulls. A naive [0, df) range however MAY
55-
// collide with null row ids and be shrunk by mask_out_null; picking the ids
56-
// from the non-null space makes the subtraction provably a no-op, preserving
57-
// cardinality == df end to end.
50+
// (FunctionMatchBase -> InvertedIndexResultBitmap::mask_out_null). For the
51+
// callers allowed to fabricate, postings never contain null docs -- the writer
52+
// adds NO tokens for a null doc (scalar add_nulls) -- so that subtraction is a
53+
// no-op on true results and df already IS the exact match count regardless of
54+
// nulls. That does NOT hold for ARRAY columns, whose add_array_values() also
55+
// indexes the nested payload of outer-null rows and whose add_array_nulls()
56+
// never retracts it; SniiIndexReader::_try_count_only_fastpath therefore
57+
// refuses an ARRAY column on a segment with nulls before reaching here. A naive
58+
// [0, df) range MAY collide with null row ids and be shrunk by mask_out_null;
59+
// picking the ids from the non-null space makes the subtraction provably a
60+
// no-op, preserving cardinality == df end to end.
5861
//
5962
// The window bound is doc-count-free: count counts only non-null docs, so
6063
// count + |nulls| <= segment doc count and [0, count + |nulls|) always holds
61-
// >= count non-null ids; every fabricated id therefore stays inside the
62-
// segment's [0, num_rows) row space. Errors (id space would exceed the uint32
63-
// docid domain, or the window unexpectedly holds fewer than `count` survivors)
64-
// only occur on a corrupt index; callers treat them as "fall through to the
65-
// decode path", never as a fabricated answer.
64+
// >= count non-null ids. Whether that lands inside the segment's real
65+
// [0, num_rows) row space rests on doc count <= num_rows, which the caller
66+
// checks against the segment itself -- not on anything this function can see.
67+
// Errors (id space would exceed the uint32 docid domain, or the window
68+
// unexpectedly holds fewer than `count` survivors) only occur on a corrupt
69+
// index; callers treat them as "fall through to the decode path", never as a
70+
// fabricated answer.
6671
Status fabricate_null_disjoint_count_bitmap(uint64_t count, const roaring::Roaring& nulls,
6772
roaring::Roaring* out);
6873

‎be/src/storage/index/snii/snii_index_reader.cpp‎

Lines changed: 54 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1054,6 +1054,8 @@ Status SniiIndexReader::_compute_query_bitmap(const IndexQueryContextPtr& contex
10541054
}
10551055
#endif
10561056

1057+
// Keep the complete count-only eligibility and null-safe fabrication contract in one linear path.
1058+
// NOLINTNEXTLINE(readability-function-size)
10571059
Status SniiIndexReader::_try_count_only_fastpath(
10581060
const IndexQueryContextPtr& context, InvertedIndexQueryType query_type,
10591061
const InvertedIndexQueryInfo& query_info, const std::vector<std::string>& terms,
@@ -1088,6 +1090,30 @@ Status SniiIndexReader::_try_count_only_fastpath(
10881090
&logical_reader));
10891091
}
10901092

1093+
// ARRAY columns: df is NOT null-free, so nothing below may fabricate from it.
1094+
// ArrayColumnWriter::append_nullable hands add_array_values() every row of the
1095+
// batch -- the offsets come from the nested ColumnArray, and
1096+
// OlapColumnDataConvertorArray::convert_to_olap reads them without ever
1097+
// consulting the outer null map -- and the add_array_nulls() that follows only
1098+
// RECORDS the null row ids, it never retracts the tokens already emitted for
1099+
// them. A nullable array whose nested payload survives under the null map does
1100+
// occur: PreparedFunctionImpl::default_implementation_for_nulls documents that
1101+
// nested columns keep "arbitrary values in rows corresponding to NULL value",
1102+
// and need_replace_null_data_to_default() is false by default, so e.g.
1103+
// array_concat(arr, nullable_arr) writes arr's tokens on a NULL row. That row
1104+
// then sits in a posting and is counted by df. The decode path stays correct --
1105+
// mask_out_null subtracts it -- but the fabrication below deliberately places
1106+
// its ids OFF the null rows, so the same subtraction removes nothing and the
1107+
// count comes out too high. CLucene writes arrays identically
1108+
// (InvertedIndexColumnWriter::add_array_nulls only touches _null_bitmap), so
1109+
// this cannot be repaired from the reader side; decline whenever this segment
1110+
// has a null bitmap at all. Scalars are unaffected: ScalarColumnWriter::
1111+
// append_nullable splits the batch into runs and sends null runs to
1112+
// append_nulls(), which emits no tokens.
1113+
if (_column_is_array && logical_reader->section_refs().null_bitmap.length > 0) {
1114+
return Status::OK();
1115+
}
1116+
10911117
std::string physical_term_scratch;
10921118
std::string_view physical_term;
10931119
bool representable = false;
@@ -1100,11 +1126,35 @@ Status SniiIndexReader::_try_count_only_fastpath(
11001126
RETURN_IF_ERROR(
11011127
::doris::snii::query::count_only_term_df(*logical_reader, physical_term, &count));
11021128
}
1129+
// df bounds the fabricated bitmap, so it has to be inside a document domain
1130+
// that is itself real. Two steps, because they fail differently.
1131+
const auto& stats = logical_reader->stats();
1132+
if (count > stats.doc_count || count > stats.indexed_doc_count) {
1133+
return Status::Error<ErrorCode::INVERTED_INDEX_FILE_CORRUPTED, false>(
1134+
"SNII count fast path: term df {} exceeds document domain (doc count {}, "
1135+
"indexed doc count {})",
1136+
count, stats.doc_count, stats.indexed_doc_count);
1137+
}
1138+
// Both limits above are CRC-valid fields of the SAME image as df, so an image
1139+
// whose stats were inflated together with df clears them: on a real 10-row
1140+
// segment, df = doc_count = indexed_doc_count = 100 fabricates 100 ids, and
1141+
// SegmentIterator -- which seeds _row_bitmap with [0, num_rows) and intersects
1142+
// -- silently reports 10. The segment's own row count is the one bound the
1143+
// image cannot move. One-sided on purpose: an index covering FEWER rows than
1144+
// the segment still fabricates ids inside [0, num_rows), so only an oversized
1145+
// domain is corruption. This mirrors the equality SniiSegmentReader::
1146+
// load_inherited_index already demands of a rewrite.
1147+
if (stats.doc_count > _rows_of_segment) {
1148+
return Status::Error<ErrorCode::INVERTED_INDEX_FILE_CORRUPTED, false>(
1149+
"SNII count fast path: index doc count {} exceeds the segment row count {}",
1150+
stats.doc_count, _rows_of_segment);
1151+
}
11031152

1104-
// Null handling. df is the exact match count REGARDLESS of nulls: the
1105-
// writer adds no tokens for a null doc (scalar add_nulls; a NULL array row
1106-
// is an empty range), so postings -- and therefore df -- never include
1107-
// null rows, exactly matching MATCH's "null never matches" semantics. The
1153+
// Null handling. df is the exact match count REGARDLESS of nulls: the writer
1154+
// adds no tokens for a null doc (scalar add_nulls; ARRAY columns cannot reach
1155+
// this point on a segment with nulls, see the guard above), so postings -- and
1156+
// therefore df -- never include null rows, exactly matching MATCH's "null never
1157+
// matches" semantics. The
11081158
// fabricated bitmap however flows through FunctionMatchBase ->
11091159
// InvertedIndexResultBitmap::mask_out_null, which subtracts the segment's
11101160
// REAL null bitmap from it; a dense [0, df) range colliding with null row

‎be/src/storage/index/snii/snii_index_reader.h‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,10 +69,21 @@ class SniiIndexReader final : public InvertedIndexReader {
6969
using SearcherOpenObserver = void (*)(void*) noexcept;
7070
#endif
7171

72+
// `rows_of_segment` and `column_is_array` describe the SEGMENT and the COLUMN,
73+
// deliberately not read back out of the index image: the count-only fast path
74+
// fabricates row ids, so it needs at least one bound a corrupt (but CRC-valid)
75+
// image cannot move, and one fact about how the column was written. Both are
76+
// already on hand where readers are built -- ColumnReader::_load_index passes
77+
// the same rows_of_segment to AnnIndexReader and already tests _meta_type for
78+
// OLAP_FIELD_TYPE_ARRAY a few lines above.
7279
SniiIndexReader(const TabletIndex* index_meta,
7380
const std::shared_ptr<IndexFileReader>& index_file_reader,
74-
InvertedIndexReaderType reader_type)
75-
: InvertedIndexReader(index_meta, index_file_reader), _reader_type(reader_type) {}
81+
InvertedIndexReaderType reader_type, uint64_t rows_of_segment,
82+
bool column_is_array)
83+
: InvertedIndexReader(index_meta, index_file_reader),
84+
_reader_type(reader_type),
85+
_rows_of_segment(rows_of_segment),
86+
_column_is_array(column_is_array) {}
7687

7788
Status new_iterator(std::unique_ptr<IndexIterator>* iterator) override;
7889
Status query(const IndexQueryContextPtr& context, const std::string& column_name,
@@ -155,8 +166,12 @@ class SniiIndexReader final : public InvertedIndexReader {
155166
// match count, because postings never contain null docs. Falls through
156167
// (*handled = false) for every other shape: every multi-term query
157168
// (including phrase and OR/AND), prefix/regexp/wildcard/phrase-prefix
158-
// expansion. Multi-term sloppy phrases fall through with every other
159-
// multi-term shape; a single-term phrase remains exactly one posting df.
169+
// expansion, and an ARRAY column on a segment that has nulls, whose df is
170+
// NOT null-free (see the guard in the .cpp). Multi-term sloppy phrases fall
171+
// through with every other multi-term shape; a single-term phrase remains
172+
// exactly one posting df. Rejects the index outright, with
173+
// INVERTED_INDEX_FILE_CORRUPTED, when df or the index's document domain
174+
// falls outside the segment's real row space.
160175
// On *handled = true, query() also raises
161176
// context->count_on_index_fastpath_hit (G03) so the SegmentIterator may
162177
// short-circuit row emission for the count-shaped bitmap.
@@ -167,6 +182,13 @@ class SniiIndexReader final : public InvertedIndexReader {
167182
const ::doris::snii::reader::LogicalIndexReader* preopened_reader = nullptr);
168183

169184
InvertedIndexReaderType _reader_type;
185+
// Row count of the segment this reader belongs to, straight from
186+
// Segment::_num_rows. The count-only fast path bounds the index's own
187+
// document domain against it; see _try_count_only_fastpath.
188+
uint64_t _rows_of_segment = 0;
189+
// True when the indexed column is an ARRAY. Disqualifies the count-only fast
190+
// path on a segment that has nulls; see _try_count_only_fastpath.
191+
bool _column_is_array = false;
170192
#ifdef BE_TEST
171193
SingleFlightFollowerJoinedObserver _single_flight_follower_joined_observer = nullptr;
172194
void* _single_flight_follower_joined_opaque = nullptr;

‎be/src/storage/segment/column_reader.cpp‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -730,8 +730,9 @@ Status ColumnReader::_load_index(const std::shared_ptr<IndexFileReader>& index_f
730730
if (is_string_type(type)) {
731731
auto reader_type = should_analyzer ? InvertedIndexReaderType::FULLTEXT
732732
: InvertedIndexReaderType::STRING_TYPE;
733-
index_reader =
734-
SniiIndexReader::create_shared(index_meta, index_file_reader, reader_type);
733+
index_reader = SniiIndexReader::create_shared(
734+
index_meta, index_file_reader, reader_type, rows_of_segment,
735+
_meta_type == FieldType::OLAP_FIELD_TYPE_ARRAY);
735736
} else if (field_is_numeric_type(type)) {
736737
index_reader = SniiBkdIndexReader::create_shared(index_meta, index_file_reader);
737738
} else {

‎be/test/storage/index/inverted/inverted_index_reader_analysis_purpose_test.cpp‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -290,8 +290,12 @@ class InvertedIndexReaderAnalysisPurposeTest : public testing::Test {
290290
_snii_file_reader = std::make_shared<IndexFileReader>(
291291
io::global_local_filesystem(), "./ut_dir/missing_snii_analysis_purpose",
292292
InvertedIndexStorageFormatPB::SNII);
293+
// The file does not exist -- these cases only exercise the analysis-purpose
294+
// router, never the count fast path -- so the segment shape is nominal.
293295
_snii_reader = SniiIndexReader::create_shared(&_meta, _snii_file_reader,
294-
InvertedIndexReaderType::FULLTEXT);
296+
InvertedIndexReaderType::FULLTEXT,
297+
/*rows_of_segment=*/0,
298+
/*column_is_array=*/false);
295299
}
296300

297301
void TearDown() override {

‎be/test/storage/index/snii/snii_doris_adapter_test.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -390,7 +390,9 @@ class SniiIndexReaderActualPathTest : public testing::Test {
390390
_file_reader->_snii_segment_reader = std::move(segment_reader);
391391
_file_reader->_inited = true;
392392
_index_reader = SniiIndexReader::create_shared(&_meta, _file_reader,
393-
InvertedIndexReaderType::FULLTEXT);
393+
InvertedIndexReaderType::FULLTEXT,
394+
/*rows_of_segment=*/kDocCount,
395+
/*column_is_array=*/false);
394396

395397
_previous_query_cache = ExecEnv::GetInstance()->get_inverted_index_query_cache();
396398
_query_cache.reset(InvertedIndexQueryCache::create_global_cache(1024 * 1024, 1));

0 commit comments

Comments
 (0)