[fix](inverted index) Count only visible rows in BM25 statistics of merge-on-write tables by guptashivam04 · Pull Request #69438 · apache/doris

guptashivam04 · 2026-10-22T06:46:07Z

What problem does this PR solve?

Issue Number: None

Related PR: #69352 (BM25 statistics shared across the parallel scanners of a tablet; this PR is its follow-up for replicas), #69313 (BM25 token totals after CLucene compaction)

Problem Summary:

On a merge-on-write table, BM25 score() gives the same row a different score on each replica of its tablet. A query picks one replica per tablet, so with replication_num = 3 a row has up to three scores. Pagination over ORDER BY score() then repeats and skips rows from one page to the next. Each replica is stable on its own (use_fix_replica pins the scores), but the replicas disagree.

CollectionStatistics::process_segment() takes the document count, the token count and each term's document frequency straight from every segment's inverted index. Under merge-on-write, a row overwritten or deleted by a later load stays in its segment's index, marked only in the delete bitmap. A row written with the delete sign, which CDC deletes and DELETE produce, stays as a normal row that queries filter out by predicate. Both kinds are counted until compaction removes them, and each replica compacts on its own schedule. So each replica counts a different set of rows nobody can see.

Reproduction (3 BEs, replication_num = 3, 8 buckets, auto compaction off): 8 loads, 3 rounds of overwriting a quarter of the rows, a DELETE, and full rows inserted with __DORIS_DELETE_SIGN__ = 1. Then a full compaction (POST /api/compaction/run?compact_type=full) of the tablets on one BE only. On master with #69352, the query below changed the score of all 3,799 matching rows from run to run: 1,900 distinct scores instead of 950. With use_fix_replica = 0/1/2 each replica was stable, but the three disagreed.

SELECT id, score() FROM rt WHERE SEARCH('content:jobs OR content:try') ORDER BY score() DESC LIMIT 100000;

The token total has a second, replica-independent problem: CLucene index compaction (inverted_index_compaction_enable, on by default) copies the source token totals even when it drops rows, which #69313 addresses. On master, the same rows scored differently in a compacted table and in a table freshly loaded with its visible rows.

Fix:

When the scan reads a delete bitmap (merge-on-write), CollectionStatistics describes only the rows the scan returns:

OlapScanner passes the scan's delete bitmap and read version (MowHiddenRows) to collect(), or to TabletCollectionStatistics::copy_to() for a split tablet. Every scanner of a tablet reads with the same bitmap and version, so the statistics shared by [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352 stay correct.
For each segment, the hidden rows are the rows the delete bitmap marks at the read version (get_agg, the same call the rowset reader uses), plus the rows whose hidden __DORIS_DELETE_SIGN__ column is set. Every unique-key table has that column, and DELETE on a merge-on-write table writes such rows unless enable_mow_light_delete is on. The delete-sign column is read only for segments whose zone map shows a non-zero sign (or cannot tell), so a segment without CDC deletes costs nothing extra. The read follows the query's enable_segment_cache and enable_page_cache.
Document count: the index's document count minus the hidden rows.
Term document frequency: IndexSource::scoring_doc_freq_among() counts how many hidden rows a term's postings list. When the hidden rows are much rarer than the term's postings, it leaps between them with seek_block. Otherwise it decodes the postings once and walks the hidden rows alongside them, with no allocation, and a term without postings costs nothing. SNII overrides it so that reserved-namespace scoring terms resolve as scoring_doc_freq resolves them.
Token count: the sum of the document lengths the norms of the returned non-NULL rows stand for, or the returned row count for an index without norms. The exact length of a hidden row is not stored anywhere: CLucene and SNII keep only the encoded norm (exact up to 40 tokens for CLucene, up to 254 for SNII). Subtracting decoded lengths from the exact sumTotalTermFreq would still leave replicas apart. A row's norm is the same in every segment it is written to, so summing norms gives every replica the same count. It is also the same quantity the BM25 tf normalization already divides, dl / avgdl, with dl taken from the same norm.
A NULL row has no tokens, but its norm stands for one (CLucene's default norm, SNII's encode_norm(0)). Its length is left out, using the index's null bitmap, and it still counts as a document, as before. So on CLucene a table whose texts all have at most 40 tokens, and none of which is an empty string, scores exactly as a duplicate-key table with the same rows.
The statistics read CLucene norms into a buffer of their own, one segment at a time, and keep none. Asking a CLucene reader for a field's norms makes it keep them for the searcher's lifetime, uncharged to the searcher cache, and the statistics visit every segment of the read set, including those the scan prunes and never scores. Reading per segment also avoids the copy of all segments' norms a multi-segment reader would keep.
What the statistics keep instead is small and kept with the cached searcher: the sum of a field's non-NULL norm lengths, and the sum of the hidden rows' lengths for the last few sets of hidden rows of a field (queries at one read version hide the same rows). So queries at an unchanged version read no norms; a segment's norms are read again only when its hidden rows change. The hidden and NULL row ID lists the statistics build while collecting are charged to the query's memory.

Why the token count has to change too: a variant that left out hidden rows only from the document count and the document frequencies, keeping today's token total, made the replicas agree only while every replica compacted through inverted index compaction, which carries dropped rows' token counts forward. With one replica compacted with inverted_index_compaction_enable = false (the index rebuilt from column data), every row again scored differently on that replica. Its scores were also 12–31 % off a fresh load of the visible rows on average, because its token total still counted the hidden rows.

A tablet whose rows are all hidden (every row deleted or overwritten, not yet compacted) has no document a query can match. Its statistics report zero documents instead of failing the query with "No data available for SimilarityCollector".

Tables without a delete bitmap (duplicate key, merge-on-read, or skip_delete_bitmap) collect the same statistics as before.

For every score() query, with or without a delete bitmap:

The statistics reads follow the query's disable_file_cache and inverted_index_snii_read_no_write_file_cache, as the scan's reads do. They run before the scan, so they are the reads that fill the file cache. CLucene reads norms without a per-read context, as scoring does.
A CLucene searcher the statistics open drops the query's I/O context once its meta is read, as the query path already does (create_index_searcher), and a CLucene error while opening it fails the query; before, it escaped the scanner thread uncaught and aborted the backend. Before, a searcher the statistics put into the searcher cache kept the opening query's I/O context, including its file cache statistics pointer, for later queries to read through.

Not covered:

Merge-on-read unique tables and duplicate tables with DELETE predicates also count rows that compaction later drops. They need merge or predicate evaluation to tell which rows are visible, and are left for a follow-up.
With skip_delete_sign = true or show_hidden_columns = true, rows written with the delete sign are returned but still left out of the statistics. Rows deleted by a delete predicate (enable_mow_light_delete) are still counted, as the delete bitmap does not mark them.
A row indexed without tokens that is not NULL keeps a norm that cannot be told from a one-token row's, so on a merge-on-write table it counts as one token: an analyzed empty string, which both formats index like a NULL but leave out of the null bitmap, and on SNII any text that yields no token (for example '!!!', or text without ASCII letters or digits under the english parser), whose norm SNII stores as that of one token. On CLucene such a text has a zero norm and counts as zero tokens. Telling these rows apart would mean reading the column.
On SNII, the sum of the norm lengths is not cached yet: each merge-on-write query reads the segment's norms.
Leaving NULL rows out relies on the index's null bitmap, which CLucene index compaction carries over; if reading a source bitmap fails there, the compacted index loses it (existing behaviour of the vendored CLucene compaction).
Statistics stay per tablet, as before.

Interaction with #69313: that PR makes the token total of a CLucene index built by compaction exact. With this PR, merge-on-write scans no longer read that total, so their scores do not depend on it. Scans of other tables still do. The two changes are independent.

Release note

Fix BM25 score() on merge-on-write tables differing between the replicas of a tablet, and before and after compaction. Statistics now count only rows visible to the query, and the average document length comes from the indexed norms.

Check List (For Author)

Test

Regression test: inverted_index_p0/test_bm25_score_replica_statistics. It loads overwrites, DELETEs and delete-sign rows, then expects every row to score exactly as in a table freshly loaded with the visible rows. It checks before compaction, with one replica compacted (≥ 3 BEs: 10 runs and use_fix_replica = 0/1/2), and after compacting all replicas (the compaction steps are skipped in cloud mode, which keeps one shared copy of each tablet). It also checks that a merge-on-write table with NULL rows scores exactly as a duplicate-key table with the same rows, and that a bucket whose rows are all deleted returns no rows instead of failing. Without this change it fails before any compaction (1352 rows score differently from a freshly loaded table); with only the first version of this change it failed the NULL check (250 rows score differently). With it, it passes on a 3-BE cluster.
Regression test: fault_injection_p0/test_bm25_score_statistics_searcher_io_context. With the CSIndexInput.readInternal debug point, which fails a postings read that still reads as index meta, a cold score() query on a merge-on-write table failed before the searcher reset ("The 'is_index_data' flag should be false for non-meta inverted index files") and passes with it.
Existing suites: all 23 suites that use score() (in inverted_index_p0, search, ann_index_p0, variant_p0 and nereids_rules_p0) pass on a 3-BE cluster. One expected value changes: in test_bm25_score, table t2 is a merge-on-write table whose row a = 3 was updated, so the old version of that row no longer counts. Two documents count instead of three, and the score of hello world goes from 1.0925692 to 0.8025915.
Unit Test: CollectionStatisticsTest.HiddenRowsAreCollectedLikeCompactedAway (real CLucene segments: a segment with a hidden 45-token row collects the same doc count, token count, df, avgdl and idf as the segment without the row), CollectionStatisticsTest.AllRowsHiddenCollectsEmptyStatistics, CollectionStatisticsTest.NullRowsCountAsDocumentsWithoutTokens (real CLucene segment with NULL rows: the same doc count and token count with and without the delete bitmap, and with a hidden NULL row), CollectionStatisticsHiddenRowsTest.* (counting hidden rows in postings by scanning and by leaping, over sparse and dense blocks; norm sums with and without NULL rows; rows beyond the index; indexes without norms), CluceneSearcherCacheValueTest.NonNullNormLengthSumIsComputedOncePerField and OlapScannerTest.BuildScoreRuntimeCollectionIoContextFollowsQueryFileCacheOptions. Run locally on macOS (ASAN build): all 243 BM25, collection-statistics, scanner, searcher cache, inverted index reader and null bitmap tests pass. A wider set of 2,732 inverted index, SNII, BM25, search and scan tests also passes, except 6 that fail on macOS in code this PR does not touch (5 SNII writer memory-accounting formulas and 1 ScoreRuntimeTest death-test message).

Manual test: master with [fix](inverted index) Share BM25 statistics across parallel scanners of a tablet #69352, with and without this change, macOS arm64, 3 BEs on one host, the reproduction above. use_fix_replica pins one replica; "random" lets the FE pick a replica per tablet per query.

State	without this change	with this change
replicas identical	0 of 3,799 rows change across 10 runs, 950 scores	0 rows change, 950 scores
full compaction on one replica, random replica, 10 runs	3,799 of 3,799 rows change, 1,900 scores	0 rows change, 950 scores
same, `use_fix_replica = 0/1/2` compared	every row differs between replicas	identical
all replicas compacted	stable, but differs from a fresh load of the same rows on every row	identical to the one-replica-compacted state and to a fresh load, bit for bit

Same result for a 2-replica table with BUCKETS AUTO, the unicode parser with lower_case, 4 indexed text columns queried by SEARCH('a:x OR b:x OR c:x OR d:x') plus a non-index filter, auto compaction on, updates by INSERT INTO ... SELECT and deletes by DELETE FROM. Before: every row differed between the replicas (e.g. 3.30 vs 2.69). After: identical on both and equal to a fresh load. The same holds with 1–10 MB text values.

Each of these also scored identically on every replica (one replica fully compacted) and matched a fresh load of its visible rows: partial-column updates, a sequence column, stream loads with __DORIS_DELETE_SIGN__ and with merge_type: DELETE, group commit, keyword and lowercase-normalizer indexes, a range-partitioned table, the SNII storage format, a variant column, MATCH_ANY / MATCH_ALL / MATCH_PHRASE, and a tablet split across scanners (enable_parallel_scan and optimize_index_scan_parallelism). A 90-second run of concurrent upserts, deletes, compactions and score queries finished without errors and left the replicas identical.

Overhead, measured from query profiles with a 4-field SEARCH(... OR ...) and a non-index filter. Each number is the median of 15 runs on the same data and query:

Table	without this change	with this change
11 k rows, 3 replicas	27 ms	15 ms (within noise)
1.75 M visible rows, 1.3 M hidden (43 % of stored rows), 8 buckets	35 ms	36 ms

On one 8 M-row tablet (20 % NULL, 80 k overwritten rows), StatisticsCollectTime went from 0.6 ms to 4.7 ms per query (median of 9), and the query from 27 ms to 22 ms (noise). The first query on a segment also sums its norms (15 ms here).

The cost grows with the rows compaction has not removed yet. To count the hidden rows that hold a term, the collector leaps from hidden row to hidden row when they are much rarer than the term's postings (32 times rarer). Otherwise it decodes the postings once and walks the hidden rows alongside. The norms of a segment's hidden rows are read once per set of hidden rows.

No need to test or manual test.

Behavior changed:

Yes. On merge-on-write tables, BM25 statistics leave out rows hidden by the delete bitmap or the delete sign, and avgdl is computed from the norms of the visible rows. Scores no longer depend on the replica or on compaction. They also shift once on tables without deletes, because norms store long lengths only approximately. Measured on freshly loaded tables:

Texts	Mean score change	Largest change	Row pairs that swap order	Top 100
short to medium (up to ~45 words)	0.27 %	0.59 %	0.021 %	unchanged
long (~380 words on average)	1.75 %	2.6 %	0.085 %, all near-ties	1 near-tie at the boundary
with 19 documents of 1–10 MB	1.47 %	6.2 %	0.11 %	unchanged

Does this need documentation?
- No.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…erge-on-write tables A merge-on-write segment keeps rows that a later load overwrote or deleted, marked only in the delete bitmap, and rows written with the delete sign, which queries filter out by predicate. BM25 statistics counted both until compaction removed them, and each replica compacts on its own schedule, so the replicas of a tablet scored the same row differently and a query that picked a different replica returned different scores and orders. When the scan reads a delete bitmap, collect statistics over the rows it returns only: per segment, leave the rows marked at the read version and the rows with the delete sign out of the document count and of each term's document frequency, counting the hidden rows a term's postings hold by leaping between them. Exact token counts of hidden rows are not stored, and norms hold lengths only approximately, so take the token count as the sum of the lengths the norms of the returned rows stand for, which every replica computes alike whether or not it has compacted.

hello-stephen · 2026-10-22T06:46:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

…eplica suite outside cloud mode Cloud mode keeps one shared copy of each tablet, so there are no replicas to compact apart, and its compaction is driven differently. The comparison with a freshly loaded table still runs in every mode.

When every row of a merge-on-write tablet is overwritten or deleted but not yet compacted, the visible-row statistics hold no documents. Scoring then treated them as missing and failed the query with "No data available for SimilarityCollector". Such a tablet has no row a query can match, so record that its segments were collected and report zero documents instead.

…ablet id The delete bitmap's aggregation cache is keyed by tablet, rowset, segment and version. The mock rowsets share their ids, so the second collection in HiddenRowsAreCollectedLikeCompactedAway read the first one's cached hidden rows.

…merge-on-write tables A NULL row has no tokens, but its norm stands for one, so summing the norms of the returned rows counted one token per NULL row. avgdl then grew with the share of NULL rows, and a merge-on-write table scored its rows differently from a duplicate table with the same data. Leave the rows of the index's null bitmap out of the token count; they still count as documents, as before. The sum of the non-NULL norm lengths of a CLucene index never changes, so keep it with the cached searcher instead of summing every row's norm on every query. Each query now reads only the norms of its hidden rows. Read the delete sign with the query's segment, page and file cache settings, and also read it when the zone map cannot vouch for its bounds or shows a non-zero minimum, since any non-zero sign deletes the row. Document the read modes whose returned rows differ from the counted ones. Follow the regression test conventions in the replica suite, and check it against a duplicate table with NULL rows.

…e options The statistics read postings, norms and null bitmaps before the scan does, so they are the reads that fill the file cache, yet their I/O context ignored disable_file_cache and inverted_index_snii_read_no_write_file_cache. Set both for a query's reads, as the rowset reader does; the delete sign read copies the context and follows them too. Document that a row indexed without tokens but not NULL keeps a norm that cannot be told from a one-token row's, and test that the non-NULL norm length sum is computed once per field of a cached searcher.

…stics searchers A CLucene searcher that BM25 statistics open is put into the searcher cache, but its index stream kept the opening query's I/O context, with that query's file cache statistics pointer, and stayed marked as reading index meta. Later reads without their own context went through the finished query's context, and the statistics' own reads ignored the query's file cache options. Reset the stream once the meta is read, as the query path's searcher opener does.

When a CLucene index holds several segments, its reader builds and keeps one more copy of a field's norms for all of them, filled without the segments' own norm caches, so the statistics read the norms a second time and held them twice for the cached searcher's lifetime, uncharged. Read the norms of each segment instead, as scoring does; a segment without the field's norms reads as zero lengths, as the combined norms do. Add a regression test for the statistics searcher's stream reset: with the CSIndexInput.readInternal debug point a cold score() query failed while the stream still read postings as index meta.

…a CLucene index The BM25 statistics opened a segment's CLucene index without catching CLuceneError, so an error reading the index, such as a failed remote read of its header, escaped the scanner thread and aborted the backend. Report it as the query's error, as the query path does. Keep the searcher reset test from passing on searchers the write path caches, test a hidden row whose length the norm holds only approximately (CLucene's norms are exact up to 40 tokens), and correct the norm comment.

yiguolei · 2026-10-24T03:17:31Z

/review

yiguolei · 2026-10-24T03:17:42Z

run buildall

github-actions

Request changes for two resource regressions: eager CLucene norms remain outside searcher-cache accounting (P1), and sparse hidden-row intersections repeatedly allocate segment-sized bitmaps (P2). Both are detailed inline. No existing inline findings were supplied.

Static review covered all 21 changed files, the scanner/read-source chain, both index formats, and the exact pinned CLucene norm implementation. The read flow is: captured rowsets + bitmap/version → scanner/shared tablet statistics → per-segment hidden IDs → document/term/norm totals → BM25 scoring.

Required checkpoint	Conclusion
Goal and proof	Bitmap/delete-sign-hidden MoW rows are removed consistently from document counts and term frequencies; norm-based lengths address replica compaction differences. New unit/regression sources exercise this goal; execution was not performed.
Scope, clarity and reuse	Changes stay within BM25 statistics and reuse common IndexSource helpers with backend-specific overrides.
Concurrency and locks	Scanner-local results and shared first-collector locking are consistent. The new sum mutex serializes its initial I/O; CLucene reader/norm locks have no identified reverse acquisition path. No new thread or deadlock found.
Lifecycle and static initialization	Cached searchers retain readers and captured rowsets remain protected during collection. Borrowed opening I/O context is cleared. Retained norm accounting is the P1 finding; no new static-initialization dependency found.
Configuration and dynamic changes	No configuration added. Existing query file-cache, segment-cache and page-cache choices are forwarded on the changed paths; CLucene norms retain the existing context-free API behavior.
Symbol/storage compatibility	Internal signatures/default virtual methods change; no persisted index or wire format changes. Existing missing-norm rejection and no-bitmap behavior remain.
Thrift/protobuf IDs	Not applicable: no generated definitions changed.
Parallel and special paths	CLucene/SNII, split scanners, physical segment IDs, reserved scoring terms, NULLs, no-norm indexes and all-hidden tablets checked. No additional correctness issue confirmed.
Conditional checks	Empty inputs, invalid row IDs, zone-map sign pruning and all-hidden handling follow concrete cases and propagate failures.
Functional/negative/unit coverage	New tests cover compaction equivalence, NULL/hidden/all-hidden rows, sparse/dense cursors, invalid IDs, cache computation errors, multiple readers and I/O-context reset. The new checked-in end-to-end suites do not explicitly select SNII; its hidden-row behavior was reviewed statically.
Changed test results	Updated BM25 score and new golden outputs are consistent with the input rows and BM25 arithmetic. Their generation provenance and runtime reproducibility were not verified.
Observability	Existing collection VLOG and error propagation remain available. Local delete-sign read counters are not merged into the parent profile; retained norms also escape cache metrics as described inline.
Transactions/persistence/failover	No transaction, EditLog or metadata-write changes. Read-version and aggregate delete-bitmap keys match the normal rowset reader.
Writes/atomicity/crashes	No new data writes. I/O/cursor failures clear partial statistics and the shared collector returns the same failure to all callers; memory exhaustion risk is P1.
FE/BE variable propagation	No new transmitted variables. FE selected-index schema includes hidden delete-sign metadata; existing cache options reach the new reader context.
CPU/memory/performance	One-time norm totals and tablet sharing avoid redundant collection; the two inline findings identify remaining unbounded retained memory and repeated sparse-intersection work.
Other issues	Final static recheck found no additional substantiated defect.

No additional user focus was supplied. Disclosed scope limits remain: light-delete predicates, merge-on-read/duplicate-key delete visibility, skip_delete_sign/show_hidden_columns, non-NULL zero-token encodings, and statistics remaining per tablet. These were considered and are not duplicate inline findings. No builds, tests, format checks or runtime experiments were run, as this invocation permits static review only.

Review converged after 2 round(s).

github-actions · 2026-10-24T03:30:54Z

+#ifdef BE_TEST
+                RETURN_IF_ERROR(sum_lengths(&non_null_length_sum));
+#else
+                RETURN_IF_ERROR(inverted_index_cache_handle.get<CluceneSearcherCacheValue>()


[P1] Account for the norms retained by statistics collection

This callback now loads norms for every segment in a MoW tablet's captured read set before normal key/zone-map pruning, even when its hidden-row set is empty. CLucene's SegmentReader::norms() allocates and retains one byte per document per scored field, but these searchers were inserted using only getTermInfosRAMUsed() and neither their cache charge nor tracked bytes are updated afterward. A selective query over a large, low-vocabulary tablet can therefore leave gigabytes of norms resident while the searcher cache accounts for only small dictionaries; repeated queries across tablets can exhaust BE memory despite the configured cache limit. Previously such pruned segments only contributed dictionary statistics and did not load these norms. Please reserve/track the added buffers and update the cache charge, including for searchers already in the cache, or compute the sum without retaining unaccounted norm buffers.

Fixed in 96a0b49: the statistics no longer ask CLucene readers for their retained norms. They read each segment's norms into their own buffer via IndexReader::norms(field, bytes) and free it after use. Only small sums are cached with the searcher: the per-field non-NULL norm-length sum, and the hidden rows' length sum for the last few sets of hidden rows per field (queries at one read version reuse it and read no norms). Measured on a 10M-row tablet with 4 scored fields and a key-pruned query: memory growth went from about +51 MB to +12–15 MB, the same as before the PR. Covered by HiddenNormLengthSumIsComputedOncePerRowSet and NormLengthSumsReadEachSegment.

github-actions · 2026-10-24T03:30:54Z

+// Decodes every posting once and tests it against a bitset of `docs`.
+Status count_listed_docs_by_scan(PostingsCursor& cursor, std::span<const uint32_t> docs,
+                                 uint64_t* out) {
+    std::vector<uint64_t> bits(docs.back() / 64 + 1, 0);


[P2] Avoid a segment-sized bitmap for sparse term intersections

The scan branch is selected from docs.size() and doc_freq(), but this allocation scales with the largest hidden row ID. For example, hidden_docs = {9999999} and a term with one posting select this branch and zero 1,250,000 bytes just to compare two IDs. CLucene also returns a non-null cursor for an absent term, so df = 0 incurs the same allocation before discovering EOF. Collection statistics repeat this for every query term and every segment with hidden rows on every query, making selective or many-term searches perform substantial segment-sized allocation/zeroing even with tiny or empty posting lists. Please use a sparse/sorted intersection for this case, account for the maximum ID when choosing a dense bitset, and short-circuit zero-frequency terms.

Fixed in 96a0b49: count_listed_docs no longer allocates a bitset. The scan path walks the hidden rows and the postings together (both ascending) and stops once every hidden row is passed, so the cost no longer depends on the largest row ID. A term with doc_freq() == 0 returns before opening anything. Covered by CountListedDocsScansFarDocsWithoutWalkingEveryBlock (IDs up to 9,999,999) and CountListedDocsOfNothing.

Asking a CLucene segment reader for a field's norms makes it keep them for the cached searcher's lifetime, uncharged to the searcher cache, and the merge-on-write statistics visit every segment of the read set, including the segments the scan prunes and never scores. Read the norms into a buffer of the statistics' own, one segment at a time, and keep only sums with the searcher: the field's non-NULL norm lengths, and the hidden rows' lengths for the last few sets of hidden rows, so queries at one read version read no norms. Count the hidden rows a term's postings list by walking both ascending lists together instead of zeroing a bitset as large as the largest hidden row, and skip terms without postings.

hello-stephen · 2026-10-24T04:08:39Z

TPC-H: Total hot run time: 28796 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7830cb129cb21c4d5ef045aef00dcb996c1db36a, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17621	5185	5206	5185
q2	2119	316	281	281
q3	10197	1366	828	828
q4	4680	461	337	337
q5	7430	810	525	525
q6	179	182	149	149
q7	749	793	615	615
q8	9259	1280	1028	1028
q9	6120	4244	4184	4184
q10	6834	1313	1014	1014
q11	417	247	238	238
q12	640	400	284	284
q13	18065	3572	2958	2958
q14	267	271	248	248
q15	q16	709	713	646	646
q17	1180	745	582	582
q18	7054	6180	6097	6097
q19	1107	944	615	615
q20	383	325	222	222
q21	4971	2487	2498	2487
q22	374	318	273	273
Total cold run time: 100355 ms
Total hot run time: 28796 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5972	5907	5924	5907
q2	744	605	544	544
q3	5540	5786	5256	5256
q4	2022	2132	1558	1558
q5	5977	5559	5710	5559
q6	249	194	145	145
q7	2289	1974	1775	1775
q8	3010	2621	2671	2621
q9	8051	7510	7527	7510
q10	3652	3579	3184	3184
q11	591	410	386	386
q12	655	699	497	497
q13	3209	3644	3050	3050
q14	283	288	264	264
q15	q16	688	686	608	608
q17	7719	7018	6898	6898
q18	12989	12152	12843	12152
q19	881	787	795	787
q20	2151	2151	1931	1931
q21	5617	4594	4736	4594
q22	485	454	407	407
Total cold run time: 72774 ms
Total hot run time: 65633 ms

hello-stephen · 2026-10-24T04:18:28Z

TPC-DS: Total hot run time: 150742 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7830cb129cb21c4d5ef045aef00dcb996c1db36a, data reload: false

query5	4306	596	435	435
query6	428	212	190	190
query7	4809	436	225	225
query8	318	164	158	158
query9	8760	3094	3094	3094
query10	481	308	260	260
query11	5933	3884	3571	3571
query12	147	90	87	87
query13	1233	463	330	330
query14	6110	3910	3624	3624
query14_1	3434	3452	3378	3378
query15	205	196	175	175
query16	955	403	423	403
query17	868	664	533	533
query18	2424	429	320	320
query19	193	169	140	140
query20	87	83	84	83
query21	207	126	111	111
query22	13046	13029	12946	12946
query23	13402	12935	12463	12463
query23_1	12555	12487	12489	12487
query24	7416	823	461	461
query24_1	496	481	471	471
query25	537	424	360	360
query26	1278	263	142	142
query27	2804	451	283	283
query28	5054	1971	1967	1967
query29	1591	600	469	469
query30	311	219	188	188
query31	1029	926	820	820
query32	145	96	94	94
query33	576	305	243	243
query34	942	845	489	489
query35	764	773	645	645
query36	811	813	735	735
query37	129	106	81	81
query38	1787	1734	1676	1676
query39	792	787	761	761
query39_1	735	740	736	736
query40	216	122	97	97
query41	72	70	68	68
query42	88	90	85	85
query43	342	367	309	309
query44	1309	679	691	679
query45	181	179	164	164
query46	823	974	564	564
query47	3013	2841	2973	2841
query48	336	313	210	210
query49	565	388	293	293
query50	670	278	212	212
query51	10559	10224	10632	10224
query52	79	77	69	69
query53	191	203	144	144
query54	251	190	174	174
query55	74	71	63	63
query56	231	204	208	204
query57	1666	1629	1578	1578
query58	284	266	259	259
query59	2082	2144	1946	1946
query60	287	244	222	222
query61	151	149	146	146
query62	384	336	286	286
query63	186	158	163	158
query64	2733	972	770	770
query65	3323	3275	3250	3250
query66	1773	419	293	293
query67	20219	19961	19971	19961
query68	2977	1022	601	601
query69	408	282	257	257
query70	850	769	773	769
query71	301	236	203	203
query72	2770	2604	2318	2318
query73	506	518	293	293
query74	4623	4495	4364	4364
query75	2316	2308	1917	1917
query76	2335	1006	583	583
query77	338	368	272	272
query78	9804	9679	9203	9203
query79	989	855	519	519
query80	1215	417	370	370
query81	580	321	270	270
query82	591	136	110	110
query83	305	192	179	179
query84	308	122	106	106
query85	854	415	369	369
query86	394	241	231	231
query87	2014	1919	1811	1811
query88	3679	2657	2620	2620
query89	327	277	238	238
query90	1868	170	167	167
query91	158	147	128	128
query92	100	90	86	86
query93	1121	997	567	567
query94	640	290	266	266
query95	575	331	374	331
query96	666	487	238	238
query97	2430	2420	2332	2332
query98	158	148	142	142
query99	768	761	644	644
Total cold run time: 236082 ms
Total hot run time: 150742 ms

hello-stephen · 2026-10-24T04:23:20Z

ClickBench: Total hot run time: 24.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7830cb129cb21c4d5ef045aef00dcb996c1db36a, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.26	0.10	0.09
query4	1.61	0.10	0.11
query5	0.23	0.24	0.23
query6	1.18	0.68	0.68
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.34	0.36
query10	0.58	0.58	0.57
query11	0.20	0.15	0.14
query12	0.19	0.14	0.14
query13	0.50	0.49	0.49
query14	1.02	1.02	0.99
query15	0.63	0.60	0.59
query16	0.36	0.34	0.35
query17	1.13	1.19	1.16
query18	0.20	0.19	0.20
query19	2.14	1.94	1.98
query20	0.02	0.01	0.01
query21	15.48	0.21	0.14
query22	4.82	0.05	0.05
query23	16.28	0.34	0.12
query24	3.06	0.55	0.47
query25	0.12	0.04	0.05
query26	1.13	0.20	0.15
query27	0.06	0.05	0.04
query28	3.96	0.78	0.40
query29	12.47	4.10	3.23
query30	0.28	0.13	0.10
query31	2.77	0.56	0.31
query32	3.23	0.60	0.50
query33	3.10	3.24	3.26
query34	15.38	4.00	3.37
query35	3.34	3.34	3.37
query36	0.54	0.43	0.43
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.38 s
Total hot run time: 24.39 s

…w-statistics

…w BM25 statistics The merge-on-write cases overwrite and delete rows, which no longer count in the BM25 statistics, so their scores are those of the visible rows. The same rows are returned; only their scores change.

hello-stephen · 2026-10-24T06:32:58Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.15% (35927/46565)
Line Coverage	62.45% (405966/650088)
Region Coverage	58.89% (342979/582437)
Branch Coverage	59.88% (158715/265057)

airborne12 · 2026-10-24T08:42:14Z

run buildall

airborne12 · 2026-10-24T08:42:20Z

/review

github-actions

One new P2 finding: temporary hidden/NULL row-ID arrays bypass query memory accounting before scan pruning. It is detailed inline. No P0/P1 issue was independently confirmed for this head.

Both existing threads were revalidated: retained CLucene norms (4365847990) is fixed for the reviewed Doris writer/reader paths by using caller-owned norm buffers and bounded scalar caches; sparse intersection allocation (4365847994) is fixed by sorted intersection and zero-frequency exits. The new finding concerns temporary uint32 row-ID arrays, distinct from retained norm buffers. No existing blocker is carried forward.

Static coverage includes all 22 changed files and their scan, reader, writer, cache and scoring interactions at a690a60. The invocation chain is:

captured rowsets + bitmap/read version
  -> scanner/shared tablet statistics
  -> per-segment bitmap/delete-sign hidden IDs
  -> CLucene or SNII visible document/term/norm totals
  -> scanner-local BM25 statistics and matched-row scores

Required checkpoint	Conclusion
Goal and tests	Bitmap/delete-sign-hidden MoW rows are removed from document counts, term frequencies and decoded norm totals. Replica/fresh-table, NULL and all-hidden fixtures support compaction invariance statically; the new memory issue remains.
Scope, clarity and reuse	Changes stay within BM25 collection, source helpers, cache sums and associated tests; existing captured read sources and segment/column readers are reused.
Concurrency and locks	Scanner tasks and queries can collect concurrently. TabletCollectionStatistics locks once-only collection and copies results into scanner-local maps; one searcher mutex protects sum caches. Its initial I/O is serialized intentionally, with no verified reverse lock order or deadlock.
Lifecycle and static initialization	Rowset readers and cache handles preserve reader lifetimes. Synchronous callbacks retain no query references after return; local resources release on errors. No new static initialization dependency or ownership cycle was found. Caller-owned norms fix the historical retention path; untracked ID buffers are the inline finding.
Configuration and dynamic changes	No configuration is added. Existing segment/page/file-cache options are forwarded where the read APIs accept them; the existing context-free CLucene norm API remains a limitation.
Symbols, formats and rolling compatibility	Internal collection signatures and default virtual helpers have matching callers. No persisted index or wire format changes; old/default delete-sign columns resolve by UID, and existing missing-norm scoring rejection remains.
Thrift/protobuf field IDs	Not applicable: no generated definition changed.
Parallel and special paths	Serial/shared split scanners, CLucene/SNII, physical cloud segment IDs, reserved scoring terms, NULL/empty inputs, no-norm keyword indexes and all-hidden collections were traced. Explicit deletion/debug exceptions are disclosed below.
Conditional checks	Bitmap presence, zero frequency, row bounds, zone-map pruning and all-hidden handling have concrete scenarios. Roaring supplies sorted distinct IDs; bounds/frequency failures return errors instead of silently continuing.
Functional, negative and unit coverage	New tests cover hidden/compacted equivalence, NULL and hidden NULL, all-hidden/refill, sparse/dense postings, distant IDs, invalid row IDs, cache callback errors, multiple readers and I/O flags. Memory-limit and concurrent-cache coverage is absent; dedicated reserved-term and schema/segment-ID edge fixtures are limited, with those paths reviewed statically.
Changed test results	Every new/modified numeric expectation is consistent with the input corpus and BM25 arithmetic, including .8025915, .60996956/.8713851 and 1.1823694. Output-generation provenance and execution were not verified.
Observability	Existing collection timing, query/tablet/term VLOG and error propagation remain available. Temporary raw arrays escape query memory metrics as described inline; context-free CLucene norms also retain the existing I/O-accounting limitation.
Transactions, persistence and failover	Read-only collection uses the captured rowsets and the same versioned physical-segment bitmap keys as ordinary reads. No transaction, EditLog, persistence or failover protocol changes.
Writes, atomicity and crash behavior	No data-write path changes. Failed collection clears partial statistics and shares the error with other scanners; cache sums are inserted only after successful computation. The verified resource-accounting risk is inline.
FE/BE variable propagation	No transmitted variables are added. Existing cache options reach the new scanner context.
CPU, memory and performance	Tablet sharing, zone-map delete-sign pruning, bounded batches and cached norm sums reduce repeated work. Sparse posting intersection no longer allocates by maximum ID. Full-cardinality untracked ID buffers remain the new P2 issue.
Further issues	Final rechecks of cursor progress, norm ownership/overloads, NULL writer metadata, TopN masks, cached-reader lifetimes and parallel paths found no additional substantiated defect.

No additional user focus was supplied. Declared scope limits remain: enable_mow_light_delete predicates are still counted until compaction; skip_delete_sign/show_hidden_columns do not change the exclusion of delete-sign rows; skip_delete_bitmap and non-MoW tables retain their existing statistics behavior; non-NULL zero-token encodings remain approximate; statistics remain per tablet. CLucene caller-buffer behavior was checked against the pinned SegmentReader implementation, with Doris NULL-writer metadata traced to exclude a synthetic missing-field branch.

No builds, tests, format checks or Doris runtime experiments were run under this static-only review contract.

Review converged after 2 round(s).

hello-stephen · 2026-10-24T09:12:51Z

TPC-H: Total hot run time: 28525 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a690a608ebdbf0e16432a1d290ccfd8dbbbd31d5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17591	5141	5082	5082
q2	2137	299	249	249
q3	10272	1376	821	821
q4	4678	453	336	336
q5	7433	809	533	533
q6	186	196	159	159
q7	758	787	595	595
q8	9251	1300	1102	1102
q9	5357	4201	4189	4189
q10	6838	1302	1015	1015
q11	427	251	233	233
q12	638	403	284	284
q13	18079	3550	3023	3023
q14	280	274	254	254
q15	q16	700	719	689	689
q17	1198	760	586	586
q18	7064	6163	6135	6135
q19	1101	955	622	622
q20	395	336	230	230
q21	4528	2140	2259	2140
q22	346	290	248	248
Total cold run time: 99257 ms
Total hot run time: 28525 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5645	5771	5547	5547
q2	666	567	545	545
q3	5213	5410	4939	4939
q4	1970	2175	1543	1543
q5	5290	5157	5499	5157
q6	253	204	169	169
q7	2411	2108	1761	1761
q8	3063	2743	2762	2743
q9	8095	7946	7965	7946
q10	3686	3658	3245	3245
q11	605	437	445	437
q12	856	719	543	543
q13	3510	3929	3200	3200
q14	319	333	297	297
q15	q16	708	711	679	679
q17	8445	7476	6997	6997
q18	13014	12203	12856	12203
q19	952	843	871	843
q20	2153	2142	1915	1915
q21	5813	4689	4827	4689
q22	486	459	413	413
Total cold run time: 73153 ms
Total hot run time: 65811 ms

hello-stephen · 2026-10-24T09:22:37Z

TPC-DS: Total hot run time: 150260 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a690a608ebdbf0e16432a1d290ccfd8dbbbd31d5, data reload: false

query5	4307	596	446	446
query6	432	203	191	191
query7	4800	435	225	225
query8	309	156	152	152
query9	8776	3135	3084	3084
query10	477	312	247	247
query11	6047	3876	3570	3570
query12	143	97	86	86
query13	1270	440	323	323
query14	6148	3958	3663	3663
query14_1	3411	3422	3422	3422
query15	201	198	181	181
query16	944	430	399	399
query17	853	663	528	528
query18	2417	433	313	313
query19	217	174	140	140
query20	85	84	85	84
query21	212	130	106	106
query22	12989	12993	12787	12787
query23	13394	12808	12351	12351
query23_1	12622	12484	12742	12484
query24	7360	816	455	455
query24_1	467	466	473	466
query25	531	423	359	359
query26	1259	263	140	140
query27	2888	444	276	276
query28	5266	1984	1997	1984
query29	1586	632	470	470
query30	299	229	186	186
query31	1038	936	821	821
query32	144	98	93	93
query33	525	307	248	248
query34	906	862	531	531
query35	755	772	660	660
query36	808	809	701	701
query37	127	104	85	85
query38	1842	1784	1677	1677
query39	801	817	758	758
query39_1	755	746	761	746
query40	221	120	98	98
query41	72	68	71	68
query42	91	85	84	84
query43	357	392	324	324
query44	1363	706	715	706
query45	184	183	164	164
query46	855	994	582	582
query47	2769	2877	2627	2627
query48	354	315	216	216
query49	566	402	292	292
query50	696	277	216	216
query51	10419	10570	10211	10211
query52	77	81	68	68
query53	204	209	152	152
query54	227	189	180	180
query55	74	67	63	63
query56	244	229	221	221
query57	1665	1613	1577	1577
query58	287	268	252	252
query59	2111	2118	1889	1889
query60	284	226	222	222
query61	147	144	143	143
query62	391	325	290	290
query63	191	155	169	155
query64	2743	953	749	749
query65	3319	3276	3268	3268
query66	1762	409	294	294
query67	20045	19920	19950	19920
query68	3052	1004	620	620
query69	373	288	245	245
query70	837	756	721	721
query71	292	228	218	218
query72	2675	2573	2282	2282
query73	506	519	298	298
query74	4622	4484	4360	4360
query75	2321	2282	1904	1904
query76	2307	1001	594	594
query77	336	374	273	273
query78	9863	9752	9151	9151
query79	975	892	527	527
query80	540	406	335	335
query81	549	313	264	264
query82	576	134	107	107
query83	302	216	183	183
query84	316	123	106	106
query85	793	413	366	366
query86	329	237	222	222
query87	1999	1956	1817	1817
query88	3649	2666	2652	2652
query89	327	285	242	242
query90	1878	175	173	173
query91	157	144	121	121
query92	87	92	92	92
query93	993	1002	568	568
query94	460	293	277	277
query95	572	330	361	330
query96	675	473	232	232
query97	2453	2414	2347	2347
query98	166	151	147	147
query99	780	758	635	635
Total cold run time: 234642 ms
Total hot run time: 150260 ms

github-actions · 2026-10-26T10:35:21Z

PR approved by anyone and no changes requested.

github-actions · 2026-10-26T10:35:21Z

PR approved by anyone and no changes requested.

github-actions · 2026-10-26T10:35:23Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-10-26T10:35:23Z

PR approved by at least one committer and no changes requested.

…tistics of merge-on-write tables #69438 (#69752) ### What problem does this PR solve? Issue Number: None Related PR: #69438 (this is its backport to branch-4.2), #69352 and its branch-4.2 pick #69440 (BM25 statistics shared across the parallel scanners of a tablet) Problem Summary: Backport of #69438 to branch-4.2. On a merge-on-write table, BM25 `score()` gives the same row a different score on each replica of its tablet. A query picks one replica per tablet, so with `replication_num = 3` a row has up to three scores, and pagination over `ORDER BY score()` repeats and skips rows from one page to the next. `CollectionStatistics::process_segment()` takes the document count, the token count and each term's document frequency straight from every segment's inverted index. Rows a later load overwrote or deleted (marked only in the delete bitmap) and rows written with the delete sign (CDC deletes, `DELETE`) stay in the index and are counted until compaction removes them, and each replica compacts on its own schedule. On branch-4.2 (3 BEs, `replication_num = 3`, 8 buckets, CDC-style upserts and delete-sign deletes, the replicas on one BE fully compacted), all 1,731 matching rows of a `SEARCH(... OR ...)` query got more than one score over 15 runs and differed between `use_fix_replica = 0/1/2`. Fix: the same change as #69438. When the scan reads a delete bitmap (merge-on-write), `CollectionStatistics` describes only the rows the scan returns: - `OlapScanner` passes the scan's delete bitmap and read version (`MowHiddenRows`) to `collect()`, or to `TabletCollectionStatistics::copy_to()` for a split tablet (#69440). - Per segment, the hidden rows are the rows the delete bitmap marks at the read version plus the rows whose `__DORIS_DELETE_SIGN__` is set. The delete-sign column is read only for segments whose zone map shows a non-zero sign (or cannot tell), following the query's `enable_segment_cache` and `enable_page_cache`. - Document count: the index's document count minus the hidden rows. Term document frequency: `IndexSource::scoring_doc_freq_among()` subtracts the hidden rows a term's postings list, leaping between them when they are much rarer than the postings, otherwise walking them alongside the decoded postings. - Token count: the sum of the document lengths the norms of the returned non-NULL rows stand for (NULL rows left out through the index's null bitmap), or the returned row count for an index without norms. Every replica counts the same, compacted or not. - CLucene norms are read one segment at a time into a buffer of the statistics' own and not kept; the non-NULL norm length sum of a field and the length sums of its last four sets of hidden rows are kept with the cached searcher (`CluceneSearcherCacheValue`). - A tablet whose rows are all hidden reports zero documents instead of failing the query. Tables without a delete bitmap collect the same statistics as before. For every `score()` query, the statistics' I/O context follows `disable_file_cache` and `inverted_index_snii_read_no_write_file_cache`, and a CLucene searcher the statistics open drops the query's I/O context once its meta is read; a CLucene error while opening it fails the query. Differences from #69438, for branch-4.2: segments are identified by their id instead of `RowsetSegmentView`, and the delete-sign column reader comes from `Segment::get_column_reader()` (branch-4.2 has no `get_column_reader_for_pruning`), a segment without the column reading as no deletes. The tests are adapted the same way. Not covered (as in #69438): merge-on-read unique tables and duplicate tables with `DELETE` predicates; rows deleted by a delete predicate (`enable_mow_light_delete`); rows written with the delete sign that the query returns (`skip_delete_sign`, `show_hidden_columns`) are still left out; an analyzed empty string, and on SNII any text that yields no token, counts as one token on a merge-on-write table; SNII norm length sums are not cached. Statistics stay per tablet. ### Release note Fix BM25 `score()` on merge-on-write tables differing between the replicas of a tablet, and before and after compaction. Statistics now count only the rows a query can return, and the average document length comes from the indexed norms. ### Check List (For Author) - Test - [x] Regression test: - New: `inverted_index_p0/test_bm25_score_replica_statistics` and `fault_injection_p0/test_bm25_score_statistics_searcher_io_context` (same suites as #69438). Updated `.out`: `test_bm25_score.out`, `test_score_filter_matrix.out` (V2, V3 and SNII merge-on-write cases), same values as #69438. - All 26 branch-4.2 suites that use `score()` (`inverted_index_p0`, `search`, `ann_index_p0`, `variant_p0`, `nereids_rules_p0`, `fault_injection_p0`) pass with this change on a 3-BE cluster. Without it, the replica suite, the fault-injection suite (the query fails with `[E-6002] The 'is_index_data' flag should be false for non-meta inverted index files.`) and the two updated suites fail. - [x] Unit Test: the tests of #69438 (`CollectionStatisticsTest.HiddenRowsAreCollectedLikeCompactedAway`, `AllRowsHiddenCollectsEmptyStatistics`, `NullRowsCountAsDocumentsWithoutTokens`, `CollectionStatisticsHiddenRowsTest.*` for `count_listed_docs`, `exclude_hidden_docs` and `exclude_hidden_term_docs`, `CluceneSearcherCacheValueTest.*`, `MultiSegmentCollectorTest.NormLengthSumsReadEachSegment`, `OlapScannerTest.BuildScoreRuntimeCollectionIoContextFollowsQueryFileCacheOptions`), with the existing collection statistics, BM25, searcher cache, multi-segment, query engine, SNII and `SEARCH` function tests: 427 tests in 35 suites pass in an ASAN build on macOS arm64. - [x] Manual test (3 BEs, macOS arm64, `replication_num = 3`, auto compaction off, `enable_sql_cache = false`), base = branch-4.2 without this change, same data and compaction state for both builds: | Check | branch-4.2 | with this change | |---|---|---| | CDC-style table (8 buckets, upserts, delete-sign deletes, NULLs, 4 fields), one BE's replicas fully compacted with inverted index compaction off: rows with more than one score over 15 runs / differing between replicas 0, 1, 2 (of 1,731) | 1,731 / 1,731 | 0 / 0 | | Same, scores differing from a freshly loaded copy of the visible rows | — | 0 of 1,731 | | Partial-column updates, sequence column, stream load with the delete sign and `merge_type: DELETE`, group commit, keyword and lowercase-normalizer indexes, partitions, variant, SNII: rows differing between replicas after one replica compacted | 84–790 per table (8 of 9 checks) | 0 in every check, all equal to a fresh copy | | Merge-on-write vs duplicate table with 50 % NULL rows, V3 and SNII | 0 of 250 differ | 0 of 250 differ | Cost, query `SEARCH('content:jobs OR content:try') ORDER BY score() DESC LIMIT 100`, one bucket, median of 15 warm runs; `StatisticsCollectTime` with one scanner, query total of `SEARCH('content:jobs')` with the default parallel scan: | Table | Statistics | Query total | |---|---|---| | V2, 8 M rows + 200 k overwritten + 50 k delete-sign rows | 0.3 ms → 6.7 ms | 68 ms → 76 ms | | V3, 6 M rows + 2 M overwritten + 200 k delete-sign rows (2.2 M hidden, 6 % NULL) | 0.3 ms → 55 ms | 61 ms → 113 ms | | V2, 6 M rows + 2 M overwritten, no NULL rows | 0.3 ms → 7.2 ms | 56 ms → 61 ms | | SNII, 2 M rows + 500 k overwritten + 200 k delete-sign rows | 0.2 ms → 23 ms | 49 ms → 71 ms | The cost grows with the rows compaction has not removed yet. With NULL rows, most of it is leaving the NULL rows out of the hidden rows on every query (`exclude_hidden_docs`), as in #69438. Memory (BE allocated bytes, sum of 3 BEs, a 4-field merge-on-write table of 41 segments): no growth beyond the noise of the measurement (after 4 queries +26 MB → +25 MB with key pruning; +142 MB → below the starting value scoring all segments). - Behavior changed: - [x] Yes. On merge-on-write tables, BM25 statistics leave out rows hidden by the delete bitmap and rows with the delete sign, and avgdl is computed from the norms of the visible rows, so scores no longer depend on the replica or on compaction. Scores of merge-on-write tables shift once, also without deletes, because norms store long lengths approximately. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label Co-authored-by: Shivam Gupta <[email protected]>

…tistics of merge-on-write tables #69438 (#69751) ### What problem does this PR solve? Issue Number: None Related PR: #69438 (this is its backport to branch-4.1), #69352 and its branch-4.1 pick #69418 (BM25 statistics shared across the parallel scanners of a tablet) Problem Summary: Backport of #69438 to branch-4.1. On a merge-on-write table, BM25 `score()` gives the same row a different score on each replica of its tablet. A query picks one replica per tablet, so with `replication_num = 3` a row has up to three scores, and pagination over `ORDER BY score()` repeats and skips rows from one page to the next. `CollectionStatistics::process_segment()` takes the document count, the token count and each term's document frequency straight from every segment's inverted index. Rows a later load overwrote or deleted (marked only in the delete bitmap) and rows written with the delete sign (CDC deletes, `DELETE`) stay in the index and are counted until compaction removes them, and each replica compacts on its own schedule. On branch-4.1 (3 BEs, `replication_num = 3`, 8 buckets, CDC-style upserts and delete-sign deletes, the replicas on one BE fully compacted), the 1,731 matching rows of a `SEARCH(... OR ...)` query had 1,294 rows with more than one score over 15 runs, and all 1,731 differed between `use_fix_replica = 0/1/2`. Fix (same as #69438): When the scan reads a delete bitmap (merge-on-write), `CollectionStatistics` describes only the rows the scan returns: - `OlapScanner` passes the scan's delete bitmap and read version (`MowHiddenRows`) to `collect()`, or to `TabletCollectionStatistics::copy_to()` for a split tablet (#69418). Every scanner of a tablet reads with the same bitmap and version. - For each segment, the hidden rows are the rows the delete bitmap marks at the read version (`get_agg`), plus the rows whose `__DORIS_DELETE_SIGN__` is set. The delete-sign column is read only for segments whose zone map shows a non-zero sign (or cannot tell), following the query's `enable_segment_cache` and `enable_page_cache`. - Document count: the index's document count minus the hidden rows. - Term document frequency: the hidden rows a term's postings list are subtracted. When the hidden rows are much rarer than the postings (32 times), the collector leaps from hidden row to hidden row; otherwise it decodes the postings once and walks the hidden rows alongside, without a bitset. - Token count: the sum of the document lengths the norms of the returned non-NULL rows stand for, or the returned row count for an index without norms. A row's norm is the same in every segment it is written to, so every replica counts the same, compacted or not. NULL rows are left out of the token count through the index's null bitmap and still count as documents. - The statistics read CLucene norms one CLucene segment at a time into a buffer of their own (`IndexReader::norms(field, bytes)`) and keep none. What they keep is small and lives with the cached searcher: a field's non-NULL norm length sum, and the length sums of the last four sets of hidden rows of a field. The hidden and NULL row ID lists are charged to the query's memory. - A tablet whose rows are all hidden reports zero documents instead of failing the query with "No data available for SimilarityCollector". Tables without a delete bitmap (duplicate key, merge-on-read, `skip_delete_bitmap`) collect the same statistics as before. For every `score()` query: the statistics' I/O context follows the query's `disable_file_cache`, and a CLucene searcher the statistics open drops the query's I/O context once its meta is read, as the query path does. A CLucene error while opening it now fails the query; before, it escaped the scanner thread and aborted the backend (reproduced on branch-4.1 by `fault_injection_p0/test_bm25_score_statistics_searcher_io_context`). Differences from #69438, for branch-4.1: - branch-4.1 has no SNII format and no `IndexSource` layer. The hidden-row counting works on `TermIterator` (`read_range` / `advance`) and the norm reading on the CLucene reader directly; both are helpers in `storage/compaction/collection_statistics.cpp` (`collection_statistics_detail`). - The norm length sums are kept on `InvertedIndexSearcherCache::CacheValue`; branch-4.1 has no `CluceneSearcherCacheValue`. - `InvertedIndexReader::read_null_bitmap` is split so that the statistics can read the null bitmap through the query cache without an `InvertedIndexReader` (`read_inverted_index_null_bitmap()`). - `inverted_index_snii_read_no_write_file_cache` does not exist on branch-4.1, so only `disable_file_cache` is followed. Not covered (as in #69438): merge-on-read unique tables and duplicate tables with `DELETE` predicates; rows deleted by a delete predicate (`enable_mow_light_delete`); rows written with the delete sign that the query returns (`skip_delete_sign`, `show_hidden_columns`) are still left out of the statistics; an analyzed empty string counts as one token on a merge-on-write table. Statistics stay per tablet. ### Release note Fix BM25 `score()` on merge-on-write tables differing between the replicas of a tablet, and before and after compaction. Statistics now count only the rows a query can return, and the average document length comes from the indexed norms. ### Check List (For Author) - Test - [x] Regression test: - New: `inverted_index_p0/test_bm25_score_replica_statistics` (3 replicas, overwrites, `DELETE`, delete-sign rows, one replica fully compacted, scores compared with a freshly loaded table; NULL rows vs a duplicate table; all rows deleted) and `fault_injection_p0/test_bm25_score_statistics_searcher_io_context` (same suites as #69438). - Updated `.out`: `test_bm25_score.out` and the V2/V3 merge-on-write cases of `test_score_filter_matrix.out`, to the same values as #69438. - All 23 branch-4.1 suites that use `score()` (`inverted_index_p0`, `search`, `ann_index_p0`, `variant_p0`, `nereids_rules_p0`, `fault_injection_p0`) pass with this change on a 3-BE cluster. Without it, the two new suites and the two updated ones fail; the fault-injection suite aborts two of the three backends with an uncaught `CLuceneError`. - [x] Unit Test: `CollectionStatisticsTest.HiddenRowsAreCollectedLikeCompactedAway` (real CLucene segments: a segment with a hidden 45-token row collects the same doc count, token count, df, avgdl and idf as the segment without it), `CollectionStatisticsTest.AllRowsHiddenCollectsEmptyStatistics`, `CollectionStatisticsTest.NullRowsCountAsDocumentsWithoutTokens` (real segment with NULL rows, with and without a delete bitmap), `CollectionStatisticsTest.CountListedDocsLeapsOrScansOverHiddenRows` (both counting paths, dense blocks, norm length sums), `CollectionStatisticsHiddenRowsTest.*` (`exclude_hidden_docs` with and without norms and NULL rows, rows beyond the index), `InvertedIndexSearcherCacheValueTest.*` (norm length sums computed once per field and per set of hidden rows, failures not kept, oldest set evicted), `MultiSegmentCollectorTest.NormLengthSumsReadEachSegment` (an index of three CLucene segments sums its norms per segment, equal to the combined norms), `OlapScannerTest.BuildScoreRuntimeCollectionIoContextFollowsQueryFileCacheOptions`. With the existing collection statistics, collection similarity, BM25, searcher cache, searcher builder, olap scanner, multi-segment, boolean query, null predicate and `SEARCH` function tests: 233 tests in 26 suites pass in an ASAN build on macOS arm64. - [x] Manual test (3 BEs, macOS arm64, `replication_num = 3`, auto compaction off, `enable_sql_cache = false`), base = branch-4.1 without this change, same data and compaction state for both builds: | Check | branch-4.1 | with this change | |---|---|---| | CDC-style table (8 buckets, upserts, delete-sign deletes, NULLs, 4 fields), one BE's replicas fully compacted with inverted index compaction off: rows with more than one score over 15 runs / differing between replicas 0, 1, 2 (of 1,731) | 1,294 / 1,731 | 0 / 0 | | Same, scores differing from a freshly loaded copy of the visible rows | — | 0 of 1,731 | | Partial-column updates, sequence column, stream load with the delete sign and `merge_type: DELETE`, group commit, keyword and lowercase-normalizer indexes, partitions, variant: rows differing between replicas after one replica compacted | 84–790 per table (7 of 8 checks) | 0 in every check, all equal to a fresh copy | | Merge-on-write vs duplicate table with 50 % NULL rows (V3) | 0 of 250 differ | 0 of 250 differ | Cost, query `SEARCH('content:jobs OR content:try') ORDER BY score() DESC LIMIT 100`, one bucket, median of 15 warm runs: | Table | `StatisticsCollectTime`, one scanner | Query total, default parallel scan | |---|---|---| | V2, 8 M rows + 200 k overwritten + 50 k delete-sign rows | 0.3 ms → 7.7 ms | 78 ms → 82 ms | | V3, 6 M rows + 2 M overwritten + 200 k delete-sign rows (2.2 M hidden) | 0.3 ms → 62 ms | 71 ms → 125 ms | The first query after a restart also sums each segment's norms (8 M rows: 5 ms → 32 ms; 2.2 M hidden: 6 ms → 99 ms). The cost grows with the rows compaction has not removed yet. Memory (BE allocated bytes, sum of 3 BEs, a 4-field merge-on-write table of 41 segments): after 4 queries +16 MB → +22 MB with key pruning and +109 MB → +75 MB scoring all segments, within the noise of the measurement. - Behavior changed: - [x] Yes. On merge-on-write tables, BM25 statistics leave out rows hidden by the delete bitmap and rows with the delete sign, and avgdl is computed from the norms of the visible rows, so scores no longer depend on the replica or on compaction. Scores of merge-on-write tables shift once, also without deletes, because norms store lengths above 40 tokens approximately. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label --------- Co-authored-by: Shivam Gupta <[email protected]>

…blet idf of the terms the prefix expands to #69698 (#69801) ### What problem does this PR solve? Issue Number: None Related PR: #69698, #69752 Problem Summary: Backport of #69698 to branch-4.2. score() over MATCH_PHRASE_PREFIX ignored the prefix and differed between inverted index storage formats: 1. The idf of a phrase prefix now counts every term the prefix expands to, as Lucene's multi-phrase query and Elasticsearch do, and all idfs come from the tablet statistics. The statistics expand the prefix in every segment and keep the `inverted_index_max_expansions` smallest terms of the tablet that a returned row holds (a session limit of 0 keeps 50), so a term's document frequency is counted in every segment that holds it and merge-on-write hidden rows do not change the scores after a compaction. A term whose bytes are not valid UTF-8 is not scored. 2. A phrase prefix of one token scores 1 on every storage format; on V2/V3 it read 0, so `score() > 0` dropped every row it matched. See #69698 for the full description and the review rounds. This change builds on #69438 (BM25 statistics count only visible rows of merge-on-write tables), whose branch-4.2 backport #69752 is merged. The PR is rebased onto the current branch-4.2 and contains only the backport commit; its patch is identical to the one validated below. ### Branch adaptations - No production code adaptation. The automatic pick conflicted only because #69438 was not on branch-4.2 yet; on top of #69752 the change applies with one conflict, in the context of `collection_statistics_test.cpp`: branch-4.2 already has `CollectConvertsAnalyzerFailureToStatus` elsewhere in the file, so the copy that master shows next to the renamed test is not added again. ### Release note score() over MATCH_PHRASE_PREFIX counts the terms the prefix expands to, and a phrase prefix of one token scores 1 on every inverted index storage format. ### Check List (For Author) - Test: Regression test, Unit Test - Validated locally on branch-4.2 at 711db43, the commit before #69625, with the same two commits on top: the newest branch-4.2 calls a lance-c API that no thirdparty build on the machine used here provides. The PR branch itself is on the current branch-4.2 and is built by CI. - test_match_phrase_prefix_score passes with the `.out` of master unchanged (V2, V3, SNII and the capped two-segment case). - test_bm25_score, test_bm25_score_parallel_scan_statistics, test_bm25_score_range_filter, test_bm25_score_replica_statistics, test_bm25_score_variant, test_score_filter_matrix, test_score_filter_usage and test_search_score_topn_predicates pass. - 1644 related BE unit tests pass (PhrasePrefixQueryV2Test, CollectionStatistics*, PrefixExpansionSetTest, SNII, InvertedIndexReaderTest, MultiSegmentCollectorTest, BM25SimilarityTest). - Behavior changed: Yes (scores of MATCH_PHRASE_PREFIX change) - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>

### What problem does this PR solve? Issue Number: None Related PR: apache#69352, apache#69438, apache#69403 Problem Summary: MoW light deletes filter rows through versioned DELETE predicates, but BM25 statistics only excluded delete-bitmap and delete-sign rows. Reuse DeleteHandler with the captured read version and historical column identities to collect the additional hidden row IDs. Read only predicate columns in bounded batches and merge disjoint, tracked row-ID lists into the existing statistics path. Keep the upstream tablet sharing, per-scanner caches and norms-based avgdl. ### Release note Correct BM25 statistics after merge-on-write light deletes without waiting for compaction. ### Check List (For Author) - Test: 93 selected ASAN tests passed, including three real-segment predicate tests. Full-master SQL passed V2/V3 with phrase true/false and SNII with phrase true, including mutation phases and forced scanner splitting. Existing parallel scan and replica-statistics suites passed. The unfixed query path fails the new regression's initial score comparison. Format, hygiene and changed-line source and test clang-tidy passed. - Behavior changed: Yes. Light-deleted rows no longer affect MoW BM25 statistics. - Does this need documentation: No new configuration or syntax.

… for the delete-sign filter ### What problem does this PR solve? Issue Number: None Related PR: apache#69438 Problem Summary: The BM25 statistics of a merge-on-write table leave out the rows written with the delete sign when the scan filters them. The scan told that by any conjunct that reads `__DORIS_DELETE_SIGN__`. With `show_hidden_columns = true`, a query's own predicate on the column, such as `WHERE __DORIS_DELETE_SIGN__ = 1`, then counted as the filter: the query returned and scored only the delete-sign rows, while the statistics left exactly those rows out, so their scores differed from the scores of the same rows without the predicate. The scan now treats as the filter only a conjunct that is `__DORIS_DELETE_SIGN__ = 0`, the one the FE adds unless the query sets `skip_delete_sign` or `show_hidden_columns`. Any other predicate on the column is a predicate of the query, which the statistics follow no more than a predicate on another column, so the delete-sign rows stay counted. A query that writes `= 0` itself keeps exactly the rows the FE's conjunct keeps and scores as the default. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - OlapScanOperatorDeleteSignTest.OtherPredicateOnDeleteSignDoesNotFilterIt: `= 1`, `!= 0` and `>= 0` on the delete sign do not filter it, `= 0` does. - test_bm25_score_delete_sign_statistics: with `show_hidden_columns = true` and `__DORIS_DELETE_SIGN__ = 1`, every returned row scores as in a duplicate-key table loaded with the rows the query returns without the predicate, on V2, V3 and SNII. Without this change the case fails on V2 at once (80 rows score differently); the .out was regenerated with -forceGenOut and adds only the `__DORIS_DELETE_SIGN__ = 1` block, whose scores equal those of the same rows under `show_hidden_columns` alone. - All 29 regression suites that use score() pass on a 3-BE cluster (release build), and the BM25, statistics, scanner and scan-operator unit tests pass on ASAN (257 tests). - Behavior changed: Yes. With `show_hidden_columns = true`, a query that filters `__DORIS_DELETE_SIGN__` other than by `= 0` scores on statistics that count the delete-sign rows. - Does this need documentation: No

…m once for concurrent queries ### What problem does this PR solve? Issue Number: None Related PR: apache#69438 Problem Summary: The norm length sums that BM25 statistics of merge-on-write tables keep with a cached index reader were computed outside the lock, so queries that missed the same sum at the same time each read the norms (and the NULL rows) to compute it. That happens after every load: the load hides older rows, so queries at the new version need a new hidden-row sum for every segment it touched, and a burst of concurrent queries read the same norms many times. Holding the lock while computing, as the CLucene sums did before, would instead make queries of other fields and other hidden rows of the index wait. A sum is now computed by the first query that needs it. A query that needs a sum being computed waits for it and reads the kept sum; queries of other fields or other sets of hidden rows go on. If computing a sum fails or throws, the key is given up and a waiting query computes it itself, so one query's failure never fails another. ### Release note None ### Check List (For Author) - Test: Unit Test - NormLengthSumsTest.ConcurrentQueriesComputeASumOnce: while one query computes a sum, four queries of the same sum wait and get it without computing, a query of another set of rows is not blocked, and after a failed computation a waiting query computes the sum. It passes 50 repeated runs on ASAN, with the other 256 BM25, statistics and scanner unit tests. - Manual test on a 3-BE cluster (release build): 16 clients starting together on cold caches, 5 rounds, on V3 and SNII tables with overwritten and deleted rows: every row has one score and paging with a tiebreaker repeats or misses no row. 90 s of 16 clients with cancelled and timed-out queries and cache evictions: no other errors, no stuck queries, no BE failure. - Behavior changed: No - Does this need documentation: No

…69510) Related PR: #69469 Problem Summary: A single transient GitHub response currently aborts the review pipeline. [Input resolution for #69322](https://github.com/apache/doris/actions/runs/39794013905/job/119876748768) stopped on a GET 503 despite `Retry-After: 2`; [aggregate resolution for #69438](https://github.com/apache/doris/actions/runs/39760934956/job/119764632659) stopped on a paginated GET 504 after emitting partial output. The shared helper now retries recognized body-free REST GETs on 502/503/504, for at most three attempts. Each attempt has a 30-second limit; attempts and waits share a 90-second budget or the caller's shorter timeout. It respects Retry-After, records every failed attempt, discards failed pagination output, and returns only a successful attempt's complete output. Exhaustion still fails. Writes, GraphQL, unknown argument forms, other HTTP errors, transport/timeouts and JSON parsing errors are not retried. Input resolution can also fail before publishing its PR-number output. Failure notices now fall back to the authorized trigger's numeric PR identity, explicitly select the repository, and name the input-resolution failure in both the notice and terminal error. This fallback does not permit status publication for an unresolved head/base. This change can merge independently. It includes the same explicit `--repo` argument as #69469; a local combination with #69469 and #69504 merged cleanly and passed the macOS suite.

guptashivam04 requested review from airborne12, csun5285 and eldenmoon as code owners October 22, 2026 06:46

Shivam Gupta added 5 commits October 22, 2026 13:30

guptashivam04 requested review from gavinchou and yiguolei as code owners October 23, 2026 14:44

Shivam Gupta added 3 commits October 23, 2026 21:02

github-actions Bot previously requested changes Oct 24, 2026

View reviewed changes

Shivam Gupta added 2 commits October 24, 2026 11:15

Merge remote-tracking branch 'origin/master' into fix/bm25-visible-ro…

8147a24

…w-statistics

github-actions Bot reviewed Oct 24, 2026

View reviewed changes

Comment thread be/src/storage/index/inverted/reader/collection_statistics.cpp Resolved

github-actions Bot added the reviewed label Oct 26, 2026

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 26, 2026

yiguolei added usercase Important user case type label dev/4.1.x dev/4.2.x labels Oct 27, 2026

github-actions Bot added dev/4.1.x-conflict dev/4.2.x-conflict labels Oct 27, 2026

This was referenced Oct 28, 2026

branch-4.1: [fix](inverted index) Count only visible rows in BM25 statistics of merge-on-write tables #69438 #69751

Merged

branch-4.2: [fix](inverted index) Count only visible rows in BM25 statistics of merge-on-write tables #69438 #69752

Merged

yiguolei added dev/4.2.0-merged and removed dev/4.2.x dev/4.2.x-conflict labels Oct 29, 2026

yiguolei added dev/4.1.6-merged and removed dev/4.1.x dev/4.1.x-conflict labels Oct 29, 2026

airborne12 mentioned this pull request Oct 29, 2026

branch-4.2: [fix](inverted index) Score MATCH_PHRASE_PREFIX on the tablet idf of the terms the prefix expands to #69698 #69801

Merged

guptashivam04 mentioned this pull request Oct 29, 2026

[fix](inverted index) Count delete-sign rows in BM25 statistics when the scan returns them #69854

Open

6 of 10 tasks

This was referenced Nov 2, 2026

[fix](be) Preserve BM25 statistics through CLucene compaction #69313

Open

[fix](inverted index) Exclude MoW light deletes from BM25 statistics #69403

Open

Conversation

guptashivam04 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 22, 2026

Uh oh!

yiguolei commented Oct 24, 2026

Uh oh!

yiguolei commented Oct 24, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

guptashivam04 Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

guptashivam04 Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

BE Regression && UT Coverage Report

Uh oh!

airborne12 commented Oct 24, 2026

Uh oh!

airborne12 commented Oct 24, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

hello-stephen commented Oct 24, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

guptashivam04 commented Oct 22, 2026 •

edited

Loading