You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 96fd1e9
Browse filesBrowse the repository at this point in the historyBrowse files
[feature](inverted index) Add SNII inverted index storage format (apache#66052)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Add **SNII**, a Doris-native inverted index storage format designed for
object storage, as an opt-in alternative to the CLucene-based V1/V2/V3.
On cold queries served directly from S3, SNII is about an order of
magnitude faster than V3.
Cold object-storage latency is dominated by the number of round trips,
not by the bytes read. CLucene spreads a single term lookup across
several index streams that sit far apart in the file, so one lookup
becomes many small scattered reads. SNII keeps each segment's indexes in
one container, packs the sections of an index contiguously, aligns them
to file-cache block boundaries, and coalesces adjacent ranges into a
single batched read that is served in parallel — a lookup that costs
CLucene many requests costs SNII a few. Positions are decoded on demand
rather than read in full.
Enable it per table; the cluster default stays V3, so existing tables
and queries are unaffected:
```sql
PROPERTIES ("inverted_index_storage_format" = "SNII")
```
The format covers the full index lifecycle: writing and querying (exact,
analyzed, phrase and sloppy phrase), BM25 scoring, BKD and ANN indexes,
VARIANT sub-columns, the SEARCH DSL, native index compaction, `BUILD
INDEX`, and local plus cloud index changes.
Beyond the format itself, this PR also fixes shared inverted index
behaviour: reader selection no longer aborts the BE process when a
caller has no runtime column type, and match-analyzer resolution is made
consistent (custom analyzers now work with the `none` parser).
### Release note
Add SNII, a new opt-in inverted index storage format designed for object
storage, giving roughly an order of magnitude faster cold queries
against S3. Enable it per table with `"inverted_index_storage_format" =
"SNII"`; the cluster default remains V3, so existing tables are
unaffected.
**Upgrade / rollback constraint:** once a table uses the SNII format,
the cluster can no longer be rolled back to a version that does not
support SNII. A pre-SNII backend reads the new storage-format enum as
the field's default (V1) rather than rejecting it, so it would silently
interpret SNII indexes as V1. Drop or convert such tables to V3 before
rolling back. Upgrades must follow the standard order: upgrade BEs
first, then FEs.
0 commit comments