You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9f3b084
Browse filesBrowse the repository at this point in the historyBrowse files
[refactor](index) Remove CommonGrams and realign the SNII on-disk layout with the released format (apache#67537)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
`CommonGrams` (the token filter, its word set, the term-key escaping
codec, the per-segment metadata, the hybrid docs-only/positioned query
planner and its result-cache gate, the SPIMI pair-key fast path, 17
profile counters, 4 BE switches, the FE DDL validation and the `score()`
admission rule) was an experiment that never reached a released build,
yet it had grown roots through the whole SNII stack and had diverged the
v1 on-disk layout from what the released 3.1-based writer produces. This
PR removes it end-to-end and realigns the physical format so that a node
running this code can read segments written by the released writer, and
vice versa, without a format version bump.
What changes:
- **Removed**: everything CommonGrams-specific in BE
(`be/src/storage/index/inverted/common_grams/`, the `common_grams` token
filter, the SNII term-key escaping codec, `SniiCommonGramsMetadataPB`,
the hybrid query planner and its cost model, the SPIMI pair-key path,
the frequency-less spill-run shape) and in FE (the `common_grams` token
filter type, the CommonGrams policy validation, the `score()` admission
rule). Loading an existing policy of that type marks it invalid with an
error log instead of crashing FE.
- **Term keys** are now the raw analyzed bytes everywhere. User terms
that overlap the internal `\x1F` phrase-bigram namespace bypass the
index (the same rule the released writer already implied).
- **Scoring (BM25) is kept** and decoupled from CommonGrams: a segment
is scorable iff it has positions *and* norms. Tokenized indexes with
positions now always write norms (an optional region referenced from the
core metadata; no new `index_config`). `tf` is the number of positions
the term has in the document. Older segments without norms answer
`NOT_SUPPORTED` for scoring only; a normal compaction reconstructs norms
from the postings (byte-identical to a rebuild), so they become scorable
after one compaction. The WAND / block-max paths are removed together
with the per-window max statistics they depended on.
- **On-disk layout (S1)**: `kFormatVersion` stays 1. The dict entry /
dict block / frq prelude / frq pod encoders and decoders are
byte-for-byte the released layout again: prelude flag bit0 = `has_prx`,
no freq regions, no per-window `max_freq` / `max_norm`, no per-entry
term statistics, no `frq_docs_len` / `inline_dd_disk_len` / `freq_meta`,
no `kNoTermStats` dict-block flag. `sum_total_term_freq` and the norms
reference live in optional proto fields the released reader ignores.
Segments carrying the experimental CommonGrams metadata fields (or the
experimental scoring `index_config`) are rejected as `NOT_SUPPORTED`
with a rebuild hint.
- Two follow-up fixes that fell out of the layout change: the
phrase-query "position work" estimate now parses prx frame headers
without verifying CRCs (`read_prx_frame(..., verify_crc=false)`), so a
phrase query no longer checksums every candidate prx window twice; and
the frequency-less spill-run record shape is gone (the production SPIMI
path always materializes frequencies).
Compatibility evidence (real binaries, golden corpus of 8 sample indexes
/ 179 query expectations / 513 checks across cache-off, cache-cold and
cache-warm):
- this branch reading segments written by the released 3.1-based writer:
8/8 open, 513/0
- segments written by this branch, checked against the expectations
recorded before the removal: 513/0 (query semantics unchanged)
- the released 3.1-based binary reading segments written by this branch:
8/8 open, 513/0 (rollback-safe: the old reader ignores the norms field)
Size: 178 files, +2321 / −20981 (be/src −8184, be/test −10066, fe −946).
### Release note
The experimental `common_grams` token filter and its scoring tier are
removed. BM25 scoring on a tokenized SNII index now requires the norms
region, which the writer always emits for tokenized indexes with
positions; segments written before this change become scorable after one
compaction.
0 commit comments