## Feature

Linear:
[OSS-1344](https://linear.app/lancedb/issue/OSS-1344/make-fts-index-block-size-configurable)

### What is the new feature?

FTS inverted index creation now accepts a `block_size` parameter for
compressed posting blocks. Supported values are `128` and `256`.

### Why do we need this feature?

The posting block size was previously fixed at `128`, which made the
block-max granularity impossible to tune for different datasets and
query profiles.

### How does it work?

- Adds `block_size` to `InvertedIndexParams`, protobuf details,
posting-list schema metadata, and cache headers.
- Uses `128` as the default for newly created indexes.
- Treats older serialized params, schema metadata, and cache entries
that omit `block_size` as legacy `128`.
- Rejects unsupported values, including `512`, with a clear validation
error.
- Uses Lance-owned `BitPacker4x` for physical 128-value posting blocks
and `BitPacker8x` for physical 256-value posting blocks.
- Marks `block_size=256` as experimental in public API docs because it
may introduce breaking changes.
- Keeps position-stream packing on the legacy 128-value block format.
- Keeps downgrade compatibility tests on explicit legacy
`block_size=128`, since older wheels cannot read current-created
physical 256 FTS posting blocks.
- Threads the configured block size through FTS build, read, iterator,
WAND, cache, and MemWAL flush paths.
- Exposes the parameter in Python and Java FTS index creation APIs, with
docs and focused tests.

## Validation

- `cargo fmt --all`
- `cargo fmt --all --check`
- `git diff --check`
- `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo test -p
lance-index block_size -- --nocapture`
- `CARGO_TARGET_DIR=/tmp/lance-target-a479-no512 cargo clippy -p
lance-index --tests -- -D warnings`
- `uv run make build` from `python/`
- `uv run pytest
python/tests/test_scalar_index.py::test_create_scalar_index_fts_block_size`
from `python/`
- `uv run ruff format --check python/tests/test_scalar_index.py
python/lance/dataset.py` from `python/`
- `uv run ruff check python/tests/test_scalar_index.py
python/lance/dataset.py` from `python/`
- `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p
lance-index block_size -- --nocapture`
- `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p
lance-index test_256_posting_block_uses_single_physical_bitpack_chunk --
--nocapture`
- `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo test -p
lance-bitpacking`
- `CARGO_TARGET_DIR=/tmp/lance-target-a479-merge-main cargo clippy -p
lance-bitpacking -p lance-index --tests -- -D warnings`
- `uv run ruff format --check
python/tests/compat/test_scalar_indices.py` from `python/`
- `uv run ruff check python/tests/compat/test_scalar_indices.py` from
`python/`
- `uv run pytest --run-compat -vvv -s
python/tests/compat/test_scalar_indices.py::test_FtsIndex_downgrade
--durations=30` from `python/`
- `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index
test_new_training_request_defaults_missing_block_size_to_128`
- `CARGO_TARGET_DIR=/tmp/lance-a479-target cargo test -p lance-index
block_size`
- `uv run ruff format --check python/lance/dataset.py` from `python/`
- `uv run ruff check python/lance/dataset.py` from `python/`

Not run locally: Java focused test / spotless check, because this
machine has no Java Runtime installed (`Unable to locate a Java
Runtime`).

---

## Update: all V3 breaking changes consolidated here

Per review direction, every breaking change for the 256-doc block format
now lands in this single PR (the follow-up stack
#7602/#7603/#7604/#7624/#7625/#7629 carries none). On top of the
configurable block size and PFOR frequency encoding, this PR now also
includes:

- **Quantized doc-length scoring (Lucene norm semantics), 256-doc blocks
only.** BM25 doc lengths are quantized to a SmallFloat-style byte code
(4 mantissa bits: 0-7 exact, <= 6.25% relative error, decode = bucket
floor). The byte-norm slab bakes lazily per loaded DocSet and quarters
the doc-length bytes scoring pulls through the cache (200M docs: 800MB
-> 200MB). 128-block indexes keep exact-length scoring bit-for-bit.
Measured top-k overlap vs exact scoring on the (score-clustered,
synthetic) mmlb corpus: 98.1% mean for phrase, 89.7% for 3-word AND;
corpora with more score spread shift less.
- **256-doc posting blocks drop the leading block-max-score f32** (~1.5G
on a 200M-doc index; 131G -> 130G). Block layout: `[first_doc u32][doc
num_bits u8][docs][pfor freqs]`;
`posting_block_score_prefix_len(block_size)` keys every reader/writer.
The impact skip data from the stacked #7602 supplies a tighter per-block
bound; until it lands, 256-block block-max pruning falls back to the
(valid, looser) list-level max score.

**BREAKING:** 256-doc-block (v3) indexes must be rebuilt; v3 is
unreleased so no migration is provided. BM25 scores on v3 differ from
exact-length BM25 by the norm quantization, matching Lucene's norm
semantics. The format discussion #7606 documents the final layout and
scoring semantics.

Additional validation for this update: bulk-vs-classic A/B under
quantized scoring is score-identical (both paths quantize identically);
the full stack's warm benchmarks vs Lucene 10.4 on mmlb-200m: OR k10
0.0249s/318qps and OR k100 0.0467s/170qps (both ahead of Lucene sliced),
AND k10 0.0443s (1.29x), AND k100 0.0883s (1.94x).

---

## Standalone results vs main (per-branch-tip wheels)

**Legacy (128) read-path parity.** Threading a runtime `block_size`
through
`PostingIterator` initially replaced the compile-time `BLOCK_SIZE`
division
(a shift) with real `div` instructions in the `doc()`/`next()` hot
loops,
measured as +11-14% on 3-word OR against the 200M legacy index
(`PostingIterator::next` grew from 16.5% to 23.6% of the profile). Block
sizes are validated powers of two, so the iterator now derives block
indices
with `trailing_zeros` shifts and masks; after that fix the legacy path
is at
parity with main: 3-word OR k10 0.131s (main 0.132-0.134s), k100
0.255-0.256s (main 0.256-0.257s), single-term 0.025s (main 0.027s)
across 3 warm passes on the 200M legacy index, 400G cache.

**block_size=256 index size** (5M-doc controlled build, same wheel,
`with_position=false`): postings shrink **3.33 GiB → 2.62 GiB (−21%)**
from
PFOR frequencies + no per-block max-score prefix + half the block
headers.
Index build 192s → 210s (+10%, PFOR encode cost). Top-10 overlap vs the
128 exact-length scoring on 10 3-word OR queries: 95% mean (5/10
identical
sets; the rest differ by 1-2 near-tie docs, from the quantized-norm
scoring).

**Query wins for 256 land in the stacked PRs.** A 256 index without
impact
skip data prunes on the (valid, looser) list-level max and is *slower*
than
128 — e.g. classic AND k10 0.547s until #7602's impacts restore
block-granular bounds (0.115s), and #7603/#7604/#7624/#7625/#7629 take
the
same index to 0.025s OR k10 / 0.045s AND k10.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configurable FTS posting `block_size` (128/256) to scalar index
creation, including updated examples and APIs.
* Enabled FTS format version 3 (`v3`) for `block_size=256`, with
quantized doc-length scoring for v3.
* **Bug Fixes**
* Enforced `block_size`/`format_version` compatibility (invalid
combinations now error).
* Persisted and restored FTS metadata for format version and posting
block size, with legacy indexes defaulting to `128`.
* **Documentation**
* Updated full-text-search and quickstart guides and parameter docs for
`block_size`, defaults, accepted values, and the experimental `256`/`v3`
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Yang Cen <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>

Port of Lucene's MaxScoreBulkScorer, enabled by default for compatible
top-k OR queries. Set `LANCE_FTS_MAXSCORE=0` to fall back to the classic
WAND loop:
per outer window (bounded by the essential clauses' blocks with adaptive
growth), clauses split into a non-essential prefix and essential rest by
window max score vs the running threshold. Essential clauses bulk-stream
decompressed blocks (single-essential windows stream with no
accumulator);
non-essential clauses are only probed for candidates that can still beat
the threshold. Dead ranges with one live clause skip by scanning the
baked
per-block bound slab. Candidate emission matches the classic path, so
results are score-identical (verified over a 40-case A/B snapshot).

## Measured vs #7602 (its base)

Per-branch-tip wheels, 200M-doc v3-256 index, 1000 3-word OR queries × 8
concurrent, warm, default settings:

| query | #7602 (classic WAND) | this PR (default MAXSCORE) |
|---|---|---|
| OR 3w k10 | 0.103s / 76 qps | **0.034s / 231 qps (3.0×)** |
| OR 3w k100 | 0.198s / 40 qps | **0.063s / 127 qps (3.1×)** |

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Improved full-text search performance via impact-aware scoring and
smarter pruning for compressed posting lists.
* Added an optional bulk MAXSCORE execution path for OR searches,
controlled by `LANCE_FTS_MAXSCORE` (with the prior approach as
fallback).
  * Enhanced OR skip behavior using improved group/block upper bounds.
* **Bug Fixes**
* More robust handling of document-boundary/score limit values, treating
malformed or missing entries safely.
* **Tests**
* Expanded coverage for impact caching, group skipping, and the
optimized search execution path.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Yang Cen <[email protected]>