You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0a3595e
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](index) Bound SNII posting workspace across spill and encoding (apache#67657)
### What problem does this PR solve?
Related PR: apache#67537 (merged into master)
**Under review: the error-cleanup and query-performance items below
remain open.** This branch is rebased onto master at `e429e59bdfc` after
apache#67537 merged. The master comparison now contains only the two
spill-specific commits (46 changed files). The CommonGrams deletion and
released-format alignment are already in master through apache#67537, so those
removed lines do not recur in this PR diff.
Problem Summary:
SNII expands compact postings into complete document, frequency and
position arrays during spill, run merging and final encoding. A frequent
term or a very long document can therefore create a large allocation
while the original posting arena is still live. Limiting the first spill
alone moves the peak into the merge or encoder. Repeatedly merging an
increasing run prefix also amplifies temporary I/O.
This change reserves one hard posting workspace before allocating,
shared across the writer's spill, merge and final encoding:
- Append compact fragments to one spool with independently sealed,
checksummed run ranges. The run directory itself can spill.
- Merge contiguous groups using fan-in constrained by workspace and
available file descriptors. Intermediate passes copy encoded fragments;
final consumption combines boundary documents in their original position
order.
- Replay large position payloads and stage DD, window metadata and
inline DD/PRX within the same budget. Ordinary admitted windows retain
the existing resident encoder.
- Keep input arena, dictionary keys/ranks, norms, NULL metadata,
allocator retention and OS page cache outside this posting-workspace
limit. It is an algorithmic workspace bound, not a whole-BE RSS limit.
- Preserve sequential read-ahead across small-term boundaries. The added
regression originally read 578,549,868 bytes from a 271,632-byte run
because every term discarded the same 64 KiB cache.
The persistent SNII layout and window policy established by apache#67537 are
retained. Persistent readers are unchanged. Large streaming ZSTD frames
can have different compressed bytes, so reader compatibility alone is
not treated as evidence of equal query performance.
0 commit comments