Fix index out of bounds when writing FST to different metaOut (#12697) #12698
Conversation
lucene/CHANGES.txt
Outdated
|
|
||
| * GITHUB#12571: Fix HNSW graph read bug when built with excessive connections. (Ben Trent). | ||
|
|
||
| * GITHUB#12698: Fix index out of bounds bug when saving FSTStore-backed FST with different DataOutput for metadata |
There was a problem hiding this comment.
Do we really never, in Lucene today, write the FST and its metadata to different files?
There was a problem hiding this comment.
Also (unless you don't want to!), please put your name in the CHANGES entry -- we try to give credit to contributors!
Maybe reword to something like Fix IndexOutOfBoundsException when saving FSTStore-backed FST with different DataOutput for metadata or so?
There was a problem hiding this comment.
Do we really never, in Lucene today, write the FST and its metadata to different files?
It seems we do, in those LuceneXXBlockTreeTermsWriter.TermsWriter.finish(). But they had used the BytesStore-backed FST (FST on-the-fly), so it would not cause the issue either.
I think this issue is to make the two branches more consistent and help to simplify the consolidation of BytesStore and FSTStore.
dungba88
force-pushed
the
fix-numBytes
branch
from
e896985 to
011fe9b
Compare
mikemccand
left a comment
mikemccand
left a comment
There was a problem hiding this comment.
Great catch, sneaky bug! I guess nowhere in Lucene are we using this particular way of saving FST...
I just left small comments about the CHANGES entry.
This can go back to 9.x right?
I think that too. I rebased the change, re-added the assertion and updated the CHANGES log :) |
|
Excellent, thanks @dungba88 -- I'll merge & backport soon. |
|
Thanks @dungba88 -- I merged to |
…ache.org * upstream/main: (239 commits) Bound the RAM used by the NodeHash (sharing common suffixes) during FST compilation (apache#12633) Fix index out of bounds when writing FST to different metaOut (apache#12697) (apache#12698) Avoid object construction when linear searching arcs (apache#12692) chore: update the Javadoc example in Analyzer (apache#12693) coorect position on entry in CHANGES.txt Refactor ByteBlockPool so it is just a "shift/mask big array" (apache#12625) Extract the hnsw graph merging from being part of the vector writer (apache#12657) Specialize `BlockImpactsDocsEnum#nextDoc()`. (apache#12670) Speed up TestIndexOrDocValuesQuery. (apache#12672) Remove over-counting of deleted terms (apache#12586) Use MergeSorter in StableStringSorter (apache#12652) Use radix sort to speed up the sorting of terms in TermInSetQuery (apache#12587) Add timeouts to github jobs. Estimates taken from empirical run times (actions history), with a generous buffer added. (apache#12687) Optimize OnHeapHnswGraph's data structure (apache#12651) Add createClassLoader to replicator permissions (block specific to jacoco). (apache#12684) Move changes entry before backporting CHANGES Move testing properties to provider class (no classloading deadlock possible) and fallback to default provider in non-test mode simple cleanups to vector code (apache#12680) Better detect vector module in non-default setups (e.g., custom module layers) (apache#12677) ...
2 participants
Description
Fix #12697. This will move the writing of numBytes from
OnHeapFSTStore.writeTo()toFST.save().OnHeapFSTStore.size()will also be modified to return only the numBytes.As such, this might change the existing behavior. However searching through the Lucene code,
size()is only referenced forOffHeapFSTStore, which already use a correct implementation. I think it could be backported to 9.9