Strange stack traces for new bitset focused doc iterators #14517
Description
Description
With Lucene 10.2, we have seen some exceptions that are rather troubling. It appears that the into bit set code is buggy when utilizing multiple layers of iterators.
It does seem strange that we have ran into this type of failure on a couple of different paths. This indicates a significant API shift (e.g. how iterators need to be handled is now fundamentally different), or some underlying bugs in this new implementation.
Here is one trace. This one seems possible if the iterator has iterated passed windowBase and windowMax in:
DocIdSetIterator lead = iterators.get(0);
if (lead.docID() < windowBase) {
lead.advance(windowBase);
}
lead.intoBitSet(windowMax, windowMatches, windowBase);
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -33 out of bounds for length 64
at [email protected]/org.apache.lucene.util.FixedBitSet.set(FixedBitSet.java:283)
at [email protected]/org.apache.lucene.search.DocIdSetIterator.intoBitSet(DocIdSetIterator.java:268)
at [email protected]/org.apache.lucene.search.comparators.TermOrdValComparator$CompetitiveIterator.intoBitSet(TermOrdValComparator.java:539)
at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindowUsingBitSet(DenseConjunctionBulkScorer.java:242)
at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:210)
at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
at [email protected]/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:460)
at [email protected]/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
at [email protected]/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
at [email protected]/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:111)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:369)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:336)
at [email protected]/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:212)
... 15 more
This following trace seems possible if DISIDocIdStream#count(upto) where upto is > max. Even though we do a check for iterator.docID() >= upTo there is no check to verify we are iterating past the configured max. Maybe we are handling the iterators incorrectly?
Suppressed: java.lang.IndexOutOfBoundsException: Range [0, -2) out of bounds for length 4096
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
at java.base/java.util.Objects.checkFromToIndex(Objects.java:391)
at [email protected]/org.apache.lucene.util.FixedBitSet.cardinality(FixedBitSet.java:212)
at [email protected]/org.apache.lucene.search.DISIDocIdStream.count(DISIDocIdStream.java:63)
at [email protected]/org.apache.lucene.search.DocIdStream.count(DocIdStream.java:50)
at [email protected]/org.apache.lucene.search.TotalHitCountCollector$1.collect(TotalHitCountCollector.java:69)
at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:208)
at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
at [email protected]/org.apache.lucene.search.ReqExclBulkScorer.score(ReqExclBulkScorer.java:66)
at [email protected]/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:460)
at [email protected]/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
at [email protected]/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
at [email protected]/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
... 6 more
Version and environment details
Lucene 10.2