Lucene Change Log

Release 9.4.2 [2022-11-21]

• Bug Fixes   (2)
  1. GITHUB#11905: Fix integer overflow when seeking the vector index for connections in a single segment. This addresses a bug that was introduced in 9.2.0 where having many vectors is not handled well in the vector connections reader.

  2. GITHUB#11939: Fix incorrect cost calculation in DocIdSetBuilder after upgradeToBitSet when doc list is growing. This addresses a bug where the cost of TermRangeQuery/TermInSetQuery and some other queries will be highly underestimated.

• Improvements   (2)
  1. GITHUB#11912, GITHUB#11918: Port generic exception handling from MemorySegmentIndexInput to ByteBufferIndexInput. This also adds the invalid position while seeking or reading to the exception message. Allows better debugging and analysis of bugs like GITHUB#11905.
     (Uwe Schindler, Robert Muir)
  2. GITHUB#11916: improve checkindex to be more thorough for vectors.
     (Ben Trent)

Release 9.4.1 [2022-10-24]

• Bug Fixes   (1)
  1. GITHUB#11858: Fix kNN vectors format validation on large segments. This addresses a regression in 9.4.0 where validation could fail, preventing further writes or searches on the index.
     (Julie Tibshirani)

Release 9.4.0 [2022-09-30]

• API Changes   (1)
  1. LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW vectors
     (Michael Sokolov, Julie Tibshirani)
• New Features   (4)
  1. LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and XYShape.
     (Nick Knize)
  2. LUCENE-10629: Support match set filtering with a query in MatchingFacetSetCounts.
     (Stefan Vodita, Shai Erera)
  3. LUCENE-10633: SortField#setOptimizeSortWithIndexedData and SortField#getOptimizeSortWithIndexedData were introduced to provide an option to disable sort optimization for various sort fields.
     (Mayya Sharipova)
  4. GITHUB#912: Support for Java 19 foreign memory support was added. Applications started with command line parameter "java --enable-preview" will automatically use the new foreign memory API of Java 19 to access indexes on disk with MMapDirectory. This is an opt-in feature and requires explicit Java command line flag! When enabled, Lucene logs a notice using java.util.logging. Please test thoroughly and report bugs/slowness to Lucene's mailing list. When the new API is used, MMapDirectory will mmap Lucene indexes in chunks of 16 GiB (instead of 1 GiB) and indexes closed while queries are running can no longer crash the JVM.
     (Uwe Schindler)
• Improvements   (4)
  1. LUCENE-10592: Build HNSW Graph on indexing.
     (Mayya Sharipova, Adrien Grand, Julie Tibshirani)
  2. LUCENE-10207: TermInSetQuery can now provide a ScoreSupplier with cost estimation, making it usable in IndexOrDocValuesQuery.
     (Greg Miller)
  3. LUCENE-10216: Use MergePolicy to define and MergeScheduler to trigger the reader merges required by addIndexes(CodecReader[]) API.
     (Vigya Sharma, Michael McCandless)
  4. GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf
     (Mike Drob)
• Optimizations   (5)
  1. LUCENE-10661: Reduce memory copy in BytesStore.
     (luyuncheng)
  2. GITHUB#1020: Support #scoreSupplier and small optimizations to DocValuesRewriteMethod.
     (Greg Miller)
  3. LUCENE-10633: Added support for dynamic pruning to queries sorted by a string field that is indexed with terms and SORTED or SORTED_SET doc values.
     (Adrien Grand)
  4. LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on compressing data.
     (luyuncheng)
  5. GITHUB#1062: Optimize TermInSetQuery when a term is present that matches all docs in a segment.
     (Greg Miller)
• Bug Fixes   (7)
  1. LUCENE-10663: Fix KnnVectorQuery explain with multiple segments.
     (Shiming Li)
  2. LUCENE-10673: Improve check of equality for latitudes for spatial3d GeoBoundingBox
     (ignacio Vera)
  3. LUCENE-10678: Fix potential overflow when building a BKD tree with more than 4 billion points. The overflow occurs when computing the partition point.
     (Ignacio Vera)
  4. LUCENE-10644: Facets#getAllChildren testing should ignore child order.
     (Yuting Gan)
  5. LUCENE-10665, GITHUB#11701: Fix classloading deadlock in analysis factories / AnalysisSPILoader initialization.
     (Uwe Schindler)
  6. LUCENE-10674: Ensure BitSetConjDISI returns NO_MORE_DOCS when sub-iterator exhausts.
     (Jack Mazanec)
  7. GITHUB#11794: Guard FieldExistsQuery against null pointers
     (Luca Cavanna)
• Build   (2)
  1. GITHUB#11720: Upgrade randomizedtesting to 2.8.1 (potential fix for odd wall clock - related timeout failures).
     (Dawid Weiss)
  2. LUCENE-10669: The build should be more helpful when generated resources are touched
     (Dawid Weiss)
• Other   (1)
  1. LUCENE-10559: Add Prefilter Option to KnnGraphTester
     (Kaival Parikh)

Release 9.3.0 [2022-07-29]

• API Changes   (2)
  1. LUCENE-10603: SortedSetDocValues#NO_MORE_ORDS marked @deprecated in favor of iterating with SortedSetDocValues#docValueCount().
     (Greg Miller)
  2. GITHUB#978: Deprecate (remove in Lucene 10) obsolete constants in oal.util.Constants; remove code which is no longer executed after Java 9.
     (Uwe Schindler)
• New Features   (4)
  1. LUCENE-10550: Add getAllChildren functionality to facets
     (Yuting Gan)
  2. LUCENE-10274: Added facetsets module for high dimensional (hyper-rectangle) faceting
  3. (Shai Erera, Marc D'Mello, Greg Miller)

  4. LUCENE-10151 Enable timeout support in IndexSearcher.
     (Deepika Sharma)
• Improvements   (5)
  1. LUCENE-10078: Merge on full flush is now enabled by default with a timeout of 500ms.
     (Adrien Grand)
  2. LUCENE-10585: Facet module code cleanup (copy/paste scrubbing, simplification and some very minor optimization tweaks).
     (Greg Miller)
  3. LUCENE-10603: Update SortedSetDocValues iteration to use SortedSetDocValues#docValueCount().
     (Greg Miller, Stefan Vodita)
  4. LUCENE-10619: Optimize the writeBytes in TermsHashPerField.
     (Tang Donghai)
  5. GITHUB#983: AbstractSortedSetDocValueFacetCounts internal code cleanup/refactoring.
     (Greg Miller)
• Optimizations   (11)
  1. LUCENE-8519: MultiDocValues.getNormValues should not call getMergedFieldInfos
     (Rushabh Shah)
  2. GITHUB#961: BooleanQuery can return quick counts for simple boolean queries.
     (Adrien Grand)
  3. LUCENE-10618: Implement BooleanQuery rewrite rules based for minimumShouldMatch.
     (Fang Hou)
  4. LUCENE-10480: Implement Block-Max-Maxscore scorer for 2 clauses disjunction.
     (Zach Chen, Adrien Grand)
  5. LUCENE-10606: For KnnVectorQuery, optimize case where filter is backed by BitSetIterator
     (Kaival Parikh)
  6. LUCENE-10593: Vector similarity function and NeighborQueue reverse removal.
     (Alessandro Benedetti)
  7. GITHUB#984: Use primitive type data structures in FloatTaxonomyFacets and IntTaxonomyFacets #getAllChildren() internal implementation to avoid some garbage creation.
     (Greg Miller)
  8. GITHUB#1010: Specialize ordinal encoding for common case in SortedSetDocValues.
     (Greg Miller)
  9. LUCENE-10657: CopyBytes now saves one memory copy on ByteBuffersDataOutput.
     (luyuncheng)
  10. GITHUB#1007: Optimize IntersectVisitor#visit implementations for certain bulk-add cases.
      (Greg Miller)
  11. LUCENE-10653: BlockMaxMaxscoreScorer uses heapify instead of individual adds.
      (Greg Miller)
• Changes in runtime behavior   (1)
  1. GITHUB#978: IndexWriter diagnostics written to index only contain java's runtime version and vendor.
     (Uwe Schindler)
• Bug Fixes   (13)
  1. LUCENE-10574: Prevent pathological O(N^2) merging.
     (Adrien Grand)
  2. LUCENE-10584: Properly support #getSpecificValue for hierarchical dims in SSDV faceting.
     (Greg Miller)
  3. LUCENE-10582: Fix merging of overridden CollectionStatistics in CombinedFieldQuery
     (Yannick Welsch)
  4. LUCENE-10563: Fix failure to tessellate complex polygon
     (Craig Taverner)
  5. LUCENE-10605: Fix error in 32bit jvm object alignment gap calculation
     (Sun Wuqiang)
  6. GITHUB#956: Make sure KnnVectorQuery applies search boost.
     (Julie Tibshirani)
  7. LUCENE-10598: SortedSetDocValues#docValueCount() should be always greater than zero.
     (Lu Xugang)
  8. LUCENE-10600: SortedSetDocValues#docValueCount should be an int, not long
     (Lu Xugang)
  9. LUCENE-10611: Fix failure when KnnVectorQuery has very selective filter
     (Kaival Parikh)
  10. LUCENE-10607: Fix potential integer overflow in maxArcs computions
      (Tang Donghai)
  11. GITHUB#986: Fix FieldExistsQuery rewrite when all docs have vectors.
      (Julie Tibshirani)
  12. LUCENE-10623: Error implementation of docValueCount for SortingSortedSetDocValues
      (Lu Xugang)
  13. GITHUB#1028: Fix error in TieredMergePolicy
      (Lin Jian)
• Other   (4)
  1. GITHUB#991: Update randomizedtesting to 2.8.0, hppc to 0.9.1, morfologik to 2.1.9.
     (Dawid Weiss)
  2. LUCENE-10370: pass proper classpath/module arguments for forking jvms from within tests.
     (Dawid Weiss)
  3. LUCENE-10604: Improve ability to test and debug triangulation algorithm in Tessellator.
     (Craig Taverner)
  4. GITHUB#922: Remove unused and confusing FacetField indexing options
     (Gautam Worah)
• Build   (1)
  1. GITHUB#976: Exclude Lucene's own JAR files from classpath entries in Eclipse config.
     (Uwe Schindler)

Release 9.2.0 [2022-05-23]

• API Changes   (3)
  1. LUCENE-10325: Facets API extended to support getTopFacets.
     (Yuting Gan)
  2. LUCENE-10482: Allow users to create their own DirectoryTaxonomyReaders with empty taxoArrays instead of letting the taxoEpoch decide. Add a test case that demonstrates the inconsistencies caused when you reuse taxoArrays on older checkpoints.
     (Gautam Worah)
  3. LUCENE-10558: Add new constructors to Kuromoji and Nori dictionary classes to support classpath / module system usage. It is now possible to use JDK's Class/ClassLoader/Module#getResource(...) apis and pass their returned URL to dictionary constructors to load resources from Classpath or Module resources.
     (Uwe Schindler, Tomoko Uchida, Mike Sokolov)
• New Features   (6)
  1. LUCENE-10312: Add PersianStemmer based on the Arabic stemmer.
     (Ramin Alirezaee)
  2. LUCENE-10539: Return a stream of completions from FSTCompletion.
     (Dawid Weiss)
  3. LUCENE-10385: Implement Weight#count on IndexSortSortedNumericDocValuesRangeQuery to speed up computing the number of hits when possible.
     (Lu Xugang, Luca Cavanna, Adrien Grand)
  4. LUCENE-10422: Monitor Improvements: `Monitor` can use a custom `Directory` implementation. `Monitor` can be created with a readonly `QueryIndex` in order to have readonly `Monitor` instances.
     (Niko Usai)
  5. LUCENE-10456: Implement rewrite and Weight#count for MultiRangeQuery by merging overlapping ranges .
     (Jianping Weng)
  6. LUCENE-10444: Support alternate aggregation functions in association facets.
     (Greg Miller)
• Improvements   (6)
  1. LUCENE-10229: return -1 for unknown offsets in ExtendedIntervalsSource. Modify highlighting to work properly with or without offsets.
     (Dawid Weiss)
  2. LUCENE-10494: Implement method to bulk add all collection elements to a PriorityQueue.
     (Bauyrzhan Sakhariyev)
  3. LUCENE-10484: Add support for concurrent random sampling by calling RandomSamplingFacetsCollector#createManager.
     (Luca Cavanna)
  4. LUCENE-10467: Throws IllegalArgumentException for Facets#getAllDims and Facets#getTopChildren if topN <= 0.
     (Yuting Gan)
  5. LUCENE-9848: Correctly sort HNSW graph neighbors when applying diversity criterion
     (Mayya Sharipova, Michael Sokolov)
  6. LUCENE-10527: Use 2*maxConn for the last layer in HNSW
     (Mayya Sharipova)
• Optimizations   (16)
  1. LUCENE-10555: avoid NumericLeafComparator#iteratorCost repeated initialization when NumericLeafComparator#setScorer is called.
     (Jianping Weng)
  2. LUCENE-10452: Hunspell: call checkCanceled less frequently to reduce the overhead
     (Peter Gromov)
  3. LUCENE-10451: Hunspell: don't perform potentially expensive spellchecking after timeout
     (Peter Gromov)
  4. LUCENE-10418: More `Query#rewrite` optimizations for the non-scoring case.
     (Adrien Grand)
  5. LUCENE-10436: Deprecate DocValuesFieldExistsQuery, NormsFieldExistsQuery and KnnVectorFieldExistsQuery with FieldExistsQuery.
     (Zach Chen, Michael McCandless, Adrien Grand)
  6. LUCENE-10481: FacetsCollector will not request scores if it does not use them.
     (Mike Drob)
  7. LUCENE-10503: Potential speedup for pure disjunctions whose clauses produce scores that are very close to each other.
     (Adrien Grand)
  8. LUCENE-10315: Use SIMD instructions to decode BKD doc IDs.
     (Guo Feng, Adrien Grand, Ignacio Vera)
  9. LUCENE-8836: Speed up calls to TermsEnum#lookupOrd on doc values terms enums and sequences of increasing ords.
     (Bruno Roustant, Adrien Grand)
  10. LUCENE-10536: Doc values terms dictionaries now use the first (uncompressed) term of each block as a dictionary when compressing suffixes of the other 63 terms of the block.
      (Adrien Grand)
  11. LUCENE-10411: Add nearest neighbors vectors support to ExitableDirectoryReader.
      (Zach Chen, Adrien Grand, Julie Tibshirani, Tomoko Uchida)
  12. LUCENE-10542: FieldSource exists implementations can avoid value retrieval
      (Kevin Risden)
  13. LUCENE-10534: MinFloatFunction / MaxFloatFunction exists check can be slow
      (Kevin Risden)
  14. LUCENE-10496: Queries sorted by field now better handle the degenerate case when the search order and the index order are in opposite directions.
      (Jianping Weng)
  15. LUCENE-10502: Use IndexedDISI to store docIds and DirectMonotonicWriter/Reader to handle ordToDoc in HNSW vectors
      (Lu Xugang)
  16. LUCENE-10488: Facets#getTopDims optimized for taxonomy faceting and ConcurrentSortedSetDocValuesFacetCounts.
      (Yuting Gan)
• Bug Fixes   (13)
  1. LUCENE-10477: Highlighter: WeightedSpanTermExtractor.extractWeightedSpanTerms to Query#rewrite multiple times if necessary.
     (Christine Poerschke, Adrien Grand)
  2. LUCENE-10491: A correctness bug in the way scores are provided within TaxonomyFacetSumValueSource was fixed.
     (Michael McCandless, Greg Miller)
  3. LUCENE-10466: Ensure IndexSortSortedNumericDocValuesRangeQuery handles sort field types besides LONG
     (Andriy Redko)
  4. LUCENE-10292: Suggest: Fix AnalyzingInfixSuggester / BlendedInfixSuggester to correctly return existing lookup() results during concurrent build(). Fix other FST based suggesters so that getCount() returned results consistent with lookup() during concurrent build().
     (hossman)
  5. LUCENE-10508: Fixes some edge cases where GeoArea were built in a way that vertical planes could not evaluate their sign, either because the planes where the same or the center between those planes was lying in one of the planes.
     (Ignacio Vera)
  6. LUCENE-10495: Fix return statement of siblingsLoaded() in TaxonomyFacets.
     (Yuting Gan)
  7. LUCENE-10533: SpellChecker.formGrams is missing bounds check
     (Kevin Risden)
  8. LUCENE-10529: Properly handle when TestTaxonomyFacetAssociations test case randomly indexes no documents instead of throwing an NPE.
     (Greg Miller)
  9. LUCENE-10470: Check if polygon has been successfully tessellated before we fail (we are failing some valid tessellations) and allow filtering edges that fold on top of the previous one.
     (Ignacio Vera)
  10. LUCENE-10530: Avoid floating point precision test case bug in TestTaxonomyFacetAssociations.
      (Greg Miller)
  11. LUCENE-10552: KnnVectorQuery has incorrect equals/ hashCode.
      (Lu Xugang)
  12. LUCENE-10558: Restore behaviour of deprecated Kuromoji and Nori dictionary constructors for custom dictionary support. Please also use new URL-based constructors for classpath/module system ressources.
      (Uwe Schindler, Tomoko Uchida, Mike Sokolov)
  13. LUCENE-10564: Make sure SparseFixedBitSet#or updates ramBytesUsed.
      (Julie Tibshirani)
• Build   (3)
  1. GITHUB#768: Upgrade forbiddenapis to version 3.3.
     (Uwe Schindler)
  2. GITHUB#890: Detect CI builds on Github or Jenkins and enable errorprone.
     (Uwe Schindler, Dawid Weiss)
  3. LUCENE-10532: Remove LuceneTestCase.Slow annotation. All tests can be fast.
     (Robert Muir)
• Other   (4)
  1. LUCENE-10526: Test-framework: Add FilterFileSystemProvider.wrapPath(Path) method for mock filesystems to override if they need to extend the Path implementation.
     (Gautam Worah, Robert Muir)
  2. LUCENE-10525: Test-framework: Add detection of illegal windows filenames to WindowsFS.
     (Gautam Worah)
  3. LUCENE-10541: Test-framework: limit the default length of MockTokenizer tokens to 255.
     (Robert Muir, Uwe Schindler, Tomoko Uchida, Dawid Weiss)
  4. GITHUB#854: Allow to link to GitHub pull request from CHANGES.
     (Tomoko Uchida, Jan Høydahl)

Release 9.1.0 [2022-03-22]

• API Changes   (16)
  1. LUCENE-10244: MultiCollector::getCollectors is now public, allowing users to access the wrapped collectors.
     (Andriy Redko)
  2. LUCENE-10197: UnifiedHighlighter now has a Builder to construct it. The UH's setters are now deprecated.
     (Animesh Pandey, David Smiley)
  3. LUCENE-10301: the test framework is now a module. All the classes have been moved from org.apache.lucene.* to org.apache.lucene.tests.* to avoid package name conflicts with the core module.
     (Dawid Weiss)
  4. LUCENE-10183: KnnVectorsWriter#writeField to take KnnVectorsReader instead of VectorValues.
     (Zach Chen, Michael Sokolov, Julie Tibshirani, Adrien Grand)
  5. LUCENE-10335: Deprecate helper methods for resource loading in IOUtils and StopwordAnalyzerBase that are not compatible with module system (Class#getResourceAsStream() and Class#getResource() are caller sensitive in Java 11). Instead add utility method IOUtils#requireResourceNonNull(T) to test existence of resource based on null return value.
     (Uwe Schindler, Dawid Weiss)
  6. LUCENE-10349: WordListLoader methods now return unmodifiable CharArraySets.
     (Uwe Schindler)
  7. LUCENE-10377: SortField.getComparator() has changed signature. The second parameter is now a boolean indicating whether or not skipping should be enabled on the comparator.
     (Alan Woodward)
  8. LUCENE-10381: Require users to provide FacetsConfig for SSDV faceting.
     (Greg Miller)
  9. LUCENE-10368: IntTaxonomyFacets has been deprecated and is no longer a supported extension point for user-created faceting implementations.
     (Greg Miller)
  10. LUCENE-10400: Add constructors that take external resource Paths to dictionary classes in Kuromoji and Nori: ConnectionCosts, TokenInfoDictionary, and UnknownDictionary. Old constructors that take resource scheme and resource path in those classes are deprecated; These are replaced with the new constructors and planned to be removed in a future release.
      (Tomoko Uchida, Uwe Schindler, Mike Sokolov)
  11. LUCENE-10050: Deprecate DrillSideways#search(Query, Collector) in favor of DrillSideways#search(Query, CollectorManager). This reflects the change (LUCENE-10002) being made in IndexSearcher#search that trends towards using CollectorManagers over Collectors.
      (Gautam Worah)
  12. LUCENE-10420: Move functional interfaces in IOUtils to top-level interfaces.
      (David Smiley, Uwe Schindler, Dawid Weiss, Tomoko Uchida)
  13. LUCENE-10398: Add static method for getting Terms from LeafReader.
      (Spike Liu)
  14. LUCENE-10440: TaxonomyFacets and FloatTaxonomyFacets have been deprecated and are no longer supported extension points for user-created faceting implementations.
      (Greg Miller)
  15. LUCENE-10431: MultiTermQuery.setRewriteMethod() has been deprecated, and constructor parameters for the various implementations added.
      (Alan Woodward)
  16. LUCENE-10171: OpenNLPOpsFactory.getLemmatizerDictionary(String, ResourceLoader) now returns a DictionaryLemmatizer object instead of a raw String serialization of the dictionary.
      (Spyros Kapnissis via Michael Gibney, Alessandro Benedetti)
• New Features   (19)
  1. LUCENE-10255: Lucene JARs are now proper modules, with module descriptors and dependency information.
     (Chris Hegarty, Uwe Schindler, Tomoko Uchida, Dawid Weiss)
  2. LUCENE-10342: Lucene Core now depends on java.logging (JUL) module and reports if MMapDirectory cannot unmap mapped ByteBuffers or RamUsageEstimator's object size calculations may be off. This was added especially for users running Lucene with the Java Module System where some optional features are not available by default or supported. For all apps using Lucene it is strongly recommended, to explicitely require non-standard JDK modules: jdk.unsupported (unmapping) and jdk.management (OOP size for RAM usage calculatons). It is also recommended to install JUL logging adapters to feed the log events into your app's logging system.
     (Uwe Schindler, Dawid Weiss, Tomoko Uchida, Robert Muir)
  3. LUCENE-10330: Make MMapDirectory tests fail by default, if unmapping does not work.
     (Uwe Schindler, Dawid Weiss)
  4. LUCENE-10223: Add interval function support to StandardQueryParser. Add min-should-match operator support to StandardQueryParser. Update and clean up package documentation in flexible query parser module.
     (Dawid Weiss, Alan Woodward)
  5. LUCENE-10220: Add an utility method to get IntervalSource from analyzed text (or token stream).
     (Uwe Schindler, Dawid Weiss, Alan Woodward)
  6. LUCENE-10085: Added Weight#count on DocValuesFieldExistsQuery to speed up the query if terms or points are indexed.
     (Quentin Pradet, Adrien Grand)
  7. LUCENE-10263: Added Weight#count to NormsFieldExistsQuery to speed up the query if all documents have the field..
     (Alan Woodward)
  8. LUCENE-10248: Add SpanishPluralStemFilter, for precise stemming of Spanish plurals. For more information, see https://s.apache.org/spanishplural
     (Xavier Sanchez Loro)
  9. LUCENE-10243: StandardTokenizer, UAX29URLEmailTokenizer, and HTMLStripCharFilter have been upgraded to Unicode 12.1
     (Robert Muir)
  10. LUCENE-10335: Add ModuleResourceLoader as complement to ClasspathResourceLoader.
      (Uwe Schindler)
  11. LUCENE-10245: MultiDoubleValues(Source) and MultiLongValues(Source) were added as multi-valued versions of DoubleValues(Source) and LongValues(Source) to the facets module. LongValueFacetCounts, LongRangeFacetCounts and DoubleRangeFacetCounts were augmented to support these new multi-valued abstractions. DoubleRange and LongRange also support creating queries from these multi-valued sources.
      (Greg Miller)
  12. LUCENE-10250: Add support for arbitrary length hierarchical SSDV facets.
      (Marc D'mello)
  13. LUCENE-10395: Add support for TotalHitCountCollectorManager, a collector manager based on TotalHitCountCollector that allows users to parallelize counting the number of hits.
      (Luca Cavanna, Adrien Grand)
  14. LUCENE-10403: Add ArrayUtil#grow(T[]).
      (Greg Miller)
  15. LUCENE-10414: Add fn:fuzzyTerm interval function to flexible query parser
      (Dawid Weiss, Alan Woodward)
  16. LUCENE-10378: Implement Weight#count for PointRangeQuery to provide a faster way to calculate the number of matching range docs when each doc has at-most one point and the points are 1-dimensional.
      (Gautam Worah, Ignacio Vera, Adrien Grand)
  17. LUCENE-10415: FunctionScoreQuery and IndexOrDocValuesQuery delegate Weight#count.
      (Ignacio Vera)
  18. LUCENE-10382: Add support for filtering in KnnVectorQuery. This allows for finding the nearest k documents that also match a query.
      (Julie Tibshirani, Joel Bernstein)
  19. LUCENE-10237: Add MergeOnFlushMergePolicy to sandbox.
      (Michael Froh, Anand Kotriwal)
• Improvements   (9)
  1. LUCENE-10313: use java util logging in Luke. Add dynamic log filtering. Drop the persistent log previously written to ~/.luke.d/luke.log. Configure Java's default logging handlers to persist Luke logs according to your needs.
     (Tomoko Uchida, Dawid Weiss)
  2. LUCENE-10238: Upgrade icu4j dependency to 70.1.
     (Dawid Weiss)
  3. LUCENE-9820: Extract BKD tree interface and move intersecting logic to the PointValues abstract class.
     (Ignacio Vera, Adrien Grand)
  4. LUCENE-10262: Lift up restrictions for navigating PointValues#PointTree added in LUCENE-9820
     (Ignacio Vera)
  5. LUCENE-9538: Detect polygon self-intersections in the Tessellator.
     (Ignacio Vera)
  6. LUCENE-10275: Speed up MultiRangeQuery by using an interval tree.
     (Ignacio Vera)
  7. LUCENE-10229: Unify behaviour of match offsets for interval queries on fields with or without offsets enabled.
     (Patrick Zhai)
  8. LUCENE-10054 Make HnswGraph hierarchical
     (Mayya Sharipova, Julie Tibshirani, Mike Sokolov, Adrien Grand)
  9. LUCENE-10371: Make IndexRearranger able to arrange segment in a determined order.
     (Patrick Zhai)
• Optimizations   (20)
  1. LUCENE-10329: Use computed block mask for DirectMonotonicReader#get.
     (Guo Feng)
  2. LUCENE-10280: Optimize BKD leaves' doc IDs codec when they are continuous.
     (Guo Feng)
  3. LUCENE-10233: Store BKD leaves' doc IDs as bitset in some cases (typically for low cardinality fields or sorted indices) to speed up addAll.
     (Guo Feng, Adrien Grand)
  4. LUCENE-10225: Improve IntroSelector with 3-ways partitioning.
     (Bruno Roustant, Adrien Grand)
  5. LUCENE-10321: Tweak MultiRangeQuery interval tree creation to skip "pulling up" mins.
     (Greg Miller)
  6. LUCENE-10252: ValueSource.asDoubleValues and asLongValues should not compute the score unless asked to -- typically never. This fixes a performance regression since 7.3 LUCENE-8099 when some older boosting queries were replaced with this.
     (David Smiley)
  7. LUCENE-10346: Optimize facet counting for single-valued TaxonomyFacetCounts.
     (Guo Feng)
  8. LUCENE-10356: Further optimize facet counting for single-valued TaxonomyFacetCounts.
     (Greg Miller)
  9. LUCENE-10379: Count directly into the dense values array in FastTaxonomyFacetCounts#countAll.
     (Guo Feng, Greg Miller)
  10. LUCENE-10375: Speed up HNSW vectors merge by first writing combined vector data to a file.
      (Julie Tibshirani, Adrien Grand)
  11. LUCENE-10388: Remove MultiLevelSkipListReader#SkipBuffer to make JVM less confused.
      (Guo Feng)
  12. LUCENE-10367: Optimize CoveringQuery for the case when the minimum number of matching clauses is a constant.
      (LuYunCheng via Adrien Grand)
  13. LUCENE-10412: More `Query#rewrite` optimizations for MatchNoDocsQuery.
      (Adrien Grand)
  14. LUCENE-10408 Better encoding of doc Ids in vectors.
      (Mayya Sharipova, Julie Tibshirani, Adrien Grand)
  15. LUCENE-10424, LUCENE-10439: Optimize the "everything matches" case for count query in PointRangeQuery.
      (Ignacio Vera, Lu Xugang)
  16. LUCENE-10084, LUCENE-10435: Rewrite DocValuesFieldExistsQuery to MatchAllDocsQuery whenever terms or points have a docCount that is equal to maxDoc.
      (Vigya Sharma, Lu Xugang)
  17. LUCENE-10442: When indexQuery or/and dvQuery be a MatchAllDocsQuery then IndexOrDocValuesQuery should be rewrite to MatchAllDocsQuery.
      (Lu Xugang)
  18. LUCENE-10450: IndexSortSortedNumericDocValuesRangeQuery could be rewrite to MatchAllDocsQuery.
      (Lu Xugang)
  19. LUCENE-10453: Indexing and search speedup with KNN vectors when using euclidean distance.
      (Adrien Grand)
  20. LUCENE-10455: IndexSortSortedNumericDocValuesRangeQuery now implements the scorerSupplier API.
      (Lu Xugang)
• Changes in runtime behavior   (2)
  1. LUCENE-10291: Lucene now only writes files for terms and postings if at least one field is indexed with postings.
     (Yannick Welsch)
  2. LUCENE-10311: FixedBitSet#approximateCardinality now trades accuracy for speed instead of delegating to FixedBitSet#cardinality.
     (Robert Muir, Adrien Grand)
• Bug Fixes   (16)
  1. LUCENE-10316: fix TestLRUQueryCache.testCachingAccountableQuery failure.
     (Patrick Zhai)
  2. LUCENE-10279: Fix equals in MultiRangeQuery.
     (Ignacio Vera)
  3. LUCENE-10349: Fix all analyzers to behave according to their documentation: getDefaultStopSet() methods now return unmodifiable CharArraySets.
     (Uwe Schindler)
  4. LUCENE-10352: Add missing service provider entries: KoreanNumberFilterFactory, DaitchMokotoffSoundexFilterFactory
     (Uwe Schindler, Robert Muir)
  5. LUCENE-10352: Fixed ctor argument checks: JapaneseKatakanaStemFilter, DoubleMetaphoneFilter
     (Uwe Schindler, Robert Muir)
  6. LUCENE-10236: Stop duplicating norms when scoring in CombinedFieldQuery.
     (Zach Chen, Jim Ferenczi, Julie Tibshirani)
  7. LUCENE-10353: Add random null injection to TestRandomChains.
     (Robert Muir, Uwe Schindler)
  8. LUCENE-10377: CheckIndex could incorrectly throw an error when checking index sorts defined on older indexes.
     (Alan Woodward)
  9. LUCENE-9952: Address inaccurate dim counts for SSDV faceting in cases where a dim is configured as multi-valued.
     (Greg Miller)
  10. LUCENE-10401: Fix lookups on empty doc-value terms dictionaries to no longer throw an ArrayIndexOutOfBoundsException.
      (Adrien Grand)
  11. LUCENE-10402: Prefix intervals should declare their automaton as binary, otherwise prefixes containing multibyte characters will not correctly match.
      (Alan Woodward)
  12. LUCENE-10407: Containing intervals could sometimes yield incorrect matches when wrapped in a disjunction.
      (Alan Woodward, Dawid Weiss)
  13. LUCENE-10405: When using the MemoryIndex, binary and Sorted doc values are stored as BytesRef instead of BytesRefHash so they don't have a limit on size.
      (Ignacio Vera)
  14. LUCENE-10428: Queries with a misbehaving score function may no longer cause infinite loops in their parent BooleanQuery.
      (Ankit Jain, Daniel Doubrovkine, Adrien Grand)
  15. LUCENE-10431: MultiTermQuery no longer includes its rewrite method in its hashcode calculation, as this could cause problems with wrapper queries like BooleanQuery which expect their child queries hashcodes to be stable.
      (Alan Woodward)
  16. LUCENE-10469: Fix ScoreMode propagation by ConstantScoreQuery.
      (Adrien Grand)
• Other   (7)
  1. LUCENE-10273: Deprecate SpanishMinimalStemFilter in favor of SpanishPluralStemFilter.
     (Robert Muir)
  2. LUCENE-10284: Upgrade morfologik-stemming to 2.1.8.
     (Dawid Weiss)
  3. LUCENE-10310: TestXYDocValuesQueries#doRandomDistanceTest does not produce random circles with radius with '0' value any longer.

  4. LUCENE-10352: Removed duplicate instances of StringMockResourceLoader and migrated class to test-framework.
     (Uwe Schindler, Robert Muir)
  5. LUCENE-10352: Convert TestAllAnalyzersHaveFactories and TestRandomChains to a global integration test and discover classes to check from module system. The test now checks all analyzer modules, so it may discover new bugs outside of analysis:common module.
     (Uwe Schindler, Robert Muir)
  6. LUCENE-10413: Make Ukrainian default stop words list available as a public getter.
     (Alan Woodward)
  7. LUCENE-10437: Polygon tessellator throws a more informative error message when the provided polygon does not contain enough no-collinear points.
     (Ignacio Vera)

Release 9.0.0 [2021-12-07]

• New Features   (8)
  1. LUCENE-9322, LUCENE-9855: Vector-valued fields, Lucene90 Codec
     (Mike Sokolov, Julie Tibshirani, Tomoko Uchida)
  2. LUCENE-9004, LUCENE-10040: Approximate nearest vector search via NSW graphs
     (Mike Sokolov, Tomoko Uchida et al.)
  3. LUCENE-9659: SpanPayloadCheckQuery now supports inequalities.
     (Kevin Watters, Gus Heck)
  4. LUCENE-9589: Swedish Minimal Stemmer
     (janhoy)
  5. LUCENE-9313: Add SerbianAnalyzer based on the snowball stemmer.
     (Dragan Ivanovic)
  6. LUCENE-10095: Add NepaliAnalyzer based on the snowball stemmer.
     (Robert Muir)
  7. LUCENE-10096: Add TamilAnalyzer based on the snowball stemmer.
     (Robert Muir)
  8. LUCENE-10102: Add JapaneseCompletionFilter for Input Method-aware auto-completion
     (Tomoko Uchida, Robert Muir, Jun Ohtani)
• System Requirements   (1)
  1. LUCENE-8738: Move to Java 11 as minimum Java version.
     (Adrien Grand, Uwe Schindler)
• API Changes   (44)
  1. LUCENE-8638: Remove many deprecated methods and classes including FST.lookupByOutput(), LegacyBM25Similarity and Jaspell suggester.

  2. LUCENE-8982: Separate out native code to another module to allow cpp build with gradle. This also changes the name of the native "posix-support" library to LuceneNativeIO.
     (Zachary Chen, Dawid Weiss)
  3. LUCENE-9562: All binary analysis packages (and corresponding Maven artifacts) with names containing '-analyzers-' have been renamed to '-analysis-'.
     (Dawid Weiss)
  4. LUCENE-8474: RAMDirectory and associated deprecated classes have been removed.
     (Dawid Weiss)
  5. LUCENE-3041: The deprecated Weight#extractTerms() method has been removed
     (Alan Woodward, Simon Willnauer, David Smiley, Luca Cavanna)
  6. LUCENE-8805: StoredFieldVisitor#stringField now takes a String rather than a byte[] that stores the UTF-8 bytes of the stored string.
     (Namgyu Kim via Adrien Grand)
  7. LUCENE-8811: BooleanQuery#setMaxClauseCount() and #getMaxClauseCount() have moved to IndexSearcher. The checks are now implemented using a QueryVisitor and apply to all queries, rather than only booleans.
     (Atri Sharma, Adrien Grand, Alan Woodward)
  8. LUCENE-8909: The deprecated IndexWriter#getFieldNames() method has been removed.
     (Adrien Grand, Munendra S N)
  9. LUCENE-8948: Change "name" argument in ICU factories to "form". Here, "form" is named after "Unicode Normalization Form".
     (Tomoko Uchida)
  10. LUCENE-8933: Validate JapaneseTokenizer user dictionary entry.
      (Tomoko Uchida)
  11. LUCENE-8905: Better defence against malformed arguments in TopDocsCollector
      (Atri Sharma)
  12. LUCENE-9089: FST Builder renamed FSTCompiler with fluent-style Builder.
      (Bruno Roustant)
  13. LUCENE-9212: Deprecated Intervals.multiterm() methods that take a bare Automaton have been removed
      (Alan Woodward)
  14. LUCENE-9264: SimpleFSDirectory has been removed in favor of NIOFSDirectory.
      (Yannick Welsch)
  15. LUCENE-9281: Use java.util.ServiceLoader to load codec components and analysis factories to be compatible with Java Module System. This allows to load factories without META-INF/service from a Java module exposing the factory in the module descriptor. This breaks backwards compatibility as custom analysis factories must now also implement the default constructor (see MIGRATE.md).
      (Uwe Schindler, Dawid Weiss)
  16. LUCENE-9307: BufferedIndexInput#setBufferSize has been removed.
      (Adrien Grand)
  17. LUCENE-9340: SimpleBindings#add(SortField) has been removed.
      (Alan Woodward)
  18. LUCENE-9462: Fields without positions should still return MatchIterator.
      (Alan Woodward, Dawid Weiss)
  19. LUCENE-9516: Removed the ability to replace the IndexingChain / DocConsumer in Lucenes IndexWriter. The interface is not sufficient to efficiently replace the functionality with reasonable efforts.
      (Simon Willnauer)
  20. LUCENE-9317 LUCENE-9318 LUCENE-9319 LUCENE-9558 LUCENE-9600 : Clean up package name conflicts between modules. See MIGRATE.md for details.
      (David Ryan, Tomoko Uchida, Uwe Schindler, Dawid Weiss)
  21. LUCENE-9646: Set BM25Similarity discountOverlaps via the constructor
      (Patrick Marty via Bruno Roustant)
  22. LUCENE-9480: Make DataInput's skipBytes(long) abstract as the implementation was not performant. IndexInput's api is unaffected: skipBytes() is implemented via seek().
      (Greg Miller)
  23. LUCENE-9796: SortedDocValues no longer extends BinaryDocValues, as binaryValue() was not performant. See MIGRATE.md for details.
      (Robert Muir)
  24. LUCENE-9853: JapaneseAnalyzer should use CJKWidthCharFilter for full-width and half-width character normalization.
      (Tomoko Uchida)
  25. LUCENE-9387: Removed CodecReader#ramBytesUsed.
      (Adrien Grand)
  26. LUCENE-9334: Require consistency between data-structures on a per-field basis. A field across all documents within an index must be indexed with the same index options and data-structures. As a consequence of this, doc values updates are only applicable for fields that are indexed with doc values only.
      (Mayya Sharipova, Adrien Grand, Simon Willnauer)
  27. LUCENE-9047: Directory API is now little endian.
      (Ignacio Vera, Adrien Grand)
  28. LUCENE-9948: No longer require the user to specify whether-or-not a field is multi-valued in LongValueFacetCounts (detect automatically based on what is indexed).
      (Greg Miller)
  29. LUCENE-9843: Remove compression option on default codec's docvalues.
      (Jack Conradson)
  30. LUCENE-9204: SpanQuery and its subclasses have been moved from core/ into the queries/ module.
      (Alan Woodward)
  31. LUCENE-9454: Analyzer no longer has a mutable version field.
      (Alan Woodward)
  32. LUCENE-9956: Expose the getBaseQuery, getDrillDownQueries APIs from DrillDownQuery
      (Gautam Worah)
  33. LUCENE-8143: SpanBoostQuery has been removed.
      (Alan Woodward)
  34. LUCENE-9998: Remove unused parameter fis in StoredFieldsWriter.finish() and TermVectorsWriter.finish(), including those subclasses.
      (kkewwei)
  35. LUCENE-7020: TieredMergePolicy#setMaxMergeAtOnceExplicit has been removed. TieredMergePolicy no longer sets a limit on the maximum number of segments that can be merged at once via a forced merge.
      (Adrien Grand, Shawn Heisey)
  36. LUCENE-10027: Directory reader open API from indexCommit and leafSorter has been modified to add an extra parameter - minSupportedMajorVersion.
      (Mayya Sharipova)
  37. LUCENE-9620: Added a (sometimes) faster implementation for IndexSearcher#count that relies on the new Weight#count API. The Weight#count API represents a cleaner way for Query classes to optimize their counting method.
      (Gautam Worah, Adrien Grand)
  38. LUCENE-10089: Add a method to SortField that allows to enable or disable numeric sort optimization to use the points index to skip over non-competitive documents, which is enabled by default from 9.0
      (Mayya Sharipova, Adrien Grand)
  39. LUCENE-10115: Add an extension point, BaseQueryParser#getFuzzyDistance, to allow custom query parsers to determine the similarity distance for fuzzy queries.
      (Chris Hegarty)
  40. LUCENE-10132: Support addition of diagnostics by custom merge policies
      (Chris Hegarty)
  41. LUCENE-9325: Sort is now final, and the `setSort()` method has been removed
      (Alan Woodward)
  42. LUCENE-9431: The UnifiedHighlighter's WEIGHT_MATCHES flag is now set by default, provided its requirements are met. It can be disabled via over-riding getFlags
      (Animesh Pandey, David Smiley)
  43. LUCENE-10158: Add a new interface Unwrappable to the utils package to allow code to unwrap wrappers/delegators that are added by Lucene's testing framework. This will allow testing new MMapDirectory implementation based on JDK Project Panama.
      (Uwe Schindler)
  44. LUCENE-10260: LucenePackage class has been removed. The implementation string can be retrieved from Version.getPackageImplementationVersion().
      (Uwe Schindler, Dawid Weiss)
• Improvements   (48)
  1. LUCENE-10234: Added Automatic-Module-Name to all JARs. This is the first step to enable full Java module system (JMS) support in later Lucene versions. At the moment, the automatic names should not be considered stable.
     (Dawid Weiss, Uwe Schindler)
  2. LUCENE-10182: TestRamUsageEstimator used RamUsageTester.sizeOf throughout, making some of the tests trivial. Now, it compares results from RamUsageEstimator with those from RamUsageTester. To prevent this error in the future, RamUsageTester.sizeOf was renamed to ramUsed.
     (Uwe Schindler, Dawid Weiss, Stefan Vodita)
  3. LUCENE-10129: RamUsageEstimator overloads the shallowSizeOf method for primitive arrays to avoid falling back on shallowSizeOf(Object), which could lead to performance traps.
     (Robert Muir, Uwe Schindler, Stefan Vodita)
  4. LUCENE-10139: ExternalRefSorter returns a covariant with a subtype of BytesRefIterator that is Closeable.
     (Dawid Weiss).

  5. LUCENE-10135: Correct passage selector behavior for long matching snippets
     (Dawid Weiss).

  6. LUCENE-9960: Avoid unnecessary top element replacement for equal elements in PriorityQueue.
     (Dawid Weiss)
  7. LUCENE-9633: Improve match highlighter behavior for degenerate intervals (on non-existing positions).
     (Dawid Weiss)
  8. LUCENE-9618: Do not call IntervalIterator.nextInterval after NO_MORE_DOCS is returned.
     (Patrick Zhai)
  9. LUCENE-9576: Improve ConcurrentMergeScheduler settings by default, assuming modern I/O. Previously Lucene was too conservative, jumping through hoops to detect if disks were SSD-backed. In many common modern cases (VMs, RAID arrays, containers, encrypted mounts, non-Linux OS), the pessimistic heuristics were wrong, resulting in slower indexing performance. Heuristics were also complex and would trigger JDK issues even on unrelated mount points. Merge scheduler defaults are now modernized and the heuristics removed. Users with spinning disks that want to maximize I/O performance should tweak ConcurrentMergeScheduler.
     (Robert Muir)
  10. LUCENE-9463: Query match region retrieval component, passage scoring and formatting for building custom highlighters.
      (Alan Woodward, Dawid Weiss)
  11. LUCENE-9370: RegExp query is no longer lenient about inappropriate backslashes and follows the Java Pattern policy for rejecting illegal syntax.
      (Mark Harwood)
  12. LUCENE-9336: RegExp query now supports \w \W \d \D \s \S expressions. This is a break with previous behaviour where these were (mis)interpreted as literally the characters w W d etc.
      (Mark Harwood)
  13. LUCENE-8757: When provided with an ExecutorService to run queries across multiple threads, IndexSearcher now groups small segments together, up to 250k docs per slice.
      (Atri Sharma via Adrien Grand)
  14. LUCENE-8857: Introduce Custom Tiebreakers in TopDocs.merge for tie breaking on docs on equal scores. Also, remove the ability of TopDocs.merge to set shard indices
      (Atri Sharma, Adrien Grand, Simon Willnauer)
  15. LUCENE-8958: Shared count early termination for relevance sorted indices
      (Atri Sharma)
  16. LUCENE-8937: Avoid aggressive stemming on numbers in the FrenchMinimalStemmer.
      (Adrien Gallou via Tomoko Uchida)
  17. LUCENE-8596: Kuromoji user dictionary now accepts entries containing hash mark (#) that were previously treated as beginning a line-ending comment
      (Satoshi Kato and Masaru Hasegawa via Michael Sokolov)
  18. LUCENE-9109: Use StackWalker to implement TestSecurityManager's detection of JVM exit
      (Uwe Schindler)
  19. LUCENE-9110: Refactor stack analysis in tests to use generalized LuceneTestCase methods that use StackWalker
      (Uwe Schindler)
  20. LUCENE-9206: IndexMergeTool gets additional options to control the merging. This tool no longer forceMerge(1)s to a single segment by default. If you rely upon this behavior, pass -max-segments 1 instead.
      (Robert Muir)
  21. LUCENE-9220: Upgrade snowball to 2.0. New snowball stemmers: Hindi, Indonesian, Nepali, Serbian, and Tamil. New stoplist: Indonesian. Adds gradle 'snowball' task to regenerate and ease future upgrades.
      (Robert Muir, Dawid Weiss)
  22. LUCENE-9354: Improvements to snowball french stopwords list, so that it is less aggressive.
      (Philippe Ouellet)
  23. LUCENE-9114: Improve ValueSourceScorer's Default Cost Implementation
      (Atri Sharma, David Smiley)
  24. LUCENE-9074: Introduce Slice Executor For Dynamic Runtime Execution Of Slices
      (Atri Sharma)
  25. LUCENE-9280: Add an ability for field comparators to skip non-competitive documents. Creating a TopFieldCollector with totalHitsThreshold less than Integer.MAX_VALUE instructs Lucene to skip non-competitive documents whenever possible. For numeric sort fields the skipping functionality works when the same field is indexed both with doc values and points. In this case, there is an assumption that the same data is stored in these points and doc values
      (Mayya Sharipova, Jim Ferenczi, Adrien Grand)
  26. LUCENE-9449: Enhance DocComparator to provide an iterator over competitive documents when searching with "after". This iterator can quickly position on the desired "after" document skipping all documents and segments before "after". Also redesign numeric comparators to provide skipping functionality by default.
      (Mayya Sharipova, Jim Ferenczi)
  27. LUCENE-9527: Upgrade javacc to 7.0.4, regenerate query parsers.
      (Dawid Weiss)
  28. LUCENE-9531: Consolidated CharStream and FastCharStream classes: these have been moved from each query parser package to org.apache.lucene.queryparser.charstream
      (Dawid Weiss).

  29. LUCENE-9450: Use BinaryDocValues for the taxonomy index instead of StoredFields. Add backwards compatibility tests for the taxonomy index.
      (Gautam Worah, Michael McCandless)
  30. LUCENE-9605: Update snowball to d8cf01ddf37a, adds Yiddish stemmer.
      (Robert Muir)
  31. LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO flag, and rename to DirectIODirectory (Zach Chen, Uwe Schindler, Mike McCandless, Dawid Weiss).

  32. LUCENE-9674: Implement faster advance on VectorValues using binary search.
      (Anand Kotriwal, Mike Sokolov)
  33. LUCENE-9794: Speed up implementations of DataInput.skipBytes().
      (Greg Miller)
  34. LUCENE-9898: Removes no longer used scorePayload method from BM25Similarity
      (Pieter van Boxtel)
  35. LUCENE-9850: Switch to PFOR encoding for doc IDs (instead of FOR).
      (Greg Miller)
  36. LUCENE-9929: Add NorwegianNormalizationFilter, which does the same as ScandinavianNormalizationFilter except it does not fold oo->ø and ao->å.
      (janhoy, Robert Muir, Adrien Grand)
  37. LUCENE-9535: Improve DocumentsWriterPerThreadPool to prefer larger instances.
      (Adrien Grand)
  38. LUCENE-10000: MultiCollectorManager now has parity with MultiCollector with respect to how it handles CollectionTerminationException and setMinCompetitiveScore calls.
      (Greg Miller)
  39. LUCENE-10019: Align file starts in CFS files to have proper alignment (8 bytes)
      (Uwe Schinder)
  40. LUCENE-9662: Make CheckIndex concurrent by parallelizing index check across segments.
      (Zach Chen, Mike McCandless, Dawid Weiss, Robert Muir)
  41. LUCENE-9476: Add new getBulkPath API to DirectoryTaxonomyReader to more efficiently retrieve FacetLabels for multiple facet ordinals at once. This API is 2-4% faster than iteratively calling getPath. The getPath API now throws an IAE instead of returning null if the ordinal is out of bounds.
      (Gautam Worah, Mike McCandless)
  42. LUCENE-10113: Use VarHandles to access int/long/short primitive types in byte arrays. This improves readability and performance of encoding/decoding of primitives to index file format in input/output classes like DataInput / DataOutput and codecs.
      (Uwe Schindler, Robert Muir)
  43. LUCENE-10112: Improve LZ4 Compression performance with direct primitive read/writes.
      (Tim Brooks, Uwe Schindler, Robert Muir, Adrien Grand)
  44. LUCENE-10125: Optimize primitive writes in OutputStreamIndexOutput.
      (Uwe Schindler, Robert Muir, Adrien Grand)
  45. LUCENE-10143: Delegate primitive writes in RateLimitedIndexOutput.
      (Uwe Schindler, Robert Muir, Adrien Grand)
  46. LUCENE-10145, LUCENE-10153: Faster flushes and merges of points by leveraging VarHandles.
      (Adrien Grand)
  47. LUCENE-10201: Spatial-Extras: Upgrading Spatial4j to 0.8 improving a varitety of minor things. See release notes. https://github.com/locationtech/spatial4j/releases/tag/spatial4j-0.8
      (David Smiley)
  48. LUCENE-10062: Switch taxonomy faceting to use numeric doc values for storing ordinals instead of binary doc values with its own custom encoding.
      (Greg Miller)
• Bug fixes   (15)
  1. LUCENE-9686: Fix read past EOF handling in DirectIODirectory.
     (Zach Chen, Julie Tibshirani)
  2. LUCENE-8663: NRTCachingDirectory.slowFileExists may open a file while it's inaccessible.
     (Dawid Weiss)
  3. LUCENE-9117: RamUsageEstimator hangs with AOT compilation. Removed any attempt to estimate Long.valueOf cache size.
     (Cleber Muramoto, Dawid Weiss)
  4. LUCENE-9290: Don't assume that different XYPoint have different hash code
     (Ignacio Vera via Mike Drob)
  5. LUCENE-9372: Fix paths for cygwin/msys before gradle wrapper jar lookup.
     (Peter Barna)
  6. LUCENE-9365: FuzzyQuery was missing matches when prefix length was equal to the term length
     (Mark Harwood, Mike Drob)
  7. LUCENE-9580: Fix bug in the polygon tessellator when introducing collinear edges during polygon splitting.
     (Ignacio Vera)
  8. LUCENE-9930: The Ukrainian analyzer was reloading its dictionary for every new TokenStreamComponents, which could lead to memory leaks.
     (Alan Woodward)
  9. LUCENE-9940: The order of disjuncts in DisjunctionMaxQuery does not matter for equality checks
     (Alan Woodward)
  10. LUCENE-9971: Requesting facet counts for unseen dimensions in SortedSetDocValueFacetCounts and ConcurrentSortedSetDocValueFacetCounts now returns null / -1 instead of throwing IllegalArgumentException as per Javadoc spec in Facets.
      (Alexander Lukyanchikov)
  11. LUCENE-9823: Prevent unsafe rewrites for SynonymQuery and CombinedFieldQuery. Before, rewriting could slightly change the scoring when weights were specified.
      (Naoto Minami via Julie Tibshirani)
  12. LUCENE-10047: Fix a value de-duping bug in LongValueFacetCounts and RangeFacetCounts
      (Greg Miller)
  13. LUCENE-10101, LUCENE-9281: Use getField() instead of getDeclaredField() to minimize security impact by analysis SPI discovery.
      (Uwe Schindler)
  14. LUCENE-10114: Remove unused byte order mark in Lucene90PostingsWriter. This was initially introduced by accident in Lucene 8.4.
      (Uwe Schindler)
  15. LUCENE-10140: Fix cases where minimizing interval iterators could return incorrect matches
      (Nikolay Khitrin, Alan Woodward)
• Changes in Backwards Compatibility Policy   (3)
  1. LUCENE-9904: regenerated UAX29URLEmailTokenizer and the corresponding analyzer with up-to-date top level domains. This may change the token sequence compared to previous Lucene versions.
     (Dawid Weiss)
  2. LUCENE-9669: DirectoryReader#open now accepts an argument to open indices created with versions older than N-1. Lucene now can open indices created with a major version of N-2 in read-only mode. Opening an index created with a major version of N-2 with an IndexWriter is not supported. Further does lucene only support file-format compatibilty which enables reading of old indices while semantic changes like analysis or certain encoding on top of the file format are only supported on a best effort basis.
     (Simon Willnauer)
  3. LUCENE-10232: Fix MultiRangeQuery to confirm all dimensions for a given range match.
     (Greg Miller)
• Build   (6)
  1. LUCENE-9077 LUCENE-9433: Support Gradle build, remove Ant support from trunk
     (Dawid Weiss, Erick Erickson, Uwe Schindler et.al.)
  2. LUCENE-8768: Fix Javadocs build in Java 11.
     (Namgyu Kim)
  3. LUCENE-9544: add regenerate gradle script for nori dictionary
     (Namgyu Kim)
  4. LUCENE-10195: Add gradle cache option and make some tasks cacheable.
     (Jerome Prinet, Dawid Weiss)
  5. LUCENE-10198: LUCENE-10198: Allow external JAVA_OPTS in gradlew scripts; use sane defaults
     ([email protected], Dawid Weiss)
  6. LUCENE-10163: Move LICENSE and NOTICE files to top level to satisfy src artifact requirements
     (janhoy)
• Other   (20)
  1. LUCENE-10122: Use NumericDocValues to store taxonomy parent array
     (Patrick Zhai)
  2. LUCENE-10136: allow 'var' declarations in source code
     (Dawid Weiss)
  3. LUCENE-9570, LUCENE-9564: Apply google java format and enforce it on source Java files. Review diffs and correct automatic formatting oddities.
     (Erick Erickson, Bruno Roustant, Dawid Weiss)
  4. LUCENE-9631: Properly override slice() on subclasses of OffsetRange.
     (Dawid Weiss)
  5. LUCENE-9391: Upgrade HPPC to 0.8.2.
     (Patrick Zhai)
  6. LUCENE-10021: Upgrade HPPC to 0.9.0. Replace usage of ...ScatterMap to ...HashMap.
     (Patrick Zhai)
  7. LUCENE-9092: upgrade randomizedtesting to 2.7.5
     (Dawid Weiss)
  8. LUCENE-8656: Deprecations in FuzzyQuery and get compiler warnings out of queryparser code
     (Alan Woodward, Erick Erickson)
  9. LUCENE-9344: Convert .txt files to properly formatted .md files.
     (Tomoko Uchida, Uwe Schindler)
  10. LUCENE-9267: Update MatchingQueries documentation to correct time unit.
      (Pierre-Luc Perron via Mike Drob)
  11. LUCENE-9411: Fail compilation on warnings, 9x gradle-only (Erick Erickson, Dawid Weiss) Deserves mention here as well as Lucene CHANGES.txt since it affects both.

  12. LUCENE-9215: Replace checkJavaDocs.py with doclet
      (Robert Muir, Dawid Weiss, Uwe Schindler)
  13. LUCENE-9497: Integrate Error Prone, a static analysis tool during compilation
      (Dawid Weiss, Varun Thacker)
  14. LUCENE-9627: Remove unused Lucene50FieldInfosFormat codec and small refactor some codecs to separate reading header/footer from reading content of the file.
      (Ignacio Vera)
  15. LUCENE-9773: Upgrade icu to 68.2
      (Robert Muir)
  16. LUCENE-9822: Add assertion to PFOR exception encoding, documenting the BLOCK_SIZE assumption.
      (Greg Miller)
  17. LUCENE-9883: Turn on ecj missingEnumCaseDespiteDefault setting.
      (Zach Chen)
  18. LUCENE-9705: Make new versions of all index formats for the Lucene90 codec and move the existing ones to the backwards codecs.
      (Julie Tibshirani, Ignacio Vera)
  19. LUCENE-9907: Remove dependency on PackedInts#getReader() from the current codecs and move the method to backwards codec.
      (Ignacio Vera)
  20. LUCENE-10024: Catch NoSuchFileException when opening index directory with Luke.
      (Michael Wechner, Tomoko Uchida)

Release 8.11.2 [2022-06-17]

• Bug Fixes   (2)
  1. LUCENE-10564: Make sure SparseFixedBitSet#or updates ramBytesUsed.
     (Julie Tibshirani)
  2. LUCENE-10477: Highlighter: WeightedSpanTermExtractor.extractWeightedSpanTerms to Query#rewrite multiple times if necessary.
     (Christine Poerschke, Adrien Grand)
• Optimizations   (1)
  1. LUCENE-10481: FacetsCollector will not request scores if it does not use them.
     (Mike Drob)

Release 8.11.1 [2021-12-17]

• Bug Fixes   (1)
  1. SOLR-15843: Update Log4J to 2.16
     (Mike Drob, janhoy)

Release 8.11.0 [2021-11-16]

• API Changes   (1)
  1. (No changes)
• New Features   (1)
  1. (No changes)
• Improvements   (2)
  1. LUCENE-9662: Make CheckIndex concurrent by parallelizing index check across segments.
     (Zach Chen, Mike McCandless, Dawid Weiss, Robert Muir)
  2. LUCENE-10103: Make QueryCache respect Accountable queries.
     (Patrick Zhai)
• Optimizations   (2)
  1. LUCENE-9673: Substantially improve RAM efficiency of how MemoryIndex stores postings in memory, and reduced a bit of RAM overhead in IndexWriter's internal postings book-keeping
     (mashudong)
  2. LUCENE-10196: Improve IntroSorter with 3-ways partitioning.
     (Bruno Roustant)
• Bug Fixes   (6)
  1. LUCENE-10111: Missing calculating the bytes used of DocsWithFieldSet in NormValuesWriter.
     (Lu Xugang)
  2. LUCENE-10116: Missing calculating the bytes used of DocsWithFieldSet and currentValues in SortedSetDocValuesWriter.
     (Lu Xugang)
  3. LUCENE-10070 Skip deleted docs when accumulating facet counts for all docs.
     (Ankur Goel, Greg Miller)
  4. LUCENE-10134: ConcurrentSortedSetDocValuesFacetCounts shouldn't share liveDocs Bits across threads.
     (Ankur Goel)
  5. LUCENE-10154: NumericLeafComparator to define getPointValues.
     (Mayya Sharipova, Adrien Grand)
  6. LUCENE-10208: Ensure that the minimum competitive score does not decrease in concurrent search.
     (Jim Ferenczi, Adrien Grand)
• Build   (1)
  1. LUCENE-10104, SOLR-15631: Upgrade forbiddenapis to version 3.2.
     (Uwe Schindler)
• Other   (1)
  1. LUCENE-10098: Add docs/links to GermanAnalyzer describing how to decompound nouns.
     (Robert Muir)

Release 8.10.1 [2021-10-18]

• Bug Fixes   (3)
  1. LUCENE-10110: MultiCollector now handles single leaf collector that wants to skip low-scoring hits but the combined score mode doesn't allow it.
     (Jim Ferenczi)
  2. LUCENE-10119: Sort optimization with search_after can wrongly skip documents whose values are equal to the last value of the previous page
     (Nhat Nguyen)
  3. LUCENE-10126: Sort optimization with a chunked bulk scorer can wrongly skip documents
     (Nhat Nguyen, Mayya Sharipova)

Release 8.10.0 [2021-09-27]

• API Changes   (5)
  1. LUCENE-9962: DrillSideways allows sub-classes to provide "drill down" FacetsCollectors. They may provide a null collector if they choose to bypass "drill down" facet collection.
     (Greg Miller)
  2. LUCENE-9902: Change the getValue method from IntTaxonomyFacets to be protected instead of private. Users can now access the count of an ordinal directly without constructing an extra FacetLabel. Also use variable length arguments for the getOrdinal call in TaxonomyReader.
     (Gautam Worah)
  3. LUCENE-10036: Replaced the ScoreCachingWrappingScorer ctor with a static factory method that ensures unnecessary wrapping doesn't occur.
     (Greg Miller)
  4. LUCENE-10027: Add a new Directory reader open API from indexCommit and a custom comparator for sorting leaf readers.
     (Mayya Sharipova)
  5. LUCENE-7020: TieredMergePolicy#setMaxMergeAtOnceExplicit is deprecated and the number of segments that get merged via explicit merges is unlimited by default.
     (Adrien Grand, Shawn Heisey)
• New Features   (2)
  1. LUCENE-10083: Analyzer and stemmer for Telugu language
     (Vinod Singh)
  2. LUCENE-10035: The SimpleText codec now writes skip lists.
     (wuda via Adrien Grand)
• Improvements   (12)
  1. LUCENE-9944: Allow DrillSideways users to provide their own CollectorManager without also requiring them to provide an ExecutorService.
     (Greg Miller)
  2. LUCENE-9946: Support for multi-value fields in LongRangeFacetCounts and DoubleRangeFacetCounts.
     (Greg Miller)
  3. LUCENE-9965: Added QueryProfilerIndexSearcher and ProfilerCollector to support debugging query execution strategy and timing.
     (Jack Conradson, Julie Tibshirani)
  4. LUCENE-9981: Operations.getCommonSuffix/Prefix(Automaton) is now much more efficient, from a worst case exponential down to quadratic cost in the number of states + transitions in the Automaton. These methods no longer use the costly determinize method, removing the risk of TooComplexToDeterminizeException
     (Robert Muir, Mike McCandless)
  5. LUCENE-9981: Operations.determinize now throws TooComplexToDeterminizeException based on too much "effort" spent determinizing rather than a precise state count on the resulting returned automaton, to better handle adversarial cases like det(rev(regexp("(.*a){2000}"))) that spend lots of effort but result in smallish eventual returned automata.
     (Robert Muir, Mike McCandless)
  6. LUCENE-9983: Stop sorting determinize powersets unnecessarily.
     (Patrick Zhai)
  7. LUCENE-9177: ICUNormalizer2CharFilter no longer requires normalization-inert characters as boundaries for incremental processing, vastly improving worst-case performance.
     (Michael Gibney)
  8. LUCENE-10030: Lazily evaluate score in DrillSidewaysScorer.doQueryFirstScoring
     (Grigoriy Troitskiy)
  9. LUCENE-9945: Extend DrillSideways to support exposing FacetCollectors directly.
     (Greg Miller, Sejal Pawar)
  10. LUCENE-10043: Decrease default for LRUQueryCache's skipCacheFactor to 10. This prevents caching a query clause when it is much more expensive than running the top-level query.
      (Julie Tibshirani)
  11. LUCENE-5309: Optimize facet counting for single-valued SSDV / StringValueFacetCounts.
      (Greg Miller)
  12. LUCENE-9917: The BEST_SPEED compression mode now trades more compression ratio in exchange of faster reads.
      (Adrien Grand)
• Optimizations   (4)
  1. LUCENE-9996: Improved memory efficiency of IndexWriter's RAM buffer, in particular in the case of many fields and many indexing threads.
     (Adrien Grand)
  2. LUCENE-10022: Rewrite empty DisjunctionMaxQuery to MatchNoDocsQuery.
     (David Harsha via Julie Tibshirani)
  3. LUCENE-10031: Slightly faster segment merging for sorted indices.
     (Adrien Grand)
  4. LUCENE-10014: Lucene90DocValuesFormat was using too many bits per value when compressing via gcd, unnecessarily wasting index storage.
     (weizijun)
• Bug Fixes   (12)
  1. LUCENE-9988: Fix DrillSideways correctness bug introduced in LUCENE-9944
     (Greg Miller)
  2. LUCENE-9964: Duplicate long values in a document field should only be counted once when using SortedNumericDocValuesFields
     (Gautam Worah)
  3. LUCENE-9999: CombinedFieldQuery can fail with an exception when document is missing some fields.
     (Jim Ferenczi, Julie Tibshirani)
  4. LUCENE-10020: DocComparator should not skip docs with the same docID on multiple sorts with search after
     (Mayya Sharipova, Julie Tibshirani)
  5. LUCENE-10026: Fix CombinedFieldQuery equals and hashCode, which ensures query rewrites don't drop CombinedFieldQuery clauses.
     (Julie Tibshirani)
  6. LUCENE-10039: Correct CombinedFieldQuery scoring when there is a single field.
     (Julie Tibshirani)
  7. LUCENE-10046: Counting bug fixed in StringValueFacetCounts.
     (Greg Miller)
  8. LUCENE-9963: FlattenGraphFilter is now more robust when handling incoming holes in the input token graph
     (Geoff Lawson)
  9. LUCENE-10008: Respect ignoreCase in CommonGramsFilterFactory
     (Vigya Sharma)
  10. LUCENE-10060: Ensure DrillSidewaysQuery instances never get cached.
      (Greg Miller, Zachary Chen)
  11. LUCENE-10081: KoreanTokenizer should check the max backtrace gap on whitespaces.
      (Jim Ferenczi)
  12. LUCENE-10106: Sort optimization can wrongly skip the first document of each segment
      (Nhat Nguyen)
• Other   (1)
  1. (No changes)

Release 8.9.0 [2021-06-16]

• API Changes   (1)
  1. LUCENE-9680: IndexWriter#getFieldNames() method added to get fields present in index. This method was removed in LUCENE-8909.
     (Oren Ovadia)
• New Features   (8)
  1. LUCENE-9507: Custom order for leaves in IndexReader and IndexWriter
     (Mayya Sharipova, Mike McCandless, Jim Ferenczi)
  2. LUCENE-9575: PatternTypingFilter has been added to allow setting a type attribute on tokens based on a configured set of regular expressions
     (Gus Heck).

  3. LUCENE-9572: TypeAsSynonymFilter has been enhanced support ignoring some types, and to allow the generated synonyms to copy some or all flags from the original token
     (Gus Heck).

  4. LUCENE-9574 A token filter to drop tokens that match all specified flags.
     (Gus Heck, Uwe Schindler)
  5. LUCENE-9537: Added smoothingScore method and default implementation to Scorable abstract class. The smoothing score allows scorers to calculate a score for a document where the search term or subquery is not present. The smoothing score acts like an idf so that documents that do not have terms or subqueries that are more frequent in the index are not penalized as much as documents that do not have less frequent terms or subqueries and prevents scores which are the product or terms or subqueries from going to zero. Added the implementation of the Indri AND and the IndriDirichletSimilarity from the academic Indri search engine: http://www.lemurproject.org/indri.php.
     (Cameron VandenBerg)
  6. LUCENE-9694: New tool for creating a deterministic index to enable benchmarking changes on a consistent multi-segment index even when they require re-indexing.
     (Patrick Zhai)
  7. LUCENE-9385: Add FacetsConfig option to control which drill-down terms are indexed for a FacetLabel
     (Zachary Chen)
  8. LUCENE-9950: New facet counting implementation for general string doc value fields (SortedSetDocValues / SortedDocValues) not created through FacetsConfig
     (Greg Miller)
• Improvements   (5)
  1. LUCENE-9725: BM25FQuery was extended to handle similarities beyond BM25Similarity. It was renamed to CombinedFieldQuery to reflect its more general scope.
     (Julie Tibshirani)
  2. LUCENE-9663: Adding compression to terms dict from SortedSet/Sorted DocValues.
     (Jaison Bi via Bruno Roustant)
  3. LUCENE-9687: Hunspell support improvements: add API for spell-checking and suggestions, support compound words, fix various behavior differences between Java and C++ implementations, improve performance
     (Peter Gromov, Dawid Weiss)
  4. LUCENE-9877: Reduce index size by increasing allowable exceptions in PForUtil from 3 to 7.
     (Greg Miller)
  5. LUCENE-9935: Enable bulk merge for stored fields with index sort.
     (Robert Muir, Adrien Grand, Nhat Nguyen)
• Optimizations   (2)
  1. LUCENE-9932: Performance improvement for BKD index building
     (neoremind)
  2. LUCENE-9827: Speed up merging of stored fields and term vectors for smaller segments.
     (Daniel Mitterdorfer, Dimitrios Liapis, Adrien Grand, Robert Muir)
• Bug Fixes   (6)
  1. LUCENE-9791: BytesRefHash.equals/find is now thread safe, fixing a Luwak/Monitor bug causing registered queries to sometimes fail to match.
     (Paweł Bugalski)
  2. LUCENE-9887: Fixed parameter use in RadixSelector.
     (liupanfeng via Adrien Grand)
  3. LUCENE-9958: Fixed performance regression for boolean queries that configure a minimum number of matching clauses.
     (Adrien Grand, Matt Weber)
  4. LUCENE-9953: LongValueFacetCounts should count each document at most once when determining the total count for a dimension. Prior to this fix, multi-value docs could contribute a > 1 count to the dimension count.
     (Greg Miller)
  5. LUCENE-9967: Do not throw NullPointerException while trying to handle another exception in ReplicaNode.start
     (Steven Schlansker)
  6. LUCENE-9991: Fix edge case failure in TestStringValueFacetCounts
     (Greg Miller)
• Other   (4)
  1. LUCENE-9836: Removed the pure Maven build. It is no longer possible to build artifacts using Maven (this feature was no longer working correctly). Due to migration to Gradle for Lucene/Solr 9.0, the maintenance of the Maven build was no longer reasonable. POM files are generated for deployment to Maven Central only. Please use "ant generate-maven-artifacts" to produce and deploy artifacts to any repository.
     (Uwe Schindler, Dawid Weiss)
  2. LUCENE-9836: Migrate Maven tasks to use "maven-resolver-ant-tasks" instead of the no longer maintained "maven-ant-tasks".
     (Uwe Schindler)
  3. LUCENE-9985: Upgrade jetty to 9.4.41
     (janhoy)
  4. LUCENE-9976: Fix WANDScorer assertion error.
     (Zach Chen, Adrien Grand, Dawid Weiss)

Release 8.8.2 [2021-04-12]

• Bug Fixes   (3)
  1. LUCENE-9870: Fix Circle2D intersectsLine t-value (distance) range clamp
     (Jørgen Nystad)
  2. LUCENE-9744: NPE on a degenerate query in MinimumShouldMatchIntervalsSource $MinimumMatchesIterator.getSubMatches().
     (Alan Woodward)
  3. LUCENE-9762: DoubleValuesSource.fromQuery (also used by FunctionScoreQuery.boostByQuery) could throw an exception when the query implements TwoPhaseIterator and when the score is requested repeatedly.
     (David Smiley, hossman)

Release 8.8.1 [2021-02-22]

• Bug Fixes   (1)
  1. (No changes)

Release 8.8.0 [2021-02-01]

• New Features   (5)
  1. LUCENE-9552: New LatLonPoint query that accepts an array of LatLonGeometries.
     (Ignacio Vera)
  2. LUCENE-9641: LatLonPoint query support for spatial relationships.
     (Ignacio Vera)
  3. LUCENE-9553: New XYPoint query that accepts an array of XYGeometries.
     (Ignacio Vera)
  4. LUCENE-9378: Doc values now allow configuring how to trade compression for retrieval speed.
     (Adrien Grand)
  5. LUCENE-9413: Add CJKWidthCharFilter and its factory
     (Tomoko Uchida)
• Improvements   (3)
  1. LUCENE-9455: ExitableTermsEnum should sample timeout and interruption check before calling next().
     (Zach Chen via Bruno Roustant)
  2. LUCENE-9023: GlobalOrdinalsWithScore should not compute occurrences when the provided min is 1.
     (Jim Ferenczi)
  3. LUCENE-9675: Binary doc values fields now expose their configured compression mode in the attributes of the field info.
     (Jim Ferenczi)
• Optimizations   (4)
  1. LUCENE-9536: Reduced memory usage for OrdinalMap when a segment has all values.
     (Julie Tibshirani via Adrien Grand)
  2. LUCENE-9021: QueryParser: re-use the LookaheadSuccess exception.
     (Przemek Bruski via Mikhail Khludnev)
  3. LUCENE-9636: Faster decoding of postings for some numbers of bits per value.
     (Guo Feng via Adrien Grand)
  4. LUCENE-9346: WANDScorer now supports queries that have a `minimumNumberShouldMatch` configured.
     (Xi Zachary Chen via Adrien Grand)
• Bug Fixes   (8)
  1. LUCENE-9508: DocumentsWriter was only stalling threads for 1 second allowing documents to be indexed even the DocumentsWriter wasn't able to keep up flushing. Unless IW can't make progress due to an ill behaving DWPT this issue was barely noticeable.
     (Simon Willnauer)
  2. LUCENE-9581: Japanese tokenizer should discard the compound token instead of disabling the decomposition of long tokens when discardCompoundToken is activated.
     (Jim Ferenczi)
  3. LUCENE-9595: Make Component2D#withinPoint implementations consistent with ShapeQuery logic.
     (Ignacio Vera)
  4. LUCENE-9606: Wrap boolean queries generated by shape fields with a Constant score query.
     (Ignacio Vera)
  5. LUCENE-9635: BM25FQuery - Mask encoded norm long value in array lookup.
     (Yilun Cui)
  6. LUCENE-9617: Fix per-field memory leak in IndexWriter.deleteAll(). Reset next available internal field number to 0 on FieldInfos.clear(), to avoid wasting FieldInfo references.
     (Michael Froh)
  7. LUCENE-9642: When encoding triangles in ShapeField, make sure generated triangles are CCW by rotating triangle points before checking triangle orientation.
     (Ignacio Vera)
  8. LUCENE-9661: Fix deadlock in TermsEnum.EMPTY that occurs when trying to initialize TermsEnum and BaseTermsEnum at the same time
     (Namgyu Kim)
• Other   (2)
  1. SOLR-14995: Update Jetty to 9.4.34
     (Mike Drob)
  2. LUCENE-9637: Removes some unused code and replaces the Point implementation on ShapeField/ShapeQuery random tests.
     (Ignacio Vera)

Release 8.7.0 [2020-11-03]

• API Changes   (2)
  1. LUCENE-9437: Lucene's facet module's DocValuesOrdinalsReader.decode method is now public, making it easier for applications to decode facet ordinals into their corresponding labels
     (Ankur Goel)
  2. LUCENE-9515: IndexingChain now accepts individual primitives rather than a DocumentsWriterPerThread instance in order to create a new DocConsumer.
     (Simon Willnauer)
• New Features   (4)
  1. LUCENE-9386: RegExpQuery added case insensitive matching option.
     (Mark Harwood)
  2. LUCENE-8962: Add IndexWriter merge-on-refresh feature to selectively merge small segments on getReader, subject to a configurable timeout, to improve search performance by reducing the number of small segments for searching.
     (Simon Willnauer)
  3. LUCENE-9484: Allow sorting an index after it was created. With SortingCodecReader, existing unsorted segments can be wrapped and merged into a fresh index using IndexWriter#addIndices API.
     (Simon Willnauer, Adrien Grand)
  4. LUCENE-9444: Add utility class to retrieve facet labels from the taxonomy index for a facet field so such fields do not also have to be redundantly stored
     (Ankur Goel)
• Improvements   (10)
  1. LUCENE-8574: Add a new ExpressionValueSource which will enforce only one value per name per hit in dependencies, ExpressionFunctionValues will no longer recompute already computed values
     (Patrick Zhai)
  2. LUCENE-9416: Fix CheckIndex to print an invalid non-zero norm as unsigned long when detecting corruption.

  3. LUCENE-9440: FieldInfo#checkConsistency called twice from Lucene50(60)FieldInfosFormat#read; Removed the (redundant?) assert and do these checks for real.
     (Yauheni Putsykovich)
  4. LUCENE-9446: In BooleanQuery rewrite, always remove MatchAllDocsQuery filter clauses when possible.
     (Julie Tibshirani)
  5. LUCENE-9501: Improve coverage for Asserting* test classes: make sure to handle singleton doc values, and sometimes exercise Weight#scorer instead of Weight#bulkScorer for top-level queries.
     (Julie Tibshirani)
  6. LUCENE-9511: Include StoredFieldsWriter in DWPT accounting to ensure that it's heap consumption is taken into account when IndexWriter stalls or should flush DWPTs.
     (Simon Willnauer)
  7. LUCENE-9514: Include TermVectorsWriter in DWPT accounting to ensure that it's heap consumption is taken into account when IndexWriter stalls or should flush DWPTs.
     (Simon Willnauer)
  8. LUCENE-9523: In query shapes over shape fields, skip points while traversing the BKD tree when the relationship with the document is already known.
     (Ignacio Vera)
  9. LUCENE-9539: Use more compact datastructures to represent sorted doc-values in memory when sorting a segment before flush and in SortingCodecReader.
     (Simon Willnauer)
  10. LUCENE-9458: WordDelimiterGraphFilter should order tokens at the same position by endOffset to emit longer tokens first. The same graph is produced.
      (David Smiley)
• Optimizations   (4)
  1. LUCENE-9395: ConstantValuesSource now shares a single DoubleValues instance across all segments
     (Tony Xu)
  2. LUCENE-9447, LUCENE-9486: Stored fields now get higer compression ratios on highly compressible data.
     (Adrien Grand)
  3. LUCENE-9373: FunctionMatchQuery now accepts a "matchCost" optimization hint.
     (Maxim Glazkov, David Smiley)
  4. LUCENE-9510: Indexing with an index sort is now faster by not compressing temporary representations of the data.
     (Adrien Grand)
• Bug Fixes   (6)
  1. LUCENE-9427: Fix a regression where the unified highlighter didn't produce highlights on fuzzy queries that correspond to exact matches.
     (Julie Tibshirani)
  2. LUCENE-9467: Fix NRTCachingDirectory to use Directory#fileLength to check if a file already exists instead of opening an IndexInput on the file which might throw a AccessDeniedException in some Directory implementations.
     (Simon Willnauer)
  3. LUCENE-9501: Fix a bug in IndexSortSortedNumericDocValuesRangeQuery where it could violate the DocIdSetIterator contract.
     (Julie Tibshirani)
  4. LUCENE-9401: Include field in ComplexPhraseQuery's toString()
     (Thomas Hecker via Munendra S N)
  5. LUCENE-9578: Fix TermRangeQuery when there is no upper bound and the lower bound is the empty string excluded. This would previously match no strings at all while it should match all non-empty strings.
     (Christoph Buescher via Adrien Grand)
  6. LUCENE-9524: Fix NPE in SpanWeight#explain when no scoring is required and SpanWeight has null Similarity.SimScorer.
     (Zach Chen)
• Documentation   (1)
  1. LUCENE-9424: Add a performance warning to AttributeSource.captureState javadocs
     (Patrick Zhai)
• Changes in Runtime Behavior   (1)
  1. LUCENE-9539: SortingCodecReader now doesn't cache doc values fields anymore. Previously, SortingCodecReader used to cache all doc values fields after they were loaded into memory. This reader should only be used to sort segments after the fact using IndexWriter#addIndices.
     (Simon Willnauer)
• Other   (3)
  1. LUCENE-9292: Refactor BKD point configuration into its own class.
     (Ignacio Vera)
  2. LUCENE-9470: Make TestXYMultiPolygonShapeQueries more resilient for CONTAINS queries.
     (Ignacio Vera)
  3. LUCENE-9512: Move LockFactory stress test to be a unit/integration test.
     (Uwe Schindler, Dawid Weiss, Robert Muir)
• Build   (1)
  1. Upgrade forbiddenapis to version 3.1.
     (Uwe Schindler)

Release 8.6.3 [2020-10-07]

• Bug Fixes   (1)
  1. (No changes)

Release 8.6.2 [2020-09-01]

• Bug Fixes   (1)
  1. LUCENE-9478: Prevent DWPTDeleteQueue from referencing itself and leaking memory. The queue passed an implicit this reference to the next queue instance on flush which leaked about 500byte of memory on each full flush, commit or getReader call.
     (Simon Willnauer)

Release 8.6.1 [2020-08-13]

• Bug Fixes   (1)
  1. LUCENE-9443: The UnifiedHighlighter was closing the underlying reader when there were multiple term-vector fields. This was a regression in 8.6.0.
     (David Smiley, Chris Beer)

Release 8.6.0 [2020-07-15]

• API Changes   (9)
  1. LUCENE-9265: SimpleFSDirectory is deprecated in favor of NIOFSDirectory.
     (Yannick Welsch)
  2. LUCENE-9304: Removed ability to set DocumentsWriterPerThreadPool on IndexWriterConfig. The DocumentsWriterPerThreadPool is a packaged protected final class which made it impossible to customize.
     (Simon Willnauer)
  3. LUCENE-9339: MergeScheduler#merge doesn't accept a parameter if a new merge was found anymore.
     (Simon Willnauer)
  4. LUCENE-9330: SortFields are now responsible for writing themselves into index headers if they are used as index sorts.
     (Alan Woodward, Uwe Schindler, Adrien Grand)
  5. LUCENE-9340: Deprecate SimpleBindings#add(SortField).
     (Alan Woodward)
  6. LUCENE-9345: MergeScheduler is now decoupled from IndexWriter. Instead it accepts a MergeSource interface that offers the basic methods to acquire pending merges, run the merge and do accounting around it.
     (Simon Willnauer)
  7. LUCENE-9349: QueryVisitor.consumeTermsMatching() now takes a Supplier<ByteRunAutomaton> to enable queries that build large automata to provide them lazily. TermsInSetQuery switches to using this method to report matching terms.
     (Alan Woodward)
  8. LUCENE-9366: DocValues.emptySortedNumeric() not longer takes a maxDoc parameter
     (Alan Woodward)
  9. LUCENE-7822: CodecUtil#checkFooter(IndexInput, Throwable) now throws a CorruptIndexException if checksums mismatch or if checksums can't be verified.
     (Martin Amirault, Adrien Grand)
• New Features   (2)
  1. LUCENE-7889: Grouping by range based on values from DoubleValuesSource and LongValuesSource
     (Alan Woodward)
  2. LUCENE-8962: Add IndexWriter merge-on-commit feature to selectively merge small segments on commit, subject to a configurable timeout, to improve search performance by reducing the number of small segments for searching
     (Michael Froh, Mike Sokolov, Mike Mccandless, Simon Willnauer)
• Improvements   (13)
  1. LUCENE-9276: Use same code-path for updateDocuments and updateDocument in IndexWriter and DocumentsWriter.
     (Simon Willnauer)
  2. LUCENE-9279: Update dictionary version for Ukrainian analyzer to 4.9.1
     (Andriy Rysin via Dawid Weiss)
  3. LUCENE-8050: PerFieldDocValuesFormat should not get the DocValuesFormat on a field that has no doc values.
     (David Smiley, Juan Rodriguez)
  4. LUCENE-9304: Removed ThreadState abstraction from DocumentsWriter which allows pooling of DWPT directly and improves the approachability of the IndexWriter code.
     (Simon Willnauer)
  5. LUCENE-9324: Add an ID to SegmentCommitInfo in order to compare commits for equality and make snapshots incremental on generational files.
     (Simon Willnauer, Mike Mccandless, Adrien Grand)
  6. LUCENE-9342: TotalHits' relation will be EQUAL_TO when the number of hits is lower than TopDocsColector's numHits
     (Tomás Fernández Löbbe)
  7. LUCENE-9353: Metadata of the terms dictionary moved to its own file, with the `.tmd` extension. This allows checksums of metadata to be verified when opening indices and helps save seeks when opening an index.
     (Adrien Grand)
  8. LUCENE-9359: SegmentInfos#readCommit now always returns a CorruptIndexException if the content of the file is invalid.
     (Adrien Grand)
  9. LUCENE-9393: Make FunctionScoreQuery use ScoreMode.COMPLETE for creating the inner query weight when ScoreMode.TOP_DOCS is requested.
     (Tomás Fernández Löbbe)
  10. LUCENE-9392: Make FacetsConfig.DELIM_CHAR publicly accessible
      (Ankur Goel)
  11. LUCENE-9397: UniformSplit supports encodable fields metadata.
      (Bruno Roustant)
  12. LUCENE-9396: Improved truncation detection for points.
      (Adrien Grand, Robert Muir)
  13. LUCENE-9402: Let MultiCollector handle minCompetitiveScore
      (Tomás Fernández Löbbe, Adrien Grand)
• Optimizations   (8)
  1. LUCENE-9254: UniformSplit keeps FST off-heap.
     (Bruno Roustant)
  2. LUCENE-8103: DoubleValuesSource and QueryValueSource now use a TwoPhaseIterator if one is provided by the Query.
     (Michele Palmia, David Smiley)
  3. LUCENE-9287: UsageTrackingQueryCachingPolicy no longer caches DocValuesFieldExistsQuery.
     (Ignacio Vera)
  4. LUCENE-9286: FST.Arc.BitTable reads directly FST bytes. Arc is lightweight again and FSTEnum traversal faster.
     (Bruno Roustant)
  5. LUCENE-7788: fail precommit on unparameterised log messages and examine for wasted work/objects
     (Erick Erickson)
  6. LUCENE-9273: Speed up geometry queries by specialising Component2D spatial operations. Instead of using a generic relate method for all relations, we use specialize methods for each one. In addition, the type of triangle is computed at deserialization time, therefore we can be more selective when decoding points of a triangle.
     (Ignacio Vera)
  7. LUCENE-9087: Build always trees with full leaves and lower the default value for maxPointsPerLeafNode to 512.
     (Ignacio Vera)
  8. LUCENE-9148: Points now write their index in a separate file.
     (Adrien Grand)
• Bug Fixes   (14)
  1. LUCENE-9259: Fix wrong NGramFilterFactory argument name for preserveOriginal option
     (Paul Pazderski)
  2. LUCENE-8849: DocValuesRewriteMethod.visit wasn't visiting its embedded query
     (Michele Palmia, David Smiley)
  3. LUCENE-9258: DocTermsIndexDocValues assumed it was operating on a SortedDocValues (single valued) field when it could be multi-valued used with a SortedSetSelector
     (Michele Palmia)
  4. LUCENE-9164: Ensure IW processes all internal events before it closes itself on a rollback.
     (Simon Willnauer, Nhat Nguyen, Dawid Weiss, Mike Mccandless)
  5. LUCENE-8908: Return default value from objectVal when doc doesn't match the query in QueryValueSource
     (Bill Bell, hossman, Munendra S N, Michele Palmia)
  6. LUCENE-9133: Fix for potential NPE in TermFilteredPresearcher for empty fields
     (Marvin Justice via Mike Drob)
  7. LUCENE-9309: Wait for #addIndexes merges when aborting merges.
     (Simon Willnauer)
  8. LUCENE-9337: Ensure CMS updates it's thread accounting datastructures consistently. CMS today releases it's lock after finishing a merge before it re-acquires it to update the thread accounting datastructures. This causes threading issues where concurrently finishing threads fail to pick up pending merges causing potential thread starvation on forceMerge calls.
     (Simon Willnauer)
  9. LUCENE-9314: Single-document monitor runs were using the less efficient MultiDocumentBatch implementation.
     (Pierre-Luc Perron, Alan Woodward)
  10. LUCENE-9362: Fix equality check in ExpressionValueSource#rewrite. This fixes rewriting of inner value sources.
      (Dmitry Emets)
  11. LUCENE-9405: IndexWriter incorrectly calls closeMergeReaders twice when the merged segment is 100% deleted.
      (Michael Froh, Simon Willnauer, Mike Mccandless, Mike Sokolov)
  12. LUCENE-9400: Tessellator might build illegal polygons when several holes share the shame vertex.
      (Ignacio Vera)
  13. LUCENE-9417: Tessellator might build illegal polygons when several holes share are connected to the same vertex.
      (Ignacio Vera)
  14. LUCENE-9418: Fix ordered intervals over interleaved terms
      (Alan Woodward)
• Other   (12)
  1. LUCENE-9257: Always keep FST off-heap. FSTLoadMode, Reader attributes and openedFromWriter removed.
     (Bruno Roustant)
  2. LUCENE-9272: Checksums of the terms index are now verified when LeafReader#checkIntegrity is called rather than when opening the index.
     (Adrien Grand)
  3. LUCENE-9270: Update Javadoc about normalizeEntry in the Kuromoji DictionaryBuilder.
     (Namgyu Kim)
  4. LUCENE-9275: Make TestLatLonMultiPolygonShapeQueries more resilient for CONTAINS queries.
     (Ignacio Vera)
  5. LUCENE-9244: Adjust TestLucene60PointsFormat#testEstimatePointCount2Dims so it does not fail when a point is shared by multiple leaves.
     (Ignacio Vera)
  6. LUCENE-9271: ByteBufferIndexInput was refactored to work on top of the ByteBuffer API.
     (Adrien Grand)
  7. LUCENE-9191: Make LineFileDocs's random seeking more efficient, making tests using LineFileDocs faster
     (Robert Muir, Mike McCandless)
  8. LUCENE-9338: Refactors SimpleBindings to improve type safety and cycle detection
     (Alan Woodward, Adrien Grand)
  9. LUCENE-9358: Change the way the multi-dimensional BKD tree builder generates the intermediate tree representation to be equal to the one dimensional case to avoid unnecessary tree and leaves rotation.
     (Ignacio Vera)
  10. LUCENE-9288: poll_mirrors.py release script can handle HTTPS mirrors.
      (Ignacio Vera)
  11. LUCENE-9232: Fix or suppress 13 resource leak precommit warnings in lucene/replicator
      (Andras Salamon via Erick Erickson)
  12. LUCENE-9398: Always keep BKD index off-heap. BKD reader does not implement Accountable any more.
      (Ignacio Vera)
• Build   (4)
  1. Upgrade forbiddenapis to version 3.0.1.
     (Uwe Schindler)
  2. LUCENE-9376: Fix or suppress 20 resource leak precommit warnings in lucene/search
     (Andras Salamon via Erick Erickson)
  3. LUCENE-9380: Fix auxiliary class warnings in Lucene
     (Erick Erickson)
  4. LUCENE-9389: Enhance gradle logging calls validation: eliminate getMessage()
     (Andras Salamon via Erick Erickson)

Release 8.5.2 [2020-05-26]

• Optimizations   (1)
  1. LUCENE-9350: Partial reversion of LUCENE-9068; holding levenshtein automata on FuzzyQuery can end up blowing up query caches which use query objects as cache keys, so building the automata is now delayed to search time again.
     (Alan Woodward, Mike Drob)

Release 8.5.1 [2020-04-16]

• Bug Fixes   (1)
  1. LUCENE-9300: Fix corruption of the new gen field infos when doc values updates are applied on a segment created externally and added to the index with IndexWriter#addIndexes(Directory).
     (Jim Ferenczi, Adrien Grand)

Release 8.5.0 [2020-03-24]

• API Changes   (9)
  1. LUCENE-9093: Not an API change but a change in behavior of the UnifiedHighlighter's LengthGoalBreakIterator that will yield Passages sized a little different due to the fact that the sizing pivot is now the center of the first match and not its left edge.

  2. LUCENE-9116: PostingsWriterBase and PostingsReaderBase no longer support setting a field's metadata via a `long[]`.
     (Adrien Grand)
  3. LUCENE-9116: The FSTOrd postings format has been removed.
     (Adrien Grand)
  4. LUCENE-8369: Remove obsolete spatial module.
     (Nick Knize, David Smiley)
  5. LUCENE-8621: Refactor LatLonShape, XYShape, and all query and utility classes to core.
     (Nick Knize)
  6. LUCENE-9218: XY geometries API works in float space.
     (Ignacio Vera)
  7. LUCENE-9212: Intervals.multiterm() takes CompiledAutomaton rather than plain Automaton
     (Alan Woodward)
  8. LUCENE-9150: Restore support for dynamic PlanetModel in spatial3d.
     (Nick Knize)
  9. LUCENE-9171: QueryBuilder.newTermQuery() and .newSynonymQuery() now take boost parameters.
     (Alessandro Benedetti, Alan Woodward)
• New Features   (3)
  1. LUCENE-8903: Add LatLonShape and XYShape point query.
     (Ignacio Vera)
  2. LUCENE-8707: Add LatLonShape and XYShape distance query.
     (Ignacio Vera)
  3. LUCENE-9238: New XYPointField field and Queries for indexing, searching and sorting cartesian points.
     (Ignacio Vera)
• Improvements   (12)
  1. LUCENE-9149: Increase data dimension limit in BKD.
     (Nick Knize)
  2. LUCENE-9102: Add maxQueryLength option to DirectSpellchecker.
     (Andy Webb via Bruno Roustant)
  3. LUCENE-9091: UnifiedHighlighter HTML escaping should only escape essentials
     (Nándor Mátravölgyi)
  4. LUCENE-9105: UniformSplit postings format detects corrupted index and better handles IO exceptions.
     (Bruno Roustant)
  5. LUCENE-9106: UniformSplit postings format allows extension of block/line serializers.
     (Bruno Roustant)
  6. LUCENE-9093: UnifiedHighlighter's LengthGoalBreakIterator has a new fragmentAlignment option to better center the first match in the passage. Also the sizing point now pivots at the center of the first match term and not its left edge. This yields Passages that won't be identical to the previous behavior.
     (Nándor Mátravölgyi, David Smiley)
  7. LUCENE-9153: Allow WhitespaceAnalyzer to set a maxTokenLength other than the default of 255
     (Alan Woodward)
  8. LUCENE-9152: Improve line intersections with polygons when they are touching from the outside.
     (Ignacio Vera)
  9. LUCENE-9123: Add new JapaneseTokenizer constructors with discardCompoundToken option that controls whether the tokenizer emits original (compound) tokens when the mode is not NORMAL.
     (Kazuaki Hiraga via Tomoko Uchida)
  10. LUCENE-9253: KoreanTokenizer now supports custom dictionaries(system, unknown).
      (Namgyu Kim)
  11. LUCENE-9171: QueryBuilder can now use BoostAttributes on input token streams to selectively boost particular terms or synonyms in parsed queries.
      (Alessandro Benedetti, Alan Woodward)
  12. LUCENE-9298: Improve RAM accounting in BufferedUpdates when deleted doc IDs and terms are cleared.
      (Yu Binglei, Simon Willnauer)
• Optimizations   (10)
  1. LUCENE-9211: Add compression for Binary doc value fields.
     (Mark Harwood)
  2. LUCENE-4702: Better compression of terms dictionaries.
     (Adrien Grand)
  3. LUCENE-9228: Sort dvUpdates in the term order before applying if they all update a single field to the same value. This optimization can reduce the flush time by around 20% for the docValues update user cases.
     (Nhat Nguyen, Adrien Grand, Simon Willnauer)
  4. LUCENE-9245: Reduce AutomatonTermsEnum memory usage.
     (Bruno Roustant, Robert Muir)
  5. LUCENE-9237: Faster UniformSplit intersect TermsEnum.
     (Bruno Roustant)
  6. LUCENE-9260: LeafReader#checkIntegrity verifies checksums of CFS files.
     (Adrien Grand)
  7. LUCENE-9068: FuzzyQuery builds its Automaton up-front
     (Alan Woodward, Mike Drob)
  8. LUCENE-9113: Faster merging of SORTED/SORTED_SET doc values.
     (Adrien Grand)
  9. LUCENE-9125: Optimize Automaton.step() with binary search and introduce Automaton.next().
     (Bruno Roustant)
  10. LUCENE-9147: The index of stored fields and term vectors in now off-heap.
      (Adrien Grand)
• Bug Fixes   (11)
  1. LUCENE-9084: Fix potential deadlock due to circular synchronization in AnalyzingInfixSuggester
     (Paul Ward)
  2. LUCENE-9115: NRTCachingDirectory no longer caches files of unknown size.
     (Adrien Grand)
  3. LUCENE-9144: Fix error message on OneDimensionBKDWriter when too many points are added to the writer.
     (Ignacio Vera)
  4. LUCENE-9135: Make UniformSplit FieldMetadata counters long.
     (Bruno Roustant)
  5. LUCENE-9200: Fix TieredMergePolicy to use double (not float) math to make its merging decisions, fixing a corner-case bug uncovered by fun randomized tests
     (Robert Muir, Mike McCandless)
  6. LUCENE-9099: Unordered and Ordered interval queries now correctly handle repeated subterms - ordered intervals could supply an 'extra' minimized interval, resulting in odd matches when combined with eg CONTAINS queries; and unordered intervals would match duplicate subterms on the same position, so an query for UNORDERED(foo, foo) would match a document containing 'foo' only once.
     (Alan Woodward)
  7. LUCENE-9250: Add support for Circle2d#intersectsLine around the dateline.
     (Ignacio Vera)
  8. LUCENE-9243: Add fudge factor when creating a bounding box of a XYCircle.
     (Ignacio Vera)
  9. LUCENE-9239: Circle2D#WithinTriangle detects properly if a triangle is Within distance.
     (Ignacio Vera)
  10. LUCENE-9251: Fix bug in the polygon tessellator where edges with different value on #isEdgeFromPolygon were bot filtered out properly.
      (Ignacio Vera)
  11. LUCENE-9263: Fix wrong transformation of distance in meters to radians in Geo3DPoint.
      (Ignacio Vera)
• Other   (6)
  1. LUCENE-9109: Backport some changes from master (except StackWalker) to improve TestSecurityManager
     (Uwe Schindler)
  2. LUCENE-9110: Backport refactored stack analysis in tests to use generalized LuceneTestCase methods
     (Uwe Schindler)
  3. LUCENE-9141: Simplify LatLonShapeXQuery API by adding a new abstract class called LatLonGeometry. Queries are executed with input objects that extend such interface.
     (Ignacio Vera)
  4. LUCENE-9194: Simplify XYShapeXQuery API by adding a new abstract class called XYGeometry. Queries are executed with input objects that extend such interface.
     (Ignacio Vera)
  5. LUCENE-9096: Simplification of CompressingTermVectorsWriter#flushOffsets.
     (kkewwei via Adrien Grand)
  6. LUCENE-9225: Rectangle extends LatLonGeometry so it can be used in a geometry collection.
     (Ignacio Vera)

Release 8.4.1 [2020-01-13]

• Bug Fixes   (1)
  1. (No changes)

Release 8.4.0 [2019-12-29]

• API Changes   (1)
  1. LUCENE-9029: Deprecate SloppyMath toRadians/toDegrees in favor of Java Math.
     (Jack Conradson via Adrien Grand)
• New Features   (1)
  1. LUCENE-8620: Add CONTAINS support for LatLonShape and XYShape.
     (Ignacio Vera)
• Improvements   (7)
  1. LUCENE-9002: Skip costly caching clause in LRUQueryCache if it makes the query many times slower.
     (Guoqiang Jiang)
  2. LUCENE-9006: WordDelimiterGraphFilter's catenateAll token is now ordered before any token parts, like WDF did.
     (David Smiley)
  3. LUCENE-9028: introducing Intervals.multiterm()
     (Mikhail Khludnev)
  4. LUCENE-9018: ConcatenateGraphFilter now has a configurable separator.
     (Stanislav Mikulchik, David Smiley)
  5. LUCENE-9036: ExitableDirectoryReader may interupt scaning over DocValues
     (Mikhail Khludnev)
  6. LUCENE-9062: QueryVisitor now has a consumeTermsMatching() method, allowing queries that match a class of terms to pass a ByteRunAutomaton matching those that class back to the visitor.
     (Alan Woodward, David Smiley)
  7. LUCENE-9073: IntervalQuery to respond field on toString() and explain()
     (Mikhail Khludnev)
• Optimizations   (9)
  1. LUCENE-8928: When building a kd-tree for dimensions n > 2, compute exact bounds for an inner node every N splits to improve the quality of the tree. N is defined by SPLITS_BEFORE_EXACT_BOUNDS which is set to 4.
     (Ignacio Vera, Adrien Grand)
  2. BaseDirectoryReader no longer sums up the `LeafReader#numDocs` of its leaves eagerly. This especially helps when creating views of readers that hide documents, since computing the number of live documents is an expensive operation.
     (Adrien Grand)
  3. LUCENE-8992: TopFieldCollector and TopScoreDocCollector can now share minimum scores across leaves concurrently.
     (Adrien Grand, Atri Sharma, Jim Ferenczi)
  4. LUCENE-8932: BKDReader's index is now stored off-heap when the IndexInput is an instance of ByteBufferIndexInput.
     (Jack Conradson via Adrien Grand)
  5. LUCENE-9024: IntroSelector now falls back to the median of medians algorithm instead of sorting when the maximum recursion level is exceeded, providing better worst-case runtime.
     (Paul Sanwald via Adrien Grand)
  6. LUCENE-8920: The denser arcs of FST now index labels with a bitset in order to provide near constant time access.
     (Bruno Roustant, Mike Sokolov via Adrien Grand)
  7. LUCENE-9027: Use SIMD instructions to decode postings.
     (Adrien Grand)
  8. LUCENE-9049: Remove FST cached root arcs now redundant with labels indexed by bitset. This frees some on-heap FST space.
     (Jack Conradson via Bruno Roustant)
  9. LUCENE-9045: Do not use TreeMap/TreeSet in BlockTree and PerFieldPostingsFormat.
     (Bruno Roustant)
• Bug Fixes   (7)
  1. LUCENE-9001: Fix race condition in SetOnce.
     (Przemko Robakowski)
  2. LUCENE-9030: Fix WordnetSynonymParser behaviour so it behaves similar to SolrSynonymParser.
     (Christoph Buescher via Alan Woodward)
  3. LUCENE-9054: Fix reproduceJenkinsFailures.py to not overwrite junit XML files when retrying
     (hossman)
  4. LUCENE-9031: UnsupportedOperationException on MatchesIterator.getQuery()
     (Alan Woodward, Mikhail Khludnev)
  5. LUCENE-8996: maxScore was sometimes missing from distributed grouped responses.
     (Julien Massenet, Diego Ceccarelli, Munendra S N, Christine Poerschke)
  6. LUCENE-9055: Fix the detection of lines crossing triangles through edge points.
     (Ignacio Vera)
  7. LUCENE-9103: Disjunctions can miss some hits in some rare conditions.
     (Adrien Grand)
• Other   (6)
  1. LUCENE-8979: Code Cleanup: Use entryset for map iteration wherever possible. - Part 2
     (Koen De Groote)
  2. LUCENE-8994: Code Cleanup - Pass values to list constructor instead of empty constructor followed by addAll().
     (Koen De Groote)
  3. LUCENE-8746: Refactor EdgeTree - Introduce a Component tree that represents the tree of components (e.g polygons). Edge tree is now just a tree of edges.
     (Ignacio Vera)
  4. LUCENE-9046: Fix wrong example in Javadoc of TermInSetQuery
     (Namgyu Kim)
  5. LUCENE-8983: Add sandbox PhraseWildcardQuery to control multi-terms expansions in a phrase.
     (Bruno Roustant)
  6. LUCENE-9067: Polygon2D#contains() is now thread safe.
     (Ignacio Vera)
• Build   (2)
  1. Upgrade forbiddenapis to version 2.7; upgrade Groovy to 2.4.17.
     (Uwe Schindler)
  2. LUCENE-9041: Upgrade ecj to 3.19.0 to fix sporadic precommit javadoc issues
     (Kevin Risden)

Release 8.3.1 [2019-12-03]

• Bug Fixes   (1)
  1. LUCENE-9050: MultiTermIntervalsSource.visit() was not calling back to its visitor.
     (Alan Woodward)

Release 8.3.0 [2019-11-02]

• API Changes   (5)
  1. LUCENE-8909: IndexWriter#getFieldNames() method is used to get fields present in index. After LUCENE-8316, this method is no longer required. Hence, deprecate IndexWriter#getFieldNames() method.
     (Adrien Grand, Munendra S N)
  2. LUCENE-8755: SpatialPrefixTreeFactory now consumes the "version" parsed with Lucene's Version class. The quad and packed quad prefix trees are sensitive to this. It's recommended to pass the version like you should do likewise for analysis components for tokenized text, or else changes to the encoding in future versions may be incompatible with older indexes.
     (Chongchen Chen, David Smiley)
  3. LUCENE-8956: QueryRescorer now only sorts the first topN hits instead of all initial hits.
     (Paul Sanwald via Adrien Grand)
  4. LUCENE-8921: IndexSearcher.termStatistics() no longer takes a TermStates; it takes the docFreq and totalTermFreq. And don't call if docFreq <= 0. The previous implementation survives as deprecated and final. It's removed in 9.0.
     (Bruno Roustant, David Smiley, Alan Woodward)
  5. LUCENE-8990: PointValues#estimateDocCount(visitor) estimates the number of documents that would be matched by the given IntersectVisitor. THe method is used to compute the cost() of ScorerSuppliers instead of PointValues#estimatePointCount(visitor).
     (Ignacio Vera, Adrien Grand)
• New Features   (6)
  1. LUCENE-8936: Add SpanishMinimalStemFilter
     (vinod kumar via Tomoko Uchida)
  2. LUCENE-8764 LUCENE-8945: Add "export all terms and doc freqs" feature to Luke with delimiters.
     (Leonardo Menezes, Amish Shah via Tomoko Uchida)
  3. LUCENE-8747: Composite Matches from multiple subqueries now allow access to their submatches, and a new NamedMatches API allows marking of subqueries and a simple way to find which subqueries have matched on a given document
     (Alan Woodward, Jim Ferenczi)
  4. LUCENE-8769: Introduce Range Query For Multiple Connected Ranges
     (Atri Sharma)
  5. LUCENE-8960: Introduce LatLonDocValuesPointInPolygonQuery for LatLonDocValuesField
     (Ignacio Vera)
  6. LUCENE-8753: New UniformSplitPostingsFormat (name "UniformSplit") primarily benefiting in simplicity and extensibility. New STUniformSplitPostingsFormat (name "SharedTermsUniformSplit") that shares a single internal term dictionary across fields.
     (Bruno Roustant, Juan Rodriguez, David Smiley)
• Improvements   (15)
  1. LUCENE-8874: Show SPI names instead of class names in Luke Analysis tab.
     (Tomoko Uchida)
  2. LUCENE-8894: Add APIs to find SPI names for Tokenizer/CharFilter/TokenFilter factory classes.
     (Tomoko Uchida)
  3. LUCENE-8914: move the logic for discarding inner modes in FloatPointNearestNeighbor to the IntersectVisitor so we take advantage of the change introduced in LUCENE-7862.
     (Ignacio Vera)
  4. LUCENE-8955: move the logic for discarding inner modes in LatLonPoint NearestNeighbor to the IntersectVisitor so we take advantage of the change introduced in LUCENE-7862.
     (Ignacio Vera)
  5. LUCENE-8918: PhraseQuery throws exceptions at construction time if it is passed null arguments.
     (Alan Woodward)
  6. LUCENE-8916: GraphTokenStreamFiniteStrings preserves all Token attributes through its finite strings TokenStreams
     (Alan Woodward)
  7. LUCENE-8906: Expose Lucene50PostingsFormat.IntBlockTermState as public so that other postings formats can re-use it.
     (Bruno Roustant)
  8. LUCENE-8942: Remove redundant parameters and improve visibility strictness in LRUQueryCache
     (Atri Sharma)
  9. SOLR-13663: Introduce <SpanPositionRange> into XML Query Parser
     (Alessandro Benedetti via Mikhail Khludnev)
  10. LUCENE-8952: Use a sort key instead of true distance in NearestNeighbor
      (Julie Tibshirani).

  11. LUCENE-8620: Tessellator labels the edges of the generated triangles whether they belong to the original polygon. This information is added to the triangle encoding.
      (Ignacio Vera)
  12. LUCENE-8964: Fix geojson shape parsing on string arrays in properties
      (Alexander Reelsen)
  13. LUCENE-8976: Use exact distance between point and bounding rectangle in FloatPointNearestNeighbor.
      (Ignacio Vera)
  14. LUCENE-8966: The Korean analyzer now splits tokens on boundaries between digits and alphabetic characters.
      (Jim Ferenczi)
  15. LUCENE-8984: MoreLikeThis MLT is biased for uncommon fields
      (Andy Hind via Anshum Gupta)
• Optimizations   (8)
  1. LUCENE-8922: DisjunctionMaxQuery more efficiently leverages impacts to skip non-competitive hits.
     (Adrien Grand)
  2. LUCENE-8935: BooleanQuery with no scoring clause can now early terminate the query when the total hits is not requested.
     (Jim Ferenczi)
  3. LUCENE-8941: Matches on wildcard queries will defer building their full disjunction until a MatchesIterator is pulled
     (Alan Woodward)
  4. LUCENE-8755: spatial-extras quad and packed quad prefix trees now index points faster.
     (Chongchen Chen, David Smiley)
  5. LUCENE-8860: add additional leaf node level optimizations in LatLonShapeBoundingBoxQuery.
     (Igor Motov via Ignacio Vera)
  6. LUCENE-8968: Improve performance of WITHIN and DISJOINT queries for Shape queries by doing just one pass whenever possible.
     (Ignacio Vera)
  7. LUCENE-8939: Introduce shared count based early termination across multiple slices
     (Atri Sharma)
  8. LUCENE-8980: Blocktree's seekExact now short-circuits false if the term isn't in the min-max range of the segment. Large perf gain for ID/time like data when populated sequentially.
     (Guoqiang Jiang)
• Bug Fixes   (2)
  1. LUCENE-8755: spatial-extras quad and packed quad prefix trees could throw a NullPointerException for certain cell edge coordinates
     (Chongchen Chen, David Smiley)
  2. LUCENE-9005: BooleanQuery.visit() would pull subVisitors from its parent visitor, rather than from a visitor for its own specific query. This could cause problems when BQ was nested under another BQ. Instead, we now pull a MUST subvisitor, pass it to any MUST subclauses, and then pull SHOULD, MUST_NOT and FILTER visitors from it rather than from the parent.
     (Alan Woodward)
• Other   (7)
  1. LUCENE-8778 LUCENE-8911 LUCENE-8957: Define analyzer SPI names as static final fields and document the names in Javadocs.
     (Tomoko Uchida, Uwe Schindler)
  2. LUCENE-8758: QuadPrefixTree: removed levelS and levelN fields which weren't used.
     (Amish Shah)
  3. LUCENE-8975: Code Cleanup: Use entryset for map iteration wherever possible.
     (Koen De Groote)
  4. LUCENE-8993, LUCENE-8807: Changed all repository and download references in build files to HTTPS.
     (Uwe Schindler)
  5. LUCENE-8998: Fix OverviewImplTest.testIsOptimized reproducible failure.
     (Tomoko Uchida)
  6. LUCENE-8999: LuceneTestCase.expectThrows now propogates assert/assumption failures up to the test w/o wrapping in a new assertion failure unless the caller has explicitly expected them
     (hossman)
  7. LUCENE-8062: GlobalOrdinalsWithScoreQuery is no longer eligible for query caching.
     (Jim Ferenczi)

Release 8.2.0 [2019-07-26]

• API Changes   (3)
  1. LUCENE-8865: IndexSearcher now uses Executor instead of ExecutorSerivce. This change is fully backwards compatible since ExecutorService directly implements Executor.
     (Simon Willnauer)
  2. LUCENE-8856: Intervals queries have moved from the sandbox to the queries module.
     (Alan Woodward)
  3. LUCENE-8893: Intervals.wildcard() and Intervals.prefix() methods now take BytesRef rather than String.
     (Alan Woodward)
• New Features   (10)
  1. LUCENE-8632: New XYShape Field and Queries for indexing and searching general cartesian geometries.
     (Nick Knize)
  2. LUCENE-8891: Snowball stemmer/analyzer for the Estonian language.
     (Gert Morten Paimla via Tomoko Uchida)
  3. LUCENE-8815: Provide a DoubleValues implementation for retrieving the value of features without requiring a separate numeric field. Note that as feature values are stored with only 8 bits of mantissa the values returned may have a delta from the original values indexed.
     (Colin Goodheart-Smithe via Adrien Grand)
  4. LUCENE-8803: Provide a FeatureSortfield to allow sorting search hits by descending value of a feature. This is exposed via the factory method FeatureField#newFeatureSort.
     (Colin Goodheart-Smithe via Adrien Grand)
  5. LUCENE-8784: The KoreanTokenizer now preserves punctuations if discardPunctuation is set to false (defaults to true).
     (Namgyu Kim via Jim Ferenczi)
  6. LUCENE-8812: Add new KoreanNumberFilter that can change Hangul character to number and process decimal point. It is similar to the JapaneseNumberFilter.
     (Namgyu Kim)
  7. LUCENE-8362: Add doc-value support to range fields.
     (Atri Sharma via Adrien Grand)
  8. LUCENE-8766: Add monitor subproject (previously Luwak monitoring library). This allows a stream of documents to be matched against a set of registered queries in an efficient manner, for use as a monitoring or classification tool.
     (Alan Woodward)
  9. LUCENE-7714: Add a numeric range query in sandbox that takes advantage of index sorting.
     (Julie Tibshirani via Jim Ferenczi)
  10. LUCENE-8859: The completion suggester's postings format now have an option to load its internal FST off-heap.
      (Jim Ferenczi)
• Bug Fixes   (9)
  1. LUCENE-8831: Fixed LatLonShapeBoundingBoxQuery .hashCode methods.
     (Ignacio Vera)
  2. LUCENE-8775: Improve tessellator to handle better cases where a hole share a vertex with the polygon.
     (Ignacio Vera)
  3. LUCENE-8785: Ensure new threadstates are locked before retrieving the number of active threadstates. This causes assertion errors and potentially broken field attributes in the IndexWriter when IndexWriter#deleteAll is called while actively indexing.
     (Simon Willnauer)
  4. LUCENE-8804: Forbid calls to putAttribute on frozen FieldType instances.
     (Vamshi Vijay Nakkirtha via Adrien Grand)
  5. LUCENE-8828: Removes the buggy 'disallow overlaps' boolean from Intervals.unordered(), and replaces it with a new Intervals.unorderedNoOverlaps() method
     (Alan Woodward)
  6. LUCENE-8843: Don't ignore exceptions that are thrown when trying to open a file in IOUtils#fsync.
     (Jason Tedor via Adrien Grand)
  7. LUCENE-8835: FileSwitchDirectory now respects the file extension when listing directory contents to ensure we don't expose pending deletes if both directory point to the same underlying filesystem directory.
     (Simon Willnauer)
  8. LUCENE-8853: FileSwitchDirectory now applies best effort to place tmp files in the same directory as the target files.
     (Simon Willnauer)
  9. LUCENE-8892: Add missing closing parentheses in MultiBoolFunction's description()
     (Florian Diebold, Munendra S N)
• Improvements   (8)
  1. LUCENE-7840: Non-scoring BooleanQuery now removes SHOULD clauses before building the scorer supplier as opposed to eliminating them during scoring construction.
     (Atri Sharma via Jim Ferenczi)
  2. LUCENE-8770: BlockMaxConjunctionScorer now leverages two-phase iterators in order to avoid executing the second phase when scorers don't intersect.
     (Adrien Grand, Jim Ferenczi)
  3. LUCENE-8818: Fix smokeTestRelease.py encoding bug
     (janhoy)
  4. LUCENE-8845: Allow Intervals.prefix() and Intervals.wildcard() to specify their maximum allowed expansions
     (Alan Woodward)
  5. LUCENE-8875: Introduce a Collector optimized for use cases when large number of hits are requested
     (Atri Sharma)
  6. LUCENE-8848 LUCENE-7757 LUCENE-8492: The UnifiedHighlighter now detects that parts of the query are not understood by it, and thus it should not make optimizations that result in no highlights or slow highlighting. This generally works best for WEIGHT_MATCHES mode. Consequently queries produced by ComplexPhraseQueryParser and the surround QueryParser will now highlight correctly.
     (David Smiley)
  7. LUCENE-8793: Luke enhanced UI for CustomAnalyzer: show detailed analysis steps.
     (Jun Ohtani via Tomoko Uchida)
  8. LUCENE-8855: Add Accountable to some Query implementations
     (ab, Adrien Grand)
• Optimizations   (8)
  1. LUCENE-8796: Use exponential search instead of binary search in IntArrayDocIdSet#advance method
     (Luca Cavanna via Adrien Grand)
  2. LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor. Now caller threads execute at least one search on an index even if there is an executor provided to minimize thread context switching.
     (Simon Willnauer)
  3. LUCENE-8868: New storing strategy for BKD tree leaves with low cardinality. It stores the distinct values once with the cardinality value reducing the storage cost.
     (Ignacio Vera)
  4. LUCENE-8885: Optimise BKD reader by exploiting cardinality information stored on leaves.
     (Ignacio Vera)
  5. LUCENE-8896: Override default implementation of IntersectVisitor#visit(DocIDSetBuilder, byte[]) for several queries.
     (Ignacio Vera)
  6. LUCENE-8901: Load frequencies lazily only when needed in BlockDocsEnum and BlockImpactsEverythingEnum
     (Mayya Sharipova).

  7. LUCENE-8888: Optimize distribution of points with data dimensions in BKD tree leaves.
     (Ignacio Vera)
  8. LUCENE-8311: Phrase queries now leverage impacts.
     (Adrien Grand)
• Test Framework   (1)
  1. LUCENE-8825: CheckHits now display the shard index in case of mismatch between top hits.
     (Atri Sharma via Adrien Grand)
• Other   (6)
  1. LUCENE-8847: Code Cleanup: Remove StringBuilder.append with concatenated strings.
     (Koen De Groote via Uwe Schindler)
  2. LUCENE-8861: Script to find open Github PRs that needs attention
     (janhoy)
  3. LUCENE-8852: ReleaseWizard tool for release managers
     (janhoy)
  4. LUCENE-8838: Remove support for Steiner points on Tessellator.
     (Ignacio Vera)
  5. LUCENE-8879: Improve BKDRadixSelector tests.
     (Ignacio Vera)
  6. LUCENE-8886: Fix TestMutablePointsReaderUtils tests.
     (Ignacio Vera)

Release 8.1.1 [2019-05-28]

• Improvements   (1)
  1. LUCENE-8781: FST lookup performance has been improved in many cases by encoding Arcs using full-sized arrays with gaps. The new encoding is enabled for postings in the default codec and for suggesters.
     (Mike Sokolov)

Release 8.1.0 [2019-05-16]

• API Changes   (2)
  1. LUCENE-3041: A query introspection API has been added. Queries should implement a visit() method, taking a QueryVisitor, and either pass the visitor down to any child queries, or call a visitX() or consumeX() method on it. All locations in the code that called Weight.extractTerms() have been changed to use this API, and the extractTerms() method has been deprecated.
     (Alan Woodward, Simon Willnauer, David Smiley, Luca Cavanna)
  2. LUCENE-8735: Directory.getPendingDeletions is now abstract to ensure subclasses override it. FilterDirectory now delegates the call, ensuring correct default behaviour for subclasses.
     (Henning Andersen)
• New Features   (1)
  1. LUCENE-2562: The well-known graphical user interface for inspecting Lucene indexes "Luke" was added as a Lucene module. It can be started from the binary distribution by calling the shell scripts in the module folder or from the source checkout by using `ant -f lucene/luke/build.xml run`. Luke provides a Swing-based user interface and can be used to open Lucene or Solr (or Elasticsearch) indexes, inspect documents, check index commits and segments, or test (custom) analyzers. It also has maintenance functions to check index structures and force merge indexes for archival. Luke was originally developed by Andrzej Bialecki, later maintained by Dmitry Kan and finally rewritten by Tomoko Uchida to use the ASF licensing compatible Swing framework (as shipped with JDKs).
     (Tomoko Uchida, Uwe Schindler)
• Bug fixes   (10)
  1. LUCENE-8736: LatLonShapePolygonQuery returns incorrect WITHIN results with shared boundaries. Point in Polygon now correctly includes boundary points. Box and Polygon relations with triangles have also been improved to correctly include boundary points.
     (Nick Knize)
  2. LUCENE-8712: Polygon2D does not detect crossings through segment edges.
     (Ignacio Vera)
  3. LUCENE-8720: NameIntCacheLRU (in the facets module) had an int overflow bug that disabled cleaning of the cache
     (Russell A Brown)
  4. LUCENE-8726: ValueSource.asDoubleValuesSource() could leak a reference to IndexSearcher
     (Alan Woodward, Yury Pakhomov)
  5. LUCENE-8719: FixedShingleFilter can miss shingles at the end of a token stream if there are multiple paths with different lengths.
     (Alan Woodward)
  6. LUCENE-8688: TieredMergePolicy#findForcedMerges now tries to create the cheapest merges that allow the index to go down to `maxSegmentCount` segments or less.
     (Armin Braun via Adrien Grand)
  7. LUCENE-8477: Interval disjunctions could miss valid hits if some of the clauses of the disjunction are minimized away. We now rewrite intervals if a source contains a disjunction and the internal gaps matter for matching. This behaviour can be disabled if users are more interested in speed rather than accuracy of matching.
     (Alan Woodward, Jim Ferenczi)
  8. LUCENE-8741: ValueSource.fromDoubleValuesSource() was casting to Scorer instead of Scorable, leading to ClassCastExceptions
     (Markus Jelsma, Alan Woodward)
  9. LUCENE-8754: Fix ConcurrentModificationException in SegmentInfo if attributes are accessed in MergePolicy while the merge is running
     (Simon Willnauer)
  10. LUCENE-8765: Fixed validation of the number of added points in KD trees.
      (Zhao Yang via Adrien Grand)
• Improvements   (13)
  1. LUCENE-8673: Use radix partitioning when merging dimensional points instead of sorting all dimensions before hand.
     (Ignacio Vera, Adrien Grand)
  2. LUCENE-8687: Optimise radix partitioning for points on heap.
     (Ignacio Vera)
  3. LUCENE-8699: Change HeapPointWriter to use a single byte array instead to a list of byte arrays. In addition a new interface PointValue is added to abstract out the different formats between offline and on-heap writers.
     (Ignacio Vera)
  4. LUCENE-8703: Build point writers in the BKD tree only when they are needed.
     (Ignacio Vera)
  5. LUCENE-8652: SynonymQuery can now deboost the document frequency of each term when blending the score of the synonym.
     (Jim Ferenczi)
  6. LUCENE-8631: The Korean's user dictionary now picks the longest-matching word and discards the other matches.
     (Yeongsu Kim via Jim Ferenczi)
  7. LUCENE-8732: ConstantScoreQuery can now early terminate the query if the minimum score is greater than the constant score and total hits are not requested.
     (Jim Ferenczi)
  8. LUCENE-8750: Implements setMissingValue() on sort fields produced from DoubleValuesSource and LongValuesSource
     (Mike Sokolov via Alan Woodward)
  9. LUCENE-8701: ToParentBlockJoinQuery now creates a child scorer that disallows skipping over non-competitive documents if the score of a parent depends on the score of multiple children (avg, max, min). Additionally the score mode `none` that assigns a constant score to each parent can early terminate top scores's collection.
     (Jim Ferenczi)
  10. LUCENE-8751: Weight#matches now use the ScorerSupplier to build scorers with a lead cost of 1 (single document).
      (Jim Ferenczi)
  11. LUCENE-8752: Japanese new era name '令和' (Reiwa) is added to the dictionary used in JapaneseTokenizer so that the analyzer handles the era name correctly. Reiwa is set to replace the Heisei Era on May 1, 2019.
      (Tomoko Uchida)
  12. LUCENE-8671: Introduced reader attributes allows a per IndexReader configuration of codec internals. This enables a per reader configuration if FSTs are on- or off-heap on a per field basis
      (Simon Willnauer)
  13. LUCENE-8787: spatial-extras DateRangePrefixTree used to only parse ISO-8601 timestamps with 0 or 3 digits of milliseconds precision but now parses other lengths (although > 3 not used).
      (Thomas Lemmé via David Smiley)
• Changes in Runtime Behavior   (4)
  1. LUCENE-8671: Load FST off-heap also for ID-like fields if reader is not opened from an IndexWriter.
     (Simon Willnauer)
  2. LUCENE-8730: WordDelimiterGraphFilter always emits its original token first. This brings its behaviour into line with the deprecated WordDelimiterFilter, so that the only difference in output between the two is in the position length attribute.
     (Alan Woodward, Jim Ferenczi)
  3. LUCENE-7386: Disjunctions nested in disjunctions are now flattened. This might trigger changes in the produced scores due to changes to the order in which scores of sub clauses are summed up.
     (Adrien Grand)
  4. LUCENE-8756: MoreLikeThisQuery now respects custom term frequencies (TermFrequencyAttribute) at search time
     (Olli Kuonanoja)
• Other   (5)
  1. LUCENE-8680: Refactor EdgeTree#relateTriangle method.
     (Ignacio Vera)
  2. LUCENE-8685: Refactor LatLonShape tests.
     (Ignacio Vera)
  3. LUCENE-8713: Add Line2D tests.
     (Ignacio Vera)
  4. LUCENE-8729: Workaround: Disable accessibility doclints (Java 13+), so compilation with recent JDK succeeds.
     (Uwe Schindler)
  5. LUCENE-8725: Make TermsQuery.SeekingTermSetTermsEnum a top level class and public
     (noble)

Release 8.0.0 [2019-03-14]

• API Changes   (31)
  1. LUCENE-8662: TermsEnum.seekExact(BytesRef) to abstract and delegate seekExact(BytesRef) in FilterLeafReader.FilterTermsEnum.
     (Jeffery Yuan via Tomás Fernández Löbbe, Simon Willnauer)
  2. LUCENE-8469: Deprecated StringHelper.compare has been removed.
     (Dawid Weiss)
  3. LUCENE-8039: Introduce a "delta distance" method set to GeoDistance. This allows distance calculations, especially for paths, to take into account an "excursion" to include the specified point.

  4. LUCENE-8007: Index statistics Terms.getSumDocFreq(), Terms.getDocCount() are now required to be stored by codecs. Additionally, TermsEnum.totalTermFreq() and Terms.getSumTotalTermFreq() are now required: if frequencies are not stored they are equal to TermsEnum.docFreq() and Terms.getSumDocFreq(), respectively, because all freq() values equal 1.
     (Adrien Grand, Robert Muir)
  5. LUCENE-8038: Deprecated PayloadScoreQuery constructors have been removed
     (Alan Woodward)
  6. LUCENE-8014: Similarity.computeSlopFactor() and Similarity.computePayloadFactor() have been removed
     (Alan Woodward)
  7. LUCENE-7996: Queries are now required to produce positive scores.
     (Adrien Grand)
  8. LUCENE-8099: CustomScoreQuery, BoostedQuery and BoostingQuery have been removed
     (Alan Woodward)
  9. LUCENE-8012: Explanation now takes Number rather than float
     (Alan Woodward, Robert Muir)
  10. LUCENE-8116: SimScorer now only takes a frequency and a norm as per-document scoring factors.
      (Adrien Grand)
  11. LUCENE-8113: TermContext has been renamed to TermStates, and can now be constructed lazily if term statistics are not required
      (Alan Woodward)
  12. LUCENE-8242: Deprecated method IndexSearcher#createNormalizedWeight() has been removed
      (Alan Woodward)
  13. LUCENE-8267: Memory codecs removed from the codebase (MemoryPostings, MemoryDocValues).
      (Dawid Weiss)
  14. LUCENE-8144: Moved QueryCachingPolicy.ALWAYS_CACHE to the test framework.
      (Nhat Nguyen via Adrien Grand)
  15. LUCENE-8356: StandardFilter and StandardFilterFactory have been removed
      (Alan Woodward)
  16. LUCENE-8373: StandardAnalyzer.ENGLISH_STOP_WORD_SET has been removed
      (Alan Woodward)
  17. LUCENE-8388: Unused PostingsEnum#attributes() method has been removed
      (Alan Woodward)
  18. LUCENE-8405: TopDocs.maxScore is removed. IndexSearcher and TopFieldCollector no longer have an option to compute the maximum score when sorting by field.
      (Adrien Grand)
  19. LUCENE-8411: TopFieldCollector no longer takes a fillFields option, it now always fills fields.
      (Adrien Grand)
  20. LUCENE-8412: TopFieldCollector no longer takes a trackDocScores option. Scores need to be set on top hits via TopFieldCollector#populateScores instead.
      (Adrien Grand)
  21. LUCENE-6228: A new Scorable abstract class has been added, containing only those methods from Scorer that should be called from Collectors. LeafCollector.setScorer() now takes a Scorable rather than a Scorer.
      (Alan Woodward, Adrien Grand)
  22. LUCENE-8475: Deprecated constants have been removed from RamUsageEstimator.
      (Dimitrios Athanasiou)
  23. LUCENE-8483: Scorers may no longer take null as a Weight
      (Alan Woodward)
  24. LUCENE-8352: TokenStreamComponents is now final, and can take a Consumer<Reader> in its constructor
      (Mark Harwood, Alan Woodward, Adrien Grand)
  25. LUCENE-8498: LowerCaseTokenizer has been removed, and CharTokenizer no longer takes a normalizer function.
      (Alan Woodward)
  26. LUCENE-7875: Moved MultiFields static methods out of the class. getLiveDocs is now in MultiBits which is now public. getMergedFieldInfos and getIndexedFields are now in FieldInfos. getTerms is now in MultiTerms. getTermPositionsEnum and getTermDocsEnum were collapsed and renamed to just getTermPostingsEnum and moved to MultiTerms.
      (David Smiley)
  27. LUCENE-8513: MultiFields.getFields is now removed. Please avoid this class, and Fields in general, when possible.
      (David Smiley)
  28. LUCENE-8497: MultiTermAwareComponent has been removed, and in its place TokenFilterFactory and CharFilterFactory now expose type-safe normalize() methods. This decouples normalization from tokenization entirely.
      (Mayya Sharipova, Alan Woodward)
  29. LUCENE-8597: IntervalIterator now exposes a gaps() method that reports the number of gaps between its component sub-intervals. This can be used in a new filter available via Intervals.maxgaps().
      (Alan Woodward)
  30. LUCENE-8609: Remove IndexWriter#numDocs() and IndexWriter#maxDoc() in favor of IndexWriter#getDocStats().
      (Simon Willnauer)
  31. LUCENE-8292: Make TermsEnum fully abstract.
      (Simon Willnauer)
• Changes in Runtime Behavior   (15)
  1. LUCENE-8333: Switch MoreLikeThis.setMaxDocFreqPct to use maxDoc instead of numDocs.
     (Robert Muir, Dawid Weiss).

  2. LUCENE-7837: Indices that were created before the previous major version will now fail to open even if they have been merged with the previous major version.
     (Adrien Grand)
  3. LUCENE-8020: Similarities are no longer passed terms that don't exist by queries such as SpanOrQuery, so scoring formulas no longer require divide-by-zero hacks. IndexSearcher.termStatistics/collectionStatistics return null instead of returning bogus values for a non-existent term or field.
     (Robert Muir)
  4. LUCENE-7996: FunctionQuery and FunctionScoreQuery now return a score of 0 when the function produces a negative value.
     (Adrien Grand)
  5. LUCENE-8116: Similarities now score fields that omit norms as if the norm was 1. This might change score values on fields that omit norms.
     (Adrien Grand)
  6. LUCENE-8134: Index options are no longer automatically downgraded.
     (Adrien Grand)
  7. LUCENE-8031: Length normalization correctly reflects omission of term frequencies.
     (Robert Muir, Adrien Grand)
  8. LUCENE-7444: StandardAnalyzer no longer defaults to removing English stopwords
     (Alan Woodward)
  9. LUCENE-8060: IndexSearcher's search and searchAfter methods now only compute total hit counts accurately up to 1,000 in order to enable top-hits optimizations such as block-max WAND (LUCENE-8135).
     (Adrien Grand)
  10. LUCENE-8505: IndexWriter#addIndices will now fail if the target index is sorted but the candidate is not.
      (Jim Ferenczi)
  11. LUCENE-8535: Highlighter and FVH doesn't support ToParent and ToChildBlockJoinQuery out of the box anymore. In order to highlight on Block-Join Queries a custom WeightedSpanTermExtractor / FieldQuery should be used.
      (Simon Willnauer, Jim Ferenczi, Julie Tibshirani)
  12. LUCENE-8563: BM25 scores don't include the (k1+1) factor in their numerator anymore. This doesn't affect ordering as this is a constant factor which is the same for every document.
      (Luca Cavanna via Adrien Grand)
  13. LUCENE-8509: WordDelimiterGraphFilter will no longer set the offsets of internal tokens by default, preventing a number of bugs when the filter is chained with tokenfilters that change the length of their tokens
      (Alan Woodward)
  14. LUCENE-8633: IntervalQuery scores do not use term weighting any more, the score is instead calculated as a function of the sloppy frequency of the matching intervals.
      (Alan Woodward, Jim Ferenczi)
  15. LUCENE-8635: FSTs can now remain off-heap, accessed via IndexInput, and the default codec's term dictionary (BlockTreeTermsReader) will now leave the FST for the terms index off-heap for non-primary-key fields using MMapDirectory, reducing heap usage for such fields.
      (Ankit Jain)
• New Features   (12)
  1. LUCENE-8340: LongPoint#newDistanceFeatureQuery may be used to boost scores based on how close a value of a long field is from an configurable origin. This is typically useful to boost by recency.
     (Adrien Grand)
  2. LUCENE-8482: LatLonPoint#newDistanceFeatureQuery may be used to boost scores based on the haversine distance of a LatLonPoint field to a provided point. This is typically useful to boost by distance.
     (Ignacio Vera)
  3. LUCENE-8216: Added a new BM25FQuery in sandbox to blend statistics across several fields using the BM25F formula.
     (Adrien Grand, Jim Ferenczi)
  4. LUCENE-8564: GraphTokenFilter is an abstract class useful for token filters that need to read-ahead in the token stream and take into account graph structures. This also changes FixedShingleFilter to extend GraphTokenFilter
     (Alan Woodward)
  5. LUCENE-8612: Intervals.extend() treats an interval as if it covered a wider span than it actually does, allowing users to force minimum gaps between intervals in a phrase.
     (Alan Woodward)
  6. LUCENE-8629: New interval functions: Intervals.before(), Intervals.after(), Intervals.within() and Intervals.overlapping().
     (Alan Woodward)
  7. LUCENE-8622: Adds a minimum-should-match interval function that produces intervals spanning a subset of a set of sources.
     (Alan Woodward)
  8. LUCENE-8645: Intervals.fixField() allows you to report intervals from one field as if they came from another.
     (Alan Woodward)
  9. LUCENE-8646: New interval functions: Intervals.prefix() and Intervals.wildcard()
     (Alan Woodward)
  10. LUCENE-8655: Add a getter in FunctionScoreQuery class in order to access to the underlying DoubleValuesSource.
      (Gérald Quaire via Alan Woodward)
  11. LUCENE-8697: GraphTokenStreamFiniteStrings correctly handles side paths containing gaps
      (Alan Woodward)
  12. LUCENE-8702: Simplify intervals returned from vararg Intervals factory methods
      (Alan Woodward)
• Improvements   (6)
  1. LUCENE-7997: Add BaseSimilarityTestCase to sanity check similarities. SimilarityBase switches to 64-bit doubles internally to help avoid common numeric issues. Add missing range checks for similarity parameters. Improve BM25 and ClassicSimilarity's explanations.
     (Robert Muir)
  2. LUCENE-8011: Improved similarity explanations.
     (Mayya Sharipova via Adrien Grand)
  3. LUCENE-4198: Codecs now have the ability to index score impacts.
     (Adrien Grand)
  4. LUCENE-8135: Boolean queries now implement the block-max WAND algorithm in order to speed up selection of top scored documents.
     (Adrien Grand)
  5. LUCENE-8279: CheckIndex now cross-checks terms with norms.
     (Adrien Grand)
  6. LUCENE-8660: TopDocsCollectors now return an accurate count (instead of a lower bound) if the total hit count is equal to the provided threshold.
     (Adrien Grand, Jim Ferenczi)
• Optimizations   (12)
  1. LUCENE-8040: Optimize IndexSearcher.collectionStatistics, avoiding MultiFields/MultiTerms
     (David Smiley, Robert Muir)
  2. LUCENE-4100: Disjunctions now support faster collection of top hits when the total hit count is not required.
     (Stefan Pohl, Adrien Grand, Robert Muir)
  3. LUCENE-7993: Phrase queries are now faster if total hit counts are not required.
     (Adrien Grand)
  4. LUCENE-8109: Boolean queries propagate information about the minimum competitive score in order to make collection faster if there are disjunctions or phrase queries as sub queries, which know how to leverage this information to run faster.
     (Adrien Grand)
  5. LUCENE-8439: Disjunction max queries can skip blocks to select the top documents if the total hit count is not required.
     (Jim Ferenczi, Adrien Grand)
  6. LUCENE-8204: Boolean queries with a mix of required and optional clauses are now faster if the total hit count is not required.
     (Jim Ferenczi, Adrien Grand)
  7. LUCENE-8448: Boolean queries now propagates the mininum score to their sub-scorers.
     (Jim Ferenczi, Adrien Grand)
  8. LUCENE-8511: MultiFields.getIndexedFields is now optimized; does not call getMergedFieldInfos
     (David Smiley)
  9. LUCENE-8507: TopFieldCollector can now update the minimum competitive score if the primary sort is by relevancy and the total hit count is not required.
     (Jim Ferenczi)
  10. LUCENE-8464: ConstantScoreScorer now implements setMinCompetitveScore in order to early terminate the iterator if the minimum score is greater than the constant score.
      (Christophe Bismuth via Jim Ferenczi)
  11. LUCENE-8607: MatchAllDocsQuery can shortcut when total hit count is not required
      (Alan Woodward, Adrien Grand)
  12. LUCENE-8585: Index-time jump-tables for DocValues, for O(1) advance when retrieving doc values.
      (Toke Eskildsen, Adrien Grand)

Release 7.7.2 [2019-06-04]

• Bug fixes   (6)
  1. LUCENE-8726: ValueSource.asDoubleValuesSource() could leak a reference to IndexSearcher
     (Alan Woodward, Yury Pakhomov)
  2. LUCENE-8735: FilterDirectory.getPendingDeletions now forwards to the delegate even the method is not abstract in the super class. This prevents issues where our best effort in carrying on generations in the IndexWriter since pending deletions are swallowed by the FilterDirectory.
     (Henning Andersen, Simon Willnauer)
  3. LUCENE-8688: TieredMergePolicy#findForcedMerges now tries to create the cheapest merges that allow the index to go down to `maxSegmentCount` segments or less.
     (Armin Braun via Adrien Grand)
  4. LUCENE-8785: Ensure new threadstates are locked before retrieving the number of active threadstates. This causes assertion errors and potentially broken field attributes in the IndexWriter when IndexWriter#deleteAll is called while actively indexing.
     (Simon Willnauer)
  5. LUCENE-8720: NameIntCacheLRU (in the facets module) had an int overflow bug that disabled cleaning of the cache
     (Russell A Brown)
  6. LUCENE-8809: Refresh and rollback concurrently can leave segment states unclosed
     (Nhat Nguyen)

Release 7.7.1 [2019-03-01]

• (No Changes)

Release 7.7.0 [2019-02-11]

• Changes in Runtime Behavior   (1)
  1. LUCENE-8527: StandardTokenizer and UAX29URLEmailTokenizer now support Unicode 9.0, and provide Unicode UTS#51 v11.0 Emoji tokenization with the "<EMOJI>" token type.

• Build   (2)
  1. LUCENE-8611: Update randomizedtesting to 2.7.2, JUnit to 4.12, add hamcrest-core dependency.
     (Dawid Weiss)
  2. LUCENE-8537: ant test command fails under lucene/tools
     (Peter Somogyi)
• Bug fixes   (9)
  1. LUCENE-8669: Fix LatLonShape WITHIN queries that fail with Multiple search Polygons that share the dateline.
     (Nick Knize)
  2. LUCENE-8603: Fix the inversion of right ids for additional nouns in the Korean user dictionary.
     (Yoo Jeongin via Jim Ferenczi)
  3. LUCENE-8624: int overflow in ByteBuffersDataOutput.size().
     (Mulugeta Mammo, Dawid Weiss)
  4. LUCENE-8625: int overflow in ByteBuffersDataInput.sliceBufferList.
     (Mulugeta Mammo, Dawid Weiss)
  5. LUCENE-8639: Newly created threadstates while flushing / refreshing can cause duplicated sequence IDs on IndexWriter.
     (Simon Willnauer)
  6. LUCENE-8649: LatLonShape's within and disjoint queries can return false positives with indexed multi-shapes.
     (Ignacio Vera)
  7. LUCENE-8654: Polygon2D#relateTriangle returns the wrong answer if polygon is inside the triangle.
     (Ignacio Vera)
  8. LUCENE-8650: ConcatenatingTokenStream did not correctly clear its state in reset(), and was not propagating final position increments from its child streams correctly.
     (Dan Meehl, Alan Woodward)
  9. LUCENE-8676: The Korean tokenizer does not update the last position if the backtrace is caused by a big buffer (1024 chars).
     (Jim Ferenczi)
• New Features   (3)
  1. LUCENE-8026: ExitableDirectoryReader may now time out queries that run on points such as range queries or geo queries.
     (Christophe Bismuth via Adrien Grand)
  2. LUCENE-8508: IndexWriter can now set the created version via IndexWriterConfig#setIndexCreatedVersionMajor. This is an expert feature.
     (Adrien Grand)
  3. LUCENE-8601: Attributes set in the IndexableFieldType for each field during indexing will now be recorded into the corresponding FieldInfo's attributes, accessible at search time
     (Murali Krishna P)
• Improvements   (8)
  1. LUCENE-8463: TopFieldCollector can now early-terminates queries when sorting by SortField.DOC.
     (Christophe Bismuth via Jim Ferenczi)
  2. LUCENE-8562: Speed up merging segments of points with data dimensions by only sorting on the indexed dimensions.
     (Ignacio Vera)
  3. LUCENE-8529: TopSuggestDocsCollector will now use the completion key to tiebreak completion suggestion with identical scores.
     (Jim Ferenczi)
  4. LUCENE-8575: SegmentInfos#toString now includes attributes and diagnostics.
     (Namgyu Kim via Adrien Grand)
  5. LUCENE-8548: The KoreanTokenizer no longer splits unknown words on combining diacritics and detects script boundaries more accurately with Character#UnicodeScript#of.
     (Christophe Bismuth, Jim Ferenczi)
  6. LUCENE-8581: Change LatLonShape encoding to use 4 bytes Per Dimension.
     (Ignacio Vera, Nick Knize, Adrien Grand)
  7. LUCENE-8527: Upgrade JFlex dependency to 1.7.0; in StandardTokenizer and UAX29URLEmailTokenizer, increase supported Unicode version from 6.3 to 9.0, and support Unicode UTS#51 v11.0 Emoji tokenization.

  8. LUCENE-8640: Date Range format validation
     (Lucky Sharma, David Smiley via Mikhail Khludnev)
• Optimizations   (6)
  1. LUCENE-8552: FieldInfos.getMergedFieldInfos no longer does any merging if there is <= 1 segment.
     (Christophe Bismuth via David Smiley)
  2. LUCENE-8590: BufferedUpdates now uses an optimized storage for buffering docvalues updates that can safe up to 80% of the heap used compared to the previous implementation and uses non-object based datastructures.
     (Simon Willnauer, Mike McCandless, Shai Erera, Adrien Grand)
  3. LUCENE-8598: Moved to the default accepted overhead ratio for packet ints in DocValuesFieldUpdats yields an up-to 4x performance improvement when applying doc values updates.
     (Simon Willnauer, Adrien Grand)
  4. LUCENE-8599: Use sparse bitset to store docs in SingleValueDocValuesFieldUpdates.
     (Simon Willnauer, Adrien Grand)
  5. LUCENE-8600: Doc-value updates get applied faster by sorting with quicksort, rather than an in-place mergesort, which needs to perform fewer swaps.
     (Adrien Grand)
  6. LUCENE-8623: Decrease I/O pressure when merging high dimensional points.
     (Ignacio Vera)
• Test Framework   (1)
  1. LUCENE-8604: TestRuleLimitSysouts now has an optional "hard limit" of bytes that can be written to stderr and stdout (anything beyond the hard limit is ignored). The default hard limit is 2 GB of logs per test class.
     (Dawid Weiss)
• Other   (3)
  1. LUCENE-8573: BKDWriter now uses FutureArrays#mismatch to compute shared prefixes.
     (Christoph Büscher via Adrien Grand)
  2. LUCENE-8605: Separate bounding box spatial logic from query logic on LatLonShapeBoundingBoxQuery.
     (Ignacio Vera)
  3. LUCENE-8609: Deprecated IndexWriter#numDocs() and IndexWriter#maxDoc() in favor of IndexWriter#getDocStats() that allows to get consistent numDocs and maxDoc stats that are not subject to concurrent changes.
     (Simon Willnauer, Nhat Nguyen)

Release 7.6.0 [2018-12-14]

• Build   (2)
  1. LUCENE-8504: Upgrade forbiddenapis to version 2.6.
     (Uwe Schindler)
  2. LUCENE-8493: Stop publishing insecure .sha1 files with releases
     (janhoy)
• Bug fixes   (13)
  1. LUCENE-8479: QueryBuilder#analyzeGraphPhrase now throws TooManyClause exception if the number of expanded path reaches the BooleanQuery#maxClause limit.
     (Jim Ferenczi)
  2. LUCENE-8522: throw InvalidShapeException when constructing a polygon and all points are coplanar.
     (Ignacio Vera)
  3. LUCENE-8531: QueryBuilder#analyzeGraphPhrase now creates one phrase query per finite strings in the graph if the slop is greater than 0. Span queries cannot be used in this case because they don't handle slop the same way than phrase queries.
     (Steve Rowe, Uwe Schindler, Jim Ferenczi)
  4. LUCENE-8524: Add the Hangul Letter Araea (interpunct) as a separator in Nori's tokenizer. This change also removes empty terms and trim surface form in Nori's Korean dictionary.
     (Trey Jones, Jim Ferenczi)
  5. LUCENE-8550: Fix filtering of coplanar points when creating linked list on polygon tesselator.
     (Ignacio Vera)
  6. LUCENE-8549: Polygon tessellator throws an error if some parts of the shape could not be processed.
     (Ignacio Vera)
  7. LUCENE-8540: Better handling of min/max values for Geo3d encoding.
     (Ignacio Vera)
  8. LUCENE-8534: Fix incorrect computation for triangles intersecting polygon edges in shape tessellation.
     (Ignacio Vera)
  9. LUCENE-8559: Fix bug where polygon edges were skipped when checking for intersections.
     (Ignacio Vera)
  10. LUCENE-8556: Use latitude and longitude instead of encoding values to check if triangle is ear when using morton optimisation.
      (Ignacio Vera)
  11. LUCENE-8586: Intervals.or() could get stuck in an infinite loop on certain indexes
      (Alan Woodward)
  12. LUCENE-8595: Fix interleaved DV update and reset. Interleaved update and reset value to the same doc in the same updates package looses an update if the reset comes before the update as well as loosing the reset if the update comes frist.
      (Simon Willnauer, Adrien Grand)
  13. LUCENE-8592: Fix index sorting corruption due to numeric overflow. The merge of sorted segments can produce an invalid sort if the sort field is an Integer/Long that uses reverse order and contains values equal to Integer/Long#MIN_VALUE. These values are always sorted first during a merge (instead of last because of the reverse order) due to this bug. Indices affected by the bug can be detected by running the CheckIndex command on a distribution that contains the fix (7.6+).
      (Jim Ferenczi, Adrien Grand, Mike McCandless, Simon Willnauer)
• New Features   (5)
  1. LUCENE-8496: Selective indexing - modify BKDReader/BKDWriter to allow users to select a fewer number of dimensions to be used for creating the index than the total number of dimensions used for field encoding. i.e., dimensions 0 to N may be used to determine how to split the inner nodes, and dimensions N+1 to D are ignored and stored as data dimensions at the leaves.
     (Nick Knize)
  2. LUCENE-8538: Add a Simple WKT Shape Parser for creating Lucene Geometries (Polygon, Line, Rectangle) from WKT format.
     (Nick Knize)
  3. LUCENE-8462: Adds an Arabic snowball stemmer based on https://github.com/snowballstem/snowball/blob/master/algorithms/arabic.sbl
     (Ryadh Dahimene via Jim Ferenczi)
  4. LUCENE-8554: Add new LatLonShapeLineQuery that queries indexed LatLonShape fields by arbitrary lines.
     (Nick Knize)
  5. LUCENE-8555: Add dateline crossing support to LatLonShapeBoundingBoxQuery.
     (Nick Knize)
• Improvements   (3)
  1. LUCENE-8521: Change LatLonShape encoding to 7 dimensions instead of 6; where the first 4 are index dimensions defining the bounding box of the Triangle and the remaining 3 data dimensions define the vertices of the triangle.
     (Nick Knize)
  2. LUCENE-8557: LeafReader.getFieldInfos is now documented and tested that it ought to return the same cached instance. MemoryIndex's impl now pre-creates the FieldInfos instead of re-calculating a new instance each time.
     (Tim Underwood, David Smiley)
  3. LUCENE-8558: Replace O(N) lookup with O(1) lookup in PerFieldMergeState#FilterFieldInfos.
     (Kranthi via Simon Willnauer)
• Other   (2)
  1. LUCENE-8523: Correct typo in JapaneseNumberFilterFactory javadocs
     (Ankush Jhalani via Alan Woodward)
  2. LUCENE-8533: Fix Javadocs of DataInput#readVInt(): Negative numbers are supported, but should be avoided.
     (Vladimir Dolzhenko via Uwe Schindler)

Release 7.5.1

• Bug Fixes   (1)
  1. LUCENE-8454: Fix incorrect vertex indexing and other computation errors in shape tessellation that would sometimes cause an infinite loop.
     (Nick Knize)

Release 7.5.0 [2018-09-24]

• API Changes   (18)
  1. LUCENE-8467: RAMDirectory, RAMFile, RAMInputStream, RAMOutputStream are deprecated
     (Dawid Weiss)
  2. LUCENE-8356: StandardFilter is deprecated
     (Alan Woodward)
  3. LUCENE-8373: ENGLISH_STOP_WORD_SET on StandardAnalyzer is deprecated. Instead use EnglishAnalyzer.ENGLISH_STOP_WORD_SET. The default constructor for StopAnalyzer is also deprecated, and a stop word set should be explicitly passed to the constructor.
     (Alan Woodward)
  4. LUCENE-8378: Add DocIdSetIterator.range static method to return an iterator matching a range of docids
     (Mike McCandless)
  5. LUCENE-8379: Add experimental TermQuery.getTermStates method
     (Mike McCandless)
  6. LUCENE-8407: Add experimental SpanTermQuery.getTermStates method
     (David Smiley)
  7. LUCENE-8390: MatchesIteratorSupplier replaced by IOSupplier
     (Alan Woodward, David Smiley)
  8. LUCENE-8397: Add DirectoryTaxonomyWriter.getCache
     (Mike McCandless)
  9. LUCENE-8387: Add experimental IndexSearcher.getSlices API to see which slices IndexSearcher is searching concurrently when it's created with an ExecutorService
     (Mike McCandless)
  10. LUCENE-8263: TieredMergePolicy's reclaimDeletesWeight has been replaced with a new deletesPctAllowed setting to control how aggressively deletes should be reclaimed.
      (Erick Erickson, Adrien Grand)
  11. LUCENE-7314: Graduate LatLonPoint and query classes to core
      (Nick Knize)
  12. LUCENE-8428: The way that oal.util.PriorityQueue creates sentinel objects has been changed from a protected method to a java.util.function.Supplier as a constructor argument.
      (Adrien Grand)
  13. LUCENE-8437: CheckIndex.Status.cantOpenSegments and missingSegmentVersion have been removed as they were not computed correctly.
      (Adrien Grand)
  14. LUCENE-8286: The UnifiedHighlighter has a new HighlightFlag.WEIGHT_MATCHES flag that will tell this highlighter to use the new MatchesIterator API as the underlying approach to navigate matching hits for a query. This mode will highlight more accurately than any other highlighter, and can mark up phrases as one span instead of word-by-word. The UH's public internal APIs changed a bit in the process.
      (David Smiley)
  15. LUCENE-8471: IndexWriter.getFlushingBytes() returns how many bytes are currently being flushed to disk.
      (Alan Woodward)
  16. LUCENE-8422: Static helper functions for Matches and MatchesIterator implementations have been moved from Matches to MatchesUtils
      (Alan Woodward)
  17. LUCENE-8343: Suggesters now require Long (versus long, previously) from weight() method while indexing, and provide double (versus long, previously) scores at lookup time
      (Alessandro Benedetti)
  18. LUCENE-8459: SearcherTaxonomyManager now has a constructor taking already opened IndexReaders, allowing the caller to pass a FilterDirectoryReader, for example.
      (Mike McCandless)
• Bug Fixes   (13)
  1. LUCENE-8445: Tighten condition when two planes are identical to prevent constructing bogus tiles when building GeoPolygons.
     (Ignacio Vera)
  2. LUCENE-8444: Prevent building functionally identical plane bounds when constructing DualCrossingEdgeIterator .
     (Ignacio Vera)
  3. LUCENE-8380: UTF8TaxonomyWriterCache inconsistency.
     (Ruslan Torobaev, Dawid Weiss)
  4. LUCENE-8164: IndexWriter silently accepts broken payload. This has been fixed via LUCENE-8165 since we are now checking for offset+length going out of bounds.
     (Robert Muir, Nhat Nyugen, Simon Willnauer)
  5. LUCENE-8370: Reproducing TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failures
     (Erick Erickson)
  6. LUCENE-8376, LUCENE-8371: ConditionalTokenFilter.end() would not propagate correctly if the last token in the stream was subsequently dropped; FixedShingleFilter did not set position increment in end()
     (Alan Woodward)
  7. LUCENE-8395: WordDelimiterGraphFilter would incorrectly insert a hole into a TokenStream if a token consisting entirely of delimiter characters was encountered, but preserve_original was set.
     (Alan Woodward)
  8. LUCENE-8398: TieredMergePolicy.getMaxMergedSegmentMB has rounding error
     (Erick Erickson)
  9. LUCENE-8429: DaciukMihovAutomatonBuilder is no longer prone to stack overflows by enforcing a maximum term length.
     (Adrien Grand)
  10. LUCENE-8441: IndexWriter now checks doc value type for index sort fields and fails the document if they are not compatible.
      (Jim Ferenczi, Mike McCandless)
  11. LUCENE-8458: Adjust initialization condition of PendingSoftDeletes and ensures it is initialized before accepting deletes
      (Simon Willnauer, Nhat Nguyen)
  12. LUCENE-8466: IndexWriter.deleteDocs(Query... query) incorrectly applies deletes on flush if the index is sorted.
      (Adrien Grand, Jim Ferenczi, Vish Ramachandran)
  13. LUCENE-8502: Allow access to delegate in FilterCodecReader. FilterCodecReader didn't allow access to it's delegate like other filter readers. This adds a new #getDelegate method to access the wrapped reader.
      (Simon Willnauer)
• Changes in Runtime Behavior   (3)
  1. LUCENE-7976: TieredMergePolicy now respects maxSegmentSizeMB by default when executing findForcedMerges and findForcedDeletesMerges
     (Erick Erickson)
  2. LUCENE-8263: TieredMergePolicy now reclaims deleted documents more aggressively by default ensuring that no more than ~1/3 of the index size is used by deleted documents.
     (Adrien Grand)
  3. LUCENE-8503: Call #getDelegate instead of direct member access during unwrap. Filter*Reader instances access the member or the delegate directly instead of calling getDelegate(). In order to track access of the delegate these methods should call #getDelegate()
     (Simon Willnauer)
• Improvements   (14)
  1. LUCENE-8468: A ByteBuffer based Directory implementation.
     (Dawid Weiss)
  2. LUCENE-8447: Add DISJOINT and WITHIN support to LatLonShape queries.
     (Nick Knize)
  3. LUCENE-8440: Add support for indexing and searching Line and Point shapes using LatLonShape encoding
     (Nick Knize)
  4. LUCENE-8435: Add new LatLonShapePolygonQuery for querying indexed LatLonShape fields by arbitrary polygons
     (Nick Knize)
  5. LUCENE-8367: Make per-dimension drill down optional for each facet dimension
     (Mike McCandless)
  6. LUCENE-8396: Add Points Based Shape Indexing and Search that decomposes shapes into a triangular mesh and indexes individual triangles as a 6 dimension point
     (Nick Knize)
  7. LUCENE-8345, GitHub PR #392: Remove instantiation of redundant wrapper classes for primitives; add wrapper class constructors to forbiddenapis.
     (Michael Braun via Uwe Schindler)
  8. LUCENE-8415: Clean up Directory contracts and JavaDoc comments.
     (Dawid Weiss)
  9. LUCENE-8414: Make segmentInfos private in IndexWriter
     (Simon Willnauer, Nhat Nguyen)
  10. LUCENE-8446: The UnifiedHighlighter's DefaultPassageFormatter now treats overlapping matches in the passage as merged (as if one larger match).
      (David Smiley)
  11. LUCENE-8460: Better argument validation in StoredField.
      (Namgyu Kim)
  12. LUCENE-8432: TopFieldComparator stops comparing documents if the index is sorted, even if hits still need to be visited to compute the hit count.
      (Nikolay Khitrin)
  13. LUCENE-8422: IntervalQuery now returns useful Matches
      (Alan Woodward)
  14. LUCENE-7862: Store the real bounds of the leaf cells in the BKD index when the number of dimensions is bigger than 1. It improves performance when there is correlation between the dimensions, for example ranges.
      (Ignacio Vera, Adrien Grand)
• Build   (1)
  1. LUCENE-5143: Stop publishing KEYS file with each version, use topmost lucene/KEYS file only. The buildAndPushRelease.py script validates that RM's PGP key is in the KEYS file. Remove unused 'copy-to-stage' and '-dist-keys' targets from ant build.
     (janhoy)
• Other   (9)
  1. LUCENE-8485: Update randomizedtesting to version 2.6.4.
     (Dawid Weiss)
  2. LUCENE-8366: Upgrade to ICU 62.1. Emoji handling now uses Unicode 11's Extended_Pictographic property.
     (Robert Muir)
  3. LUCENE-8408: original Highlighter: Remove obsolete static AttributeFactory instance in TokenStreamFromTermVector.
     (Michael Braun, David Smiley)
  4. LUCENE-8420: Upgrade OpenNLP to 1.9.0 so OpenNLP tool can read the new model format which 1.8.x cannot read. 1.9.0 can read the old format.
     (Koji Sekiguchi)
  5. LUCENE-8453: Add documentation to analysis factories of Korean (Nori) analyzer module.
     (Tomoko Uchida via Uwe Schindler)
  6. LUCENE-8455: Upgrade ECJ compiler to 4.6.1 in lucene/common-build.xml
     (Erick Erickson)
  7. LUCENE-8456: Upgrade Apache Commons Compress to v1.18
     (Steve Rowe)
  8. LUCENE-765: Improved org.apache.lucene.index javadocs.
     (Mike Sokolov)
  9. LUCENE-8476: Remove redundant nullity check and switch to optimized List.sort in the Korean's user dictionary.
     (Namgyu Kim)

Release 7.4.1

• Bug Fixes   (4)
  1. LUCENE-8365: Fix ArrayIndexOutOfBoundsException in UnifiedHighlighter. This fixes a "off by one" error in the UnifiedHighlighter's code that is only triggered when two nested SpanNearQueries contain the same term.
     (Marc-Andre Morissette via Simon Willnauer)
  2. LUCENE-8381: Fix IndexWriter incorrectly interprets hard-deletes as soft-deletes while wrapping reader for merges.
     (Simon Willnauer, Nhat Nguyen)
  3. LUCENE-8384: Fix missing advance docValues generation while handling docValues update in PendingSoftDeletes.
     (Simon Willnauer, Nhat Nguyen)
  4. LUCENE-8472: Always rewrite the soft-deletes merge retention query.
     (Adrien Grand, Nhat Nguyen)

Release 7.4.0 [2018-06-27]

• Upgrading   (1)
  1. LUCENE-8344: If you are using the AnalyzingSuggester or FuzzySuggester subclass, and if you explicitly use the preservePositionIncrements=false setting (not the default), then you ought to rebuild your suggester index. If you don't, queries or indexed data with trailing position gaps (e.g. stop words) may not work correctly.
     (David Smiley, Jim Ferenczi)
• API Changes   (3)
  1. LUCENE-8242: IndexSearcher.createNormalizedWeight() has been deprecated. Instead use IndexSearcher.createWeight(), rewriting the query first.
     (Alan Woodward)
  2. LUCENE-8248: MergePolicyWrapper is renamed to FilterMergePolicy and now also overrides getMaxCFSSegmentSizeMB
     (Mike Sokolov via Mike McCandless)
  3. LUCENE-8303: LiveDocsFormat is now only responsible for (de)serialization of live docs.
     (Adrien Grand)
• Changes in Runtime Behavior   (2)
  1. LUCENE-8309: Live docs are no longer backed by a FixedBitSet.
     (Adrien Grand)
  2. LUCENE-8330: Detach IndexWriter from MergePolicy. MergePolicy now instead of requiring IndexWriter as a hard dependency expects a MergeContext which IndexWriter implements.
     (Simon Willnauer, Robert Muir, Dawid Weiss, Mike McCandless)
• New Features   (19)
  1. LUCENE-8200: Allow doc-values to be updated atomically together with a document. Doc-Values updates now can be used as a soft-delete mechanism to all keeping several version of a document or already deleted documents around for later reuse. See "IW.softUpdateDocument(...)" for reference.
     (Simon Willnauer)
  2. LUCENE-8197: A new FeatureField makes it easy and efficient to integrate static relevance signals into the final score.
     (Adrien Grand, Robert Muir)
  3. LUCENE-8202: Add a FixedShingleFilter
     (Alan Woodward, Adrien Grand, Jim Ferenczi)
  4. LUCENE-8125: ICUTokenizer support for emoji/emoji sequence tokens.
     (Robert Muir)
  5. LUCENE-8196, LUCENE-8300: A new IntervalQuery in the sandbox allows efficient proximity searches based on minimum-interval semantics.
     (Alan Woodward, Adrien Grand, Jim Ferenczi, Simon Willnauer, Matt Weber)
  6. LUCENE-8233: Add support for soft deletes to IndexWriter delete accounting. Soft deletes are accounted for inside the index writer and therefor also by merge policies. A SoftDeletesRetentionMergePolicy is added that allows to selectively carry over soft_deleted document across merges for retention policies
     (Simon Willnauer, Mike McCandless, Robert Muir)
  7. LUCENE-8237: Add a SoftDeletesDirectoryReaderWrapper that allows to respect soft deletes if the reader is opened form a directory.
     (Simon Willnauer, Mike McCandless, Uwe Schindler, Adrien Grand)
  8. LUCENE-8229, LUCENE-8270: Add a method Weight.matches(LeafReaderContext, doc) that returns an iterator over matching positions for a given query and document. This allows exact hit extraction and will enable implementation of accurate highlighters.
     (Alan Woodward, Adrien Grand, David Smiley)
  9. LUCENE-8249: Implement Matches API for phrase queries
     (Alan Woodward, Adrien Grand)
  10. LUCENE-8246: Allow to customize the number of deletes a merge claims. This helps merge policies in the soft-delete case to correctly implement retention policies without triggering uncessary merges.
      (Simon Willnauer, Mike McCandless)
  11. LUCENE-8231: A new analysis module (nori) similar to Kuromoji but to handle Korean using mecab-ko-dic and morphological analysis.
      (Robert Muir, Jim Ferenczi)
  12. LUCENE-8265: WordDelimter/GraphFilter now have an option to skip tokens marked with KeywordAttribute
      (Mike Sokolov via Mike McCandless)
  13. LUCENE-8297: Add IW#tryUpdateDocValues(Reader, int, Fields...) IndexWriter can update doc values for a specific term but this might affect all documents containing the term. With tryUpdateDocValues users can update doc-values fields for individual documents. This allows for instance to soft-delete individual documents.
      (Simon Willnauer)
  14. LUCENE-8298: Allow DocValues updates to reset a value. Passing a DV field with a null value to IW#updateDocValues or IW#tryUpdateDocValues will now remove the value from the provided document. This allows to undelete a soft-deleted document unless it's been claimed by a merge.
      (Simon Willnauer)
  15. LUCENE-8273: ConditionalTokenFilter allows analysis chains to skip particular token filters based on the attributes of the current token. This generalises the keyword token logic currently used for stemmers and WDF. It is integrated into CustomAnalyzer by using the `when` and `whenTerm` builder methods, and a new ProtectedTermFilter is added as an example.
      (Alan Woodward, Robert Muir, David Smiley, Steve Rowe, Mike Sokolov)
  16. LUCENE-8310: Ensure IndexFileDeleter accounts for pending deletes. Today we fail creating the IndexWriter when the directory has a pending delete. Yet, this is mainly done to prevent writing still existing files more than once. IndexFileDeleter already accounts for that for existing files which we can now use to also take pending deletes into account which ensures that all file generations per segment always go forward.
      (Simon Willnauer)
  17. LUCENE-7960: Add preserveOriginal option to the NGram and EdgeNGram filters.
      (Ingomar Wesp, Shawn Heisey via Robert Muir)
  18. LUCENE-8335: Enforce soft-deletes field up-front. Soft deletes field must be marked as such once it's introduced and can't be changed after the fact.
      (Nhat Nguyen via Simon Willnauer)
  19. LUCENE-8332: New ConcatenateGraphFilter for concatenating all tokens into one (or more in the event of a graph input). This is useful for fast analyzed exact-match lookup, suggesters, and as a component of a named entity recognition system. This was excised out of CompletionTokenStream in the NRT doc suggester.
      (David Smiley, Jim Ferenczi)
• Bug Fixes   (19)
  1. LUCENE-8221: MoreLikeThis.setMaxDocFreqPct can easily int-overflow on larger indexes.

  2. LUCENE-8266: Detect bogus tiles when creating a standard polygon and throw a TileException.
     (Ignacio Vera)
  3. LUCENE-8234: Fixed bug in how spatial relationship is computed for GeoStandardCircle when it covers the whole world.
     (Ignacio Vera)
  4. LUCENE-8236: Filter duplicated points when creating GeoPath shapes to avoid creation of bogus planes.
     (Ignacio Vera)
  5. LUCENE-8243: IndexWriter.addIndexes(Directory[]) did not properly preserve index file names for updated doc values fields
     (Simon Willnauer, Michael McCandless, Nhat Nguyen)
  6. LUCENE-8275: Push up #checkPendingDeletes to Directory to ensure IW fails if the directory has pending deletes files even if the directory is filtered or a FileSwitchDirectory
     (Simon Willnauer, Robert Muir)
  7. LUCENE-8244: Do not leak open file descriptors in SearcherTaxonomyManager's refresh on exception
     (Mike McCandless)
  8. LUCENE-8305: ComplexPhraseQuery.rewrite now handles an embedded MultiTermQuery that rewrites to a MatchNoDocsQuery instead of throwing an exception.
     (Bjarke Mortensen, Andy Tran via David Smiley)
  9. LUCENE-8287: Ensure that empty regex completion queries always return no results.
     (Julie Tibshirani via Jim Ferenczi)
  10. LUCENE-8317: Prevent concurrent deletes from being applied during full flush. Future deletes could potentially be exposed to flushes/commits/refreshes if the amount of RAM used by deletes is greater than half of the IW RAM buffer.
      (Simon Willnauer)
  11. LUCENE-8320: Fix WindowsFS to correctly account for rename and hardlinks.
      (Simon Willnauer, Nhat Nguyen)
  12. LUCENE-8328: Ensure ReadersAndUpdates consistently executes under lock.
      (Nhat Nguyen via Simon Willnauer)
  13. LUCENE-8325: Fixed the smartcn tokenizer to not split UTF-16 surrogate pairs.
      (chengpohi via Jim Ferenczi)
  14. LUCENE-8186: LowerCaseTokenizerFactory now lowercases text in multi-term queries.
      (Tim Allison via Adrien Grand)
  15. LUCENE-8278: Some end-of-input no-scheme domain-only URL tokens are typed as <ALPHANUM> rather than <URL>.
      (Junte Zhang, Steve Rowe)
  16. LUCENE-8355: Prevent IW from opening an already dropped segment while DV updates are written.
      (Nhat Nguyen via Simon Willnauer)
  17. LUCENE-8344: TokenStreamToAutomaton (used by some suggesters) was not ignoring a trailing position increment when the preservePositionIncrement setting is false.
      (David Smiley, Jim Ferenczi)
  18. LUCENE-8357: FunctionScoreQuery.boostByQuery() and boostByValue() were producing truncated Explanations
      (Markus Jelsma, Alan Woodward)
  19. LUCENE-8360: NGramTokenFilter and EdgeNGramTokenFilter did not correctly set position increments in end()
      (Alan Woodward)
• Other   (9)
  1. LUCENE-8301: Update randomizedtesting to 2.6.0.
     (Dawid Weiss)
  2. LUCENE-8299: Geo3D wrapper uses new polygon method factory that gives better support for polygons with many points (>100).
     (Ignacio vera)
  3. LUCENE-8261: InterpolatedProperties.interpolate and recursive property references.
     (Steve Rowe, Dawid Weiss)
  4. LUCENE-8228: removed obsolete IndexDeletionPolicy clone() requirements from the javadoc.
     (Dawid Weiss)
  5. LUCENE-8219: Use a realistic estimate of the number of nodes and links in LevensteinAutomaton.java, to save reallocation of arrays.
     (Christian Ziech)
  6. LUCENE-8214: Improve selection of testPoint for GeoComplexPolygon.
     (Ignacio Vera)
  7. SOLR-10912: Add automatic patch validation.
     (Mano Kovacs, Steve Rowe)
  8. LUCENE-8122, LUCENE-8175: Upgrade analysis/icu to ICU 61.1.
     (Robert Muir, Adrien Grand, Uwe Schindler)
  9. LUCENE-8291: Remove QueryTemplateManager utility class from XML queryparser. This class is just a general XML transforming tool (using property files and XSLT) and has nothing to do with query parsing. It can easily be implemented using more sophisticated libraries or using XSL transformers from the JDK. This change also removes the Lucene demo webapp to prevent XSS issues in untested/unmaintained code.
     (Uwe Schindler)
• Build   (2)
  1. LUCENE-7935: Publish .sha512 hash files with the release artifacts and stop publishing .md5 hashes since the algorithm is broken
     (janhoy)
  2. LUCENE-8230: Upgrade forbiddenapis to version 2.5.
     (Uwe Schindler)
• Documentation   (1)
  1. LUCENE-8238: Improve WordDelimiterFilter and WordDelimiterGraphFilter javadocs
     (Mike Sokolov via Mike McCandless)

Release 7.3.1 [2018-05-15]

• Bug fixes   (1)
  1. LUCENE-8254: LRUQueryCache could cause IndexReader to hang on close, when shared with another reader with no CacheHelper
     (Alan Woodward, Simon Willnauer, Adrien Grand)

Release 7.3.0 [2018-04-04]