Better detect vector module in non-default setups (e.g., custom module layers) #12677
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
uschindler
commented
Oct 13, 2023
lucene/core/src/java/org/apache/lucene/internal/vectorization/VectorizationProvider.java
Outdated
Show resolved
Hide resolved
ChrisHegarty
approved these changes
Oct 15, 2023
lucene/core/src/java/org/apache/lucene/internal/vectorization/VectorizationProvider.java
Show resolved
Hide resolved
asfgit
pushed a commit
that referenced
this pull request
Oct 15, 2023
clayburn
added a commit
to runningcode/lucene
that referenced
this pull request
Oct 20, 2023
…ache.org * upstream/main: (239 commits) Bound the RAM used by the NodeHash (sharing common suffixes) during FST compilation (apache#12633) Fix index out of bounds when writing FST to different metaOut (apache#12697) (apache#12698) Avoid object construction when linear searching arcs (apache#12692) chore: update the Javadoc example in Analyzer (apache#12693) coorect position on entry in CHANGES.txt Refactor ByteBlockPool so it is just a "shift/mask big array" (apache#12625) Extract the hnsw graph merging from being part of the vector writer (apache#12657) Specialize `BlockImpactsDocsEnum#nextDoc()`. (apache#12670) Speed up TestIndexOrDocValuesQuery. (apache#12672) Remove over-counting of deleted terms (apache#12586) Use MergeSorter in StableStringSorter (apache#12652) Use radix sort to speed up the sorting of terms in TermInSetQuery (apache#12587) Add timeouts to github jobs. Estimates taken from empirical run times (actions history), with a generous buffer added. (apache#12687) Optimize OnHeapHnswGraph's data structure (apache#12651) Add createClassLoader to replicator permissions (block specific to jacoco). (apache#12684) Move changes entry before backporting CHANGES Move testing properties to provider class (no classloading deadlock possible) and fallback to default provider in non-test mode simple cleanups to vector code (apache#12680) Better detect vector module in non-default setups (e.g., custom module layers) (apache#12677) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @ChrisHegarty,
while working on #12667 and #12676 I noticed that the code which detects if vector module is enabled and readable has some limitations because it only looks into the boot layer.
If you create your own module layer with Lucene and the incubator module, Lucene does not detect it correctly. This code uses the
ModuleLayerof the Lucene module and usesfindModuleto check if it is there (we do this in other Lucene code in the same way, e.g. to detectsun.misc.Unsafe,...). Only in case of unnamed module (thengetLayer()returns null), then the code uses boot layer.In addition to #12676 it also only calls
thisModule.addReads(vectorModule)when it actually wants to enable it. For the pure check (e.g., java 17) ist does not add a read.