Add on disk 4x compression with Faiss by naveentatikonda · Pull Request #2425 · opensearch-project/k-NN

naveentatikonda · 2025-01-23T05:15:53Z

Description

Add on disk 4x compression with Faiss which accepts fp32 vectors as input and dynamically quantizes them into byte sized vectors using the Faiss SQ8 quantizer.

Related Issues

Resolves #1723

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

shatejas

Reviewed partially

shatejas · 2025-01-24T01:55:00Z

    }
 }

+jlong IndexService::initIndexFromTemplate(


The only difference between this and initIndex is the index creation call to faiss, can we abstract out the logic and reuse the rest please. you can pass in the pointer returned by faiss to reuse the logic and set the index uniq pointer maybe.

refactored as discussed and also validated that the index is getting deleted

naveentatikonda · 2025-01-27T19:27:25Z

Update - After looking at the benchmarks and having a chat with other maintainers of k-NN we want to put this feature on hold from releasing in 2.19. Will run more benchmarking tests and compare them with lucene (with rescoring POC) and then decide if we need to switch the default engine for on_disk 4x compression.

navneet1v · 2025-01-24T22:45:41Z

+            if ((quantizationParams.getTypeIdentifier()).equals(
+                ScalarQuantizationParams.generateTypeIdentifier(ScalarQuantizationType.EIGHT_BIT)
+            )) {


to ensure that NPE doesn't come in case quantizationParams.getTypeIdentifier() == null, we should reverse the check.

ScalarQuantizationParams.generateTypeIdentifier(ScalarQuantizationType.EIGHT_BIT).equal(quantizationParams.getTypeIdentifier())

navneet1v · 2025-01-24T22:46:52Z

+                quantizationState = quantizationService.train(quantizationParams, knnVectorValues, totalLiveDocs, fieldInfo);
+            } else {
+                initQuantizationStateWriterIfNecessary();
+                quantizationState = quantizationService.train(quantizationParams, knnVectorValues, totalLiveDocs, fieldInfo);
+                quantizationStateWriter.writeState(fieldInfo.getFieldNumber(), quantizationState);
+            }


I think this whole logic can be simplified where we create the QS first and then we just see if writer needs to init and it needs to write the state,.

navneet1v · 2025-01-24T22:48:27Z

+        return (indexInfo.getQuantizationState() instanceof ByteScalarQuantizationState);
+    }
+
+    private byte[] getIndexTemplate(BuildIndexParams indexInfo) {


nit pick: make the param final for all the functions

navneet1v · 2025-01-24T22:49:12Z

        int dimensions;

-        if (quantizationState != null) {
+        if (quantizationState != null && !(quantizationState instanceof ByteScalarQuantizationState)) {


need java doc on this. This kind of instanceOf check is making me nervous. can we think of something here.

navneet1v · 2025-01-24T22:53:05Z

+    public QuantizationState train(final TrainingRequest<float[]> trainingRequest, final FieldInfo fieldInfo) throws IOException {
+        return null;
+    }


same as above

navneet1v · 2025-01-24T22:53:12Z

+    public QuantizationState train(final TrainingRequest<float[]> trainingRequest, final FieldInfo fieldInfo) throws IOException {
+        return null;
+    }


same as above

navneet1v · 2025-01-24T22:53:47Z

     */
    QuantizationState train(TrainingRequest<T> trainingRequest) throws IOException;

+    QuantizationState train(TrainingRequest<T> trainingRequest, FieldInfo fieldInfo) throws IOException;


please add java doc and also why we need this function? I thought we are keeping QF free from fieldInfo and other things.

navneet1v · 2025-01-24T22:55:39Z

+import static org.opensearch.knn.common.FieldInfoExtractor.extractVectorDataType;
+import static org.opensearch.knn.index.codec.transfer.OffHeapVectorTransferFactory.getVectorTransfer;
+
+public class ByteScalarQuantizer implements Quantizer<float[], byte[]> {


Please add java doc on all your new classes.

navneet1v · 2025-01-24T23:02:20Z

+        if (sampledIndices.length == 0) {
+            return null;
+        }


should we have some logs here.

Signed-off-by: Naveen Tatikonda <[email protected]>

naveentatikonda added Features Introduces a new unit of functionality that satisfies a requirement backport 2.x labels Jan 23, 2025

naveentatikonda force-pushed the faiss_ondisk_4x branch from 101c94f to 58a66e4 Compare January 23, 2025 05:18

naveentatikonda added the v2.19.0 label Jan 23, 2025

naveentatikonda force-pushed the faiss_ondisk_4x branch 2 times, most recently from 6721604 to 7739606 Compare January 23, 2025 23:03

naveentatikonda changed the base branch from main to 2.x January 23, 2025 23:03

naveentatikonda added backport main and removed backport 2.x labels Jan 23, 2025

naveentatikonda force-pushed the faiss_ondisk_4x branch from 7739606 to e09cdfd Compare January 23, 2025 23:05

shatejas reviewed Jan 24, 2025

View reviewed changes

naveentatikonda marked this pull request as ready for review January 24, 2025 17:28

naveentatikonda requested review from 0ctopus13prime, VijayanB, heemin32, jmazanec15, junqiu-lei, luyuncheng, martin-gaievski, navneet1v, ryanbogan and vamshin as code owners January 24, 2025 17:28

naveentatikonda force-pushed the faiss_ondisk_4x branch 2 times, most recently from 18c4a8a to 4449fde Compare January 25, 2025 17:05

naveentatikonda requested a review from shatejas January 25, 2025 17:08

vibrantvarun reviewed Jan 25, 2025

View reviewed changes

naveentatikonda force-pushed the faiss_ondisk_4x branch from 0ea7bc6 to 2bb9e05 Compare January 26, 2025 05:21

naveentatikonda removed the v2.19.0 label Jan 27, 2025

navneet1v reviewed Jan 28, 2025

View reviewed changes

navneet1v mentioned this pull request Jan 29, 2025

Add a Faiss codec for KNN searches apache/lucene#14178

Merged

naveentatikonda closed this Jun 4, 2025

naveentatikonda deleted the faiss_ondisk_4x branch June 4, 2025 04:02

naveentatikonda reopened this Jun 4, 2025

naveentatikonda requested a review from Vikasht34 as a code owner June 4, 2025 04:12

naveentatikonda force-pushed the faiss_ondisk_4x branch from 2bb9e05 to e3bfbf1 Compare June 4, 2025 04:20

naveentatikonda changed the base branch from 2.x to main June 4, 2025 04:20

naveentatikonda added 10 commits June 3, 2025 23:25

Add support for Faiss OnDisk 4x compression

8d60800

Signed-off-by: Naveen Tatikonda <[email protected]>

Predefined configuration changes

36cac07

Signed-off-by: Naveen Tatikonda <[email protected]>

Ingestion and Querying

3344ca7

Signed-off-by: Naveen Tatikonda <[email protected]>

Optimize create index from template for 4x compression

b9eef93

Signed-off-by: Naveen Tatikonda <[email protected]>

Add Backwards compatibility for Lucene 4x

48cce0d

Signed-off-by: Naveen Tatikonda <[email protected]>

Add ivf validation

2fe0069

Signed-off-by: Naveen Tatikonda <[email protected]>

Fix vector datatype

54b1112

Signed-off-by: Naveen Tatikonda <[email protected]>

Fix failing tests

ada17b8

Signed-off-by: Naveen Tatikonda <[email protected]>

Address review comments and add tests

96fdebd

Signed-off-by: Naveen Tatikonda <[email protected]>

Rollback changes to support Faiss 4x with OnDisk

d86a1b6

Signed-off-by: Naveen Tatikonda <[email protected]>

naveentatikonda force-pushed the faiss_ondisk_4x branch from e3bfbf1 to d86a1b6 Compare June 4, 2025 04:25

naveentatikonda removed the backport main label Jun 4, 2025

Conversation

naveentatikonda commented Jan 23, 2025

Description

Related Issues

Check List

Uh oh!

shatejas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

naveentatikonda commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

naveentatikonda commented Jan 27, 2025 •

edited

Loading