This PR addresses a potential buffer read overflow issue resulting from incorrect blob size handling during vector preprocessing.
The core issue was confusion between storedDataSize (processed blob size) and the actual input blob size, passing dataSize (stored data size) to the preprocessors instead of the actual input blob size.
This mismatch caused preprocessors to copy more data than available in the input blob, leading to out-of-bounds reads.

Correct Behavior

The system should properly distinguish between two different size concepts:

• inputBlobSize: Size of the original input vector blob (dim * sizeof(type))
• storedDataSize: Size of vectors after preprocessing (may include extra bytes for normalization)

Data Size Relationships:

Non-Tiered Indexes:

• No preprocessing required: storedDataSize = inputBlobSize = dim * sizeof(type)
• INT8/UINT8 + Cosine: storedDataSize = inputBlobSize + 4 (norm stored at end)

Tiered Indexes:

• Frontend index: Follows non-tiered rules above
• Backend index: Receives preprocessed blobs, so storedDataSize = inputBlobSize

What Was Fixed

The fix ensures memory safety by properly distinguishing between input blob sizes and stored data sizes throughout the vector similarity pipeline.

The fix was implemented by:

1. Adding inputBlobSize as a new member to the VecSimIndexAbstract class and AbstractIndexInitParams struct to explicitly track the original input vector size
2. Modifying all preprocessor API calls to use inputBlobSize instead of storedDataSize when processing input vectors, ensuring preprocessors only access the actual available input data
3. Establishing clear size relationships where inputBlobSize represents the original input size (dim * sizeof(type)) while storedDataSize represents the final processed size (which may include extra bytes for normalization

Additional Changes

• Centralized Factory Logic:

  • File: src/VecSim/index_factories/factory_utils.h (new)
  • Changes: Created NewAbstractInitParams() template function to standardize parameter initialization across all factories, eliminating code duplication and ensuring consistent inputBlobSize calculation logic throughout the codebase

• Renamed dataSize → StoredDataSize for Clarity

• Test Fixes - Buffer Overflow in Element Size Estimation:

  • Files: test_int8.cpp, test_uint8.cpp
  • Fix: Fixed buffer overflow in INT8/UINT8 element size estimation tests where the TieredIndexParams was created from HNSWParams that didn't explicitly set the type field.
    The buffer overflow occurred because when the test called addVector, DataBlock::addElement() tried to copy dim * sizeof(float) bytes (16 bytes for dim=4), but the allocated buffer was only dim * sizeof(int8_t) bytes (4 bytes for dim=4), leading to out-of-bounds memory access during vector operations.

• Test Fixes - Compiler Warnings:

  • Files: test_svs.cpp, test_svs_multi.cpp, test_svs_tiered.cpp
  • Fix: Fixed compiler warnings about potential division by zero by making quantBits constexpr. The compiler couldn't prove at compile-time that quantBits was never zero, even with runtime checks. Using constexpr moves the evaluation to compile-time, allowing if constexpr to eliminate the division code when quantBits == VecSimSvsQuant_NONE, satisfying static analysis requirements.

• Fixed Element Size Estimation in Factory Functions:

  • Files: src/VecSim/index_factories/brute_force_factory.cpp, src/VecSim/index_factories/hnsw_factory.cpp
  • Changes: Updated EstimateElementSize() functions to use VecSimParams_GetStoredDataSize() instead of dim * sizeof(type) for accurate memory estimation. This ensures size calculations account for additional storage requirements (such as normalization data for INT8/UINT8 with Cosine metric) and provides consistent estimation across all index types. Added comprehensive test coverage for Cosine metric element size estimation in INT8 and UINT8 test suites to validate the fix.

• Fixed Query Blob Access in Tiered Batch Iterator:

  • Files: src/VecSim/batch_iterator.h, src/VecSim/algorithms/hnsw/hnsw_tiered.h
  • Changes: Made getQueryBlob() virtual in the base VecSimBatchIterator class and overrode it in TieredHNSW_BatchIterator to properly delegate to the frontend iterator. This was necessary because the tiered batch iterator itself doesn't own any query vector - it relies on its internal frontend and backend iterators to manage their own query copies.

• Enhanced Batch Iterator Testing for INT8 Cosine Metric:

  • Files: tests/unit/test_int8.cpp, src/VecSim/algorithms/hnsw/hnsw_tiered.h, src/VecSim/batch_iterator.h
  • Changes: Extended the CosineBlobCorrectness test to explicitly validate query blob transfer from frontend to backend in tiered batch iterators:
  • Added direct verification that the query blob (including the appended norm) is correctly preserved during the transfer from the frontend BruteForce iterator to the backend HNSW iterator.