Summary

• Adds a new hnsw:initial_capacity metadata parameter that controls the initial max_elements for the HNSW index
• Replaces all hardcoded DEFAULT_CAPACITY references in index initialization and resize logic with the user-configurable value
• Default value (1000) preserves existing behavior — no breaking changes

Problem

When adding many vectors to a collection incrementally, the HNSW index starts at DEFAULT_CAPACITY=1000 and grows by resize_factor=1.2x each time capacity is exceeded:

1000 → 1200 → 1440 → 1728 → 2074 → 2489 → 2987  (6 resizes for ~3000 vectors)

Each resize_index() call in hnswlib allocates a new contiguous buffer and frees the old one. However, memory allocators (glibc and jemalloc) typically do not return these large freed buffers to the OS. The old buffers remain mapped in the process address space, causing RSS to grow far beyond actual data requirements.

In our production environment, we observed 50GB+ RSS for 54MB of vector data (2786 vectors, 1024-dim embeddings). Analysis of /proc/PID/smaps revealed a descending pattern of anonymous memory mappings (7GB, 6GB, 5GB, 4GB...) — each one a ghost of a previous resize operation.

Solution

Allow users to pre-size the index to avoid unnecessary resizes:

collection = client.get_or_create_collection(
    name="my_collection",
    metadata={"hnsw:initial_capacity": 5000}  # match expected dataset size
)

This follows the existing pattern of hnsw:* metadata parameters (hnsw:resize_factor, hnsw:batch_size, etc.).

Changes

Test plan

• Existing tests pass (no behavior change with default value 1000)
• Property-based tests now exercise hnsw:initial_capacity with random values
• Manual verification: creating a collection with hnsw:initial_capacity=5000 and adding 3000 vectors results in 0 resize operations (vs 6 previously)

🤖 Generated with Claude Code