Skip to content

in ram single mmap file#7971

Merged
timvisee merged 11 commits intodevfrom
in-ram-single-mmap-file
Jan 23, 2026
Merged

in ram single mmap file#7971
timvisee merged 11 commits intodevfrom
in-ram-single-mmap-file

Conversation

@generall
Copy link
Copy Markdown
Member

@generall generall commented Jan 22, 2026

  • WIP: introduce new vector store type
  • handling of InRamMmap
  • fmt
  • feature-flag
  • fmt

@ffuugoo ffuugoo mentioned this pull request Jan 22, 2026
Merged
9 tasks
ffuugoo
ffuugoo reviewed Jan 22, 2026
View reviewed changes
Comment thread lib/collection/src/collection_manager/optimizers/segment_optimizer.rs Outdated
Comment thread lib/segment/src/vector_storage/dense/appendable_dense_vector_storage.rs
ffuugoo
ffuugoo requested changes Jan 22, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ffuugoo ffuugoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same everywhere, pls.

Comment thread lib/segment/src/segment_constructor/segment_constructor_base.rs Outdated
@qdrant qdrant deleted a comment from coderabbitai Bot Jan 22, 2026
timvisee
timvisee reviewed Jan 22, 2026
View reviewed changes
Comment thread lib/common/common/src/flags.rs Outdated
timvisee
timvisee reviewed Jan 22, 2026
View reviewed changes
Comment thread lib/segment/src/vector_storage/dense/memmap_dense_vector_storage.rs Outdated
@timvisee @ffuugoo
Use if else
e7eeb03 
Co-authored-by: Roman Titov <[email protected]>
@timvisee timvisee force-pushed the in-ram-single-mmap-file branch from a9ec179 to e7eeb03 Compare January 22, 2026 15:25
coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

@qdrant qdrant deleted a comment from coderabbitai Bot Jan 22, 2026
@ffuugoo
Copy link
Copy Markdown
Contributor

ffuugoo commented Jan 22, 2026

I have a proposal how to make this sane, because right now it's completely unreadable.

match (vector_config.storage_type, vector_storage.multivector_config) {
    (VectorStorageType::Mmap, None) => open_mmap_vector_storage(..., true),
    (VectorStorageType::InRamMmap, None) => open_mmap_vector_storage(..., true),
    (VectorStorageType::Mmap, Some(multivector)) => open_appendable_memmap_multi_vector_storage(..., false),
    (VectorStorageType::InRamMmap, Some(multivector)) => open_appendable_memmap_multi_vector_storage(..., true),
    // etc...
}

@coderabbitai

This comment was marked as resolved.

Sign in to view
coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

ffuugoo
ffuugoo approved these changes Jan 22, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ffuugoo ffuugoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, we can fix cosmetic stuff later

@github-actions github-actions Bot mentioned this pull request Jan 22, 2026
Open
coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

ffuugoo
ffuugoo approved these changes Jan 22, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ffuugoo ffuugoo left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image

@timvisee
Copy link
Copy Markdown
Member

timvisee commented Jan 23, 2026

Since this is feature flagged it should be safe to merge. Doing so now.

@timvisee timvisee merged commit f1fb912 into dev Jan 23, 2026
15 checks passed
@timvisee timvisee deleted the in-ram-single-mmap-file branch January 23, 2026 10:12
@IvanPleshkov IvanPleshkov mentioned this pull request Jan 23, 2026
Merged
generall added a commit that referenced this pull request Feb 9, 2026
@generall @timvisee @ffuugoo
in ram single mmap file (#7971)
b2ad969 
* WIP: introduce new vector store type

* handling of InRamMmap

* fmt

* feature-flag

* fmt

* Use if else

Co-authored-by: Roman Titov <[email protected]>

* Update lib/common/common/src/flags.rs

Co-authored-by: Tim Visée <[email protected]>

* also choose madvise for single-file in-ram-mmap

* simplify generics

* gpu fix

* fix bug

---------

Co-authored-by: Tim Visée <[email protected]>
Co-authored-by: Roman Titov <[email protected]>
@timvisee timvisee mentioned this pull request Feb 17, 2026
Merged
5 tasks
@coderabbitai coderabbitai Bot mentioned this pull request Feb 24, 2026
Merged
@coderabbitai coderabbitai Bot mentioned this pull request Mar 25, 2026
Merged
9 tasks
KShivendu pushed a commit that referenced this pull request Mar 30, 2026
@generall @timvisee
@ffuugoo @KShivendu
in ram single mmap file (#7971)
6598476 
* WIP: introduce new vector store type

* handling of InRamMmap

* fmt

* feature-flag

* fmt

* Use if else

Co-authored-by: Roman Titov <[email protected]>

* Update lib/common/common/src/flags.rs

Co-authored-by: Tim Visée <[email protected]>

* also choose madvise for single-file in-ram-mmap

* simplify generics

* gpu fix

* fix bug

---------

Co-authored-by: Tim Visée <[email protected]>
Co-authored-by: Roman Titov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@timvisee timvisee timvisee approved these changes

@ffuugoo ffuugoo ffuugoo approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@generall @ffuugoo @timvisee