Skip to content

Add mutable map payload index on top of Gridstore#6634

Merged
timvisee merged 8 commits intodevfrom
mutable-map-index-gridstore
Jun 5, 2025
Merged

Add mutable map payload index on top of Gridstore#6634
timvisee merged 8 commits intodevfrom
mutable-map-index-gridstore

Conversation

@timvisee
Copy link
Copy Markdown
Member

@timvisee timvisee commented Jun 3, 2025 •
edited
Loading

Depends on #6609

Similar to #6609, this implements Gridstore as alternative storage backend for a payload index. This implements it for the map index.

Tasks

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

@timvisee timvisee added this to the Remove RocksDB milestone Jun 3, 2025
@timvisee
Copy link
Copy Markdown
Member Author

timvisee commented Jun 3, 2025 •
edited
Loading

Here's a quick and dirty test with the following setup:

bfb -d 4 --int-payloads 100 --max-int-payloads 10 --indexing-threshold 0 -n 1000000

On RocksDB:

  • Cold load: 1.64s
  • Hot load: 1.59s
  • Upsert time: 18s
  • Disk usage: 13M + 13M = 25M

On Gridstore:

  • Cold load: 1.27s
  • Hot load: 0.55s
  • Upsert time: 5s
  • Disk usage: 36M + 36M = 71M

The Gridstore implementation currently consumes more space because the configured Gridstore options aren't ideal for very small numeric values. This we can tune later.

@timvisee timvisee force-pushed the mutable-map-index-gridstore branch 2 times, most recently from a62b7cf to 7d31d7b Compare June 4, 2025 11:37
generall
generall reviewed Jun 4, 2025
View reviewed changes
Comment thread lib/segment/src/index/field_index/map_index/mutable_map_index.rs
@timvisee timvisee force-pushed the mutable-numeric-index-gridstore branch from 067cbf5 to c7d0b2b Compare June 4, 2025 15:47
@timvisee timvisee force-pushed the mutable-map-index-gridstore branch from 7d31d7b to 0972557 Compare June 4, 2025 15:50
@timvisee timvisee mentioned this pull request Jun 4, 2025
Merged
12 tasks
Base automatically changed from mutable-numeric-index-gridstore to dev June 5, 2025 07:10
@timvisee timvisee force-pushed the mutable-map-index-gridstore branch from 0972557 to 6488bdd Compare June 5, 2025 07:11
@timvisee timvisee marked this pull request as ready for review June 5, 2025 07:12
@timvisee timvisee requested a review from Copilot June 5, 2025 07:12

This comment was marked as resolved.

Sign in to view

@coderabbitai

This comment was marked as resolved.

Sign in to view
coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

JojiiOfficial
JojiiOfficial reviewed Jun 5, 2025
View reviewed changes
Comment thread lib/segment/src/index/field_index/map_index/mod.rs
coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

@timvisee timvisee merged commit 4d71f06 into dev Jun 5, 2025
17 checks passed
@timvisee timvisee deleted the mutable-map-index-gridstore branch June 5, 2025 07:54
@coderabbitai coderabbitai Bot mentioned this pull request Jun 6, 2025
Merged
12 tasks
generall pushed a commit that referenced this pull request Jul 17, 2025
@timvisee @generall
Add mutable map payload index on top of Gridstore (#6634)
705607f 
* Serialize vector of ecow strings using cbor

* Adjust trait bounds for Gridstore support in map index

* Add Gridstore backend to mutable map index

* Add Gridstore builder

* Enable new map index in index selector

* Add tests

* Use different Gridstore block size depending on map data type

* Fix review remarks
@coderabbitai coderabbitai Bot mentioned this pull request Oct 23, 2025
Merged
6 tasks
@coderabbitai coderabbitai Bot mentioned this pull request Dec 15, 2025
Merged
@coderabbitai coderabbitai Bot mentioned this pull request Mar 5, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

Copilot code review Copilot Copilot left review comments

@generall generall generall approved these changes

@JojiiOfficial JojiiOfficial JojiiOfficial approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

Remove RocksDB

Development

Successfully merging this pull request may close these issues.

4 participants

@timvisee @generall @JojiiOfficial