MOD-12212: Fix high temporary memory consumption when loading search data from RDB. by Itzikvaknin · Pull Request #7336 · RediSearch/RediSearch

Itzikvaknin · 2025-11-12T12:19:28Z

Manually backported to 2.10, 2.8:

Note

Defers GC startup and registration to non-duplicate indexes when loading from RDB, and adds a test verifying only one index is stored when duplicates are present.

Backend (RDB load):
- Move IndexSpec_StartGC to run only in the non-duplicate path in IndexSpec_StoreAfterRdbLoad, preventing GC startup for duplicate indexes.
Tests:
- Add testDuplicateIndexRdbLoad to write the same index 30 times to RDB and verify only one is loaded/stored.

^{Written by Cursor Bugbot for commit c9b15e1. This will update automatically on new commits. Configure here.}

…cate == false

codecov · 2025-11-12T12:57:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.20%. Comparing base (3d9a8cb) to head (c9b15e1).
⚠️ Report is 21 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7336      +/-   ##
==========================================
+ Coverage   85.18%   85.20%   +0.01%     
==========================================
  Files         344      346       +2     
  Lines       53001    52978      -23     
  Branches    13708    13731      +23     
==========================================
- Hits        45150    45139      -11     
+ Misses       7656     7645      -11     
+ Partials      195      194       -1

Flag	Coverage Δ
flow	`84.57% <100.00%> (+0.11%)`	⬆️
unit	`52.39% <100.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

JoanFM · 2025-11-12T15:23:42Z

pack/ramp-enterprise.yml

 license: Redis Source Available License 2.0 (RSALv2) or the Server Side Public License v1 (SSPLv1) or the GNU Affero General Public License version 3 (AGPLv3)
 command_line_args: ""
-compatible_redis_version: "99.99"
+compatible_redis_version: "8.2"


can we put this into another PR to isolate changes?

this would clarify the context and socpe, and make backporting easier

This is a temporary fix, so i could upload it and test it on master cluster

This reverts commit 0d1e0bd.

Itzikvaknin · 2025-11-13T12:20:11Z

/backport

redisearch-backport-pull-request · 2025-11-13T12:20:20Z

Only merged pull requests can be backported.

GuyAv46

Nice and clean

src/spec.c

tests/cpptests/test_cpp_rdb.cpp

redisearch-backport-pull-request · 2025-11-16T17:03:02Z

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-7336-to-8.2 origin/8.2
cd .worktree/backport-7336-to-8.2
git switch --create backport-7336-to-8.2
git cherry-pick -x 0408cdeb1045059e7982becd4e0b5153b553c783

…data from RDB. (#7336) * Condition IndexSpec_StartGC and Cursors_initSpec calls on sp->isDuplicate == false * change compatible_redis_version: "8.2" * Revert "change compatible_redis_version: "8.2"" This reverts commit 0d1e0bd. * Add testDuplicateIndexRdbLoad to test the duplicate spec case * Minor * Remove index spec validation since the env is not isolated * Improve testDuplicateIndexRdbLoad

Itzikvaknin · 2025-11-17T08:29:54Z

/backport

redisearch-backport-pull-request · 2025-11-17T08:30:10Z

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-7336-to-8.2 origin/8.2
cd .worktree/backport-7336-to-8.2
git switch --create backport-7336-to-8.2
git cherry-pick -x 0408cdeb1045059e7982becd4e0b5153b553c783

…data from RDB. (#7336) * Condition IndexSpec_StartGC and Cursors_initSpec calls on sp->isDuplicate == false * change compatible_redis_version: "8.2" * Revert "change compatible_redis_version: "8.2"" This reverts commit 0d1e0bd. * Add testDuplicateIndexRdbLoad to test the duplicate spec case * Minor * Remove index spec validation since the env is not isolated * Improve testDuplicateIndexRdbLoad (cherry picked from commit 0408cde)

redisearch-backport-pull-request · 2025-11-17T08:30:17Z

Successfully created backport PR for 8.4:

[8.4] MOD-12212: Fix high temporary memory consumption when loading search data from RDB. #7385

…data from RDB. (#7336) * Condition IndexSpec_StartGC and Cursors_initSpec calls on sp->isDuplicate == false * change compatible_redis_version: "8.2" * Revert "change compatible_redis_version: "8.2"" This reverts commit 0d1e0bd. * Add testDuplicateIndexRdbLoad to test the duplicate spec case * Minor * Remove index spec validation since the env is not isolated * Improve testDuplicateIndexRdbLoad

…earch data from RDB. (#7385) MOD-12212: Fix high temporary memory consumption when loading search data from RDB. (#7336) * Condition IndexSpec_StartGC and Cursors_initSpec calls on sp->isDuplicate == false * change compatible_redis_version: "8.2" * Revert "change compatible_redis_version: "8.2"" This reverts commit 0d1e0bd. * Add testDuplicateIndexRdbLoad to test the duplicate spec case * Minor * Remove index spec validation since the env is not isolated * Improve testDuplicateIndexRdbLoad (cherry picked from commit 0408cde) Co-authored-by: Itzikvaknin <[email protected]>

…7384) * MOD-12212: Fix high temporary memory consumption when loading search data from RDB. (#7336) * Condition IndexSpec_StartGC and Cursors_initSpec calls on sp->isDuplicate == false * change compatible_redis_version: "8.2" * Revert "change compatible_redis_version: "8.2"" This reverts commit 0d1e0bd. * Add testDuplicateIndexRdbLoad to test the duplicate spec case * Minor * Remove index spec validation since the env is not isolated * Improve testDuplicateIndexRdbLoad * cleanup * Minor fix

Condition IndexSpec_StartGC and Cursors_initSpec calls on sp->isDupli…

389861c

…cate == false

github-actions bot added the size:XS label Nov 12, 2025

change compatible_redis_version: "8.2"

0d1e0bd

JoanFM reviewed Nov 12, 2025

View reviewed changes

Revert "change compatible_redis_version: "8.2""

2c7299e

This reverts commit 0d1e0bd.

Itzikvaknin added the backport 8.2 label Nov 13, 2025

Add testDuplicateIndexRdbLoad to test the duplicate spec case

80d3998

github-actions bot added the size:S label Nov 13, 2025

Itzikvaknin added 3 commits November 13, 2025 17:21

Minor

96621ee

Remove index spec validation since the env is not isolated

a802e20

Improve testDuplicateIndexRdbLoad

c9b15e1

Itzikvaknin requested review from GuyAv46 and JoanFM November 16, 2025 09:10

GuyAv46 reviewed Nov 16, 2025

View reviewed changes

src/spec.c Show resolved Hide resolved

tests/cpptests/test_cpp_rdb.cpp Show resolved Hide resolved

Itzikvaknin requested a review from GuyAv46 November 16, 2025 12:06

GuyAv46 approved these changes Nov 16, 2025

View reviewed changes

Itzikvaknin requested a review from oshadmi November 16, 2025 14:58

oshadmi approved these changes Nov 16, 2025

View reviewed changes

Itzikvaknin added this pull request to the merge queue Nov 16, 2025

Merged via the queue into master with commit 0408cde Nov 16, 2025
19 checks passed

Itzikvaknin deleted the ivaknin-rdb_load_refactor branch November 16, 2025 17:02

Itzikvaknin mentioned this pull request Nov 17, 2025

[8.2] MOD-12212: Fix index load from RDB temporary memory overhead #7384

Merged

Itzikvaknin added the backport 8.4 label Nov 17, 2025

redisearch-backport-pull-request bot mentioned this pull request Nov 17, 2025

[8.4] MOD-12212: Fix high temporary memory consumption when loading search data from RDB. #7385

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MOD-12212: Fix high temporary memory consumption when loading search data from RDB.#7336

MOD-12212: Fix high temporary memory consumption when loading search data from RDB.#7336
Itzikvaknin merged 7 commits intomasterfrom
ivaknin-rdb_load_refactor

Itzikvaknin commented Nov 12, 2025 •

edited

Loading

Uh oh!

codecov bot commented Nov 12, 2025 •

edited

Loading

Uh oh!

JoanFM Nov 12, 2025

Uh oh!

JoanFM Nov 12, 2025

Uh oh!

Itzikvaknin Nov 13, 2025

Uh oh!

Itzikvaknin commented Nov 13, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 13, 2025

Uh oh!

GuyAv46 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

redisearch-backport-pull-request bot commented Nov 16, 2025

Uh oh!

Itzikvaknin commented Nov 17, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 17, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Itzikvaknin commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JoanFM Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

JoanFM Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Itzikvaknin Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Itzikvaknin commented Nov 13, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 13, 2025

Uh oh!

GuyAv46 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

redisearch-backport-pull-request bot commented Nov 16, 2025

Uh oh!

Itzikvaknin commented Nov 17, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 17, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Itzikvaknin commented Nov 12, 2025 •

edited

Loading

codecov bot commented Nov 12, 2025 •

edited

Loading