Serialize and Restore Schema command - [MOD-11803] by GuyAv46 · Pull Request #7032 · RediSearch/RediSearch

GuyAv46 · 2025-10-12T11:15:19Z

Implement schema serialization and deserialization as part of the ASM effort

Serialization & Deserialization

Implement IndexSpec_Serialize and IndexSpec_Deserialize functions to generate a RedisModuleString from a schema and a schema (IndexSpec) from a serialization string.
The implementations rely on the type rdb_load and rdb_save implementations, so we now re-expose these methods to redis. This is okay, as we don't actually save the schemas as part of the key-space, so they won't be called by Redis if we don't call the manual implementations.

Testing is focused on the new logic, as the RDB serialization/deserialization is already well tested

_FT._RESTOREIFNX Command

A new command for manual RESTORE logic for non-key module-type objects of the module. Currently, there is a single "SCHEMA" sub-command to restore indexes.
An additional FT.DEBUG DUMP_SCHEMA command was added to generate a schema dump for the restore command.

Testing is again focused on the behavior of the restore command. The new debug command and the ACL registration are also tested

Main objects this PR modified

Schema's type (ft_index0) RDB load and save implementations
Implement manual dump and restore functions for IndexSpec
Implement a new internal restore command
Implement a debug dump command

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

Note

Introduce IndexSpec serialization/deserialization, an internal schema restore command, and a debug command to dump schemas, with supporting RDB refactors and tests.

Core / RDB:
- Add IndexSpec_Serialize and IndexSpec_Deserialize to convert schemas to/from RDB-encoded RedisModuleString.
- Expose type rdb methods via IndexSpec_RdbLoad_Logic and wrapper for IndexSpec_RdbSave; split store logic into IndexSpec_StoreAfterRdbLoad.
- Refactor IndexSpec_StartGC to remove RedisModuleCtx*; switch GC logs to RSDummyContext.
Commands:
- New internal command '_FT._RESTOREIFNX SCHEMA <encver> <blob>' (macro RS_RESTORE_IF_NX) to restore schema; register subcommand.
- New debug command FT.DEBUG DUMP_SCHEMA <index> returning serialized schema and encoding version; register in debug help.
Module wiring:
- Register restore command and subcommand; adjust command flags/ACL categories.
Mocks / Tests:
- Redismock: add SaveDataTypeToString/LoadDataTypeFromStringEncver helpers.
- C++ tests for RDB roundtrip and string serialize/deserialize.
- Python tests for debug help, DUMP_SCHEMA, and end-to-end restore (_FT._RESTOREIFNX SCHEMA).

^{Written by Cursor Bugbot for commit 17b4fe3. This will update automatically on new commits. Configure here.}

…ema_to_command

alonre24

Nice!

src/module.c

src/spec.c

alonre24 · 2025-10-12T22:35:16Z

src/spec.c

+      .rdb_load = IndexSpec_RdbLoad_Logic,    // We don't store the index spec in the key space,
+      .rdb_save = IndexSpec_RdbSave_Wrapper,  // but these are useful for serialization/deserialization (and legacy loading)


Can you remind me of the motivation for this change?
Is the reason for not using simply using Indexes_RdbLoad upon handling the new ASM event of adding a new node (next PR) supporting legacy loading?

We can't serialize all the schemas together with aux_save, first and foremost, because there is no API to do so. It's also simpler to handle the schemas one by one, as some may exist already, while others may not.

This change gives back the existing load and save functionalities of single schema serialization and deserialization, allowing us to use RedisModule_LoadDataTypeFromStringEncver and RedisModule_SaveDataTypeToString, and maybe other APIs in the future.
We still don't actually create any key of the schema type, so these should not be called automatically by redis while saving to RDB (only by the aux_save callback)

tests/pytests/test_index.py

codecov · 2025-10-13T06:56:38Z

Codecov Report

❌ Patch coverage is 98.27586% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 85.51%. Comparing base (6cd3771) to head (17b4fe3).
⚠️ Report is 41 commits behind head on master.

Files with missing lines	Patch %	Lines
src/debug_commands.c	92.85%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7032      +/-   ##
==========================================
- Coverage   85.51%   85.51%   -0.01%     
==========================================
  Files         321      321              
  Lines       50738    50786      +48     
  Branches    10351    10351              
==========================================
+ Hits        43391    43430      +39     
- Misses       7196     7205       +9     
  Partials      151      151

Flag	Coverage Δ
flow	`83.99% <98.27%> (-0.11%)`	⬇️
unit	`51.52% <46.55%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

raz-mon

LGTM overall
One question

src/spec.c

raz-mon

LGTM
Regarding the ASM design (upcoming PRs to use this), I think it is not a great design (for our module at least), since replication of the index should occur upon shard addition rather than upon slot migration. It's quite wasteful to send all the aux data for every slot migration.
Nonetheless this functionality is good for us and nicely done! 💪

* layout basic schema command generation * implement IndexSpec_Serialize and IndexSpec_Deserialize * added a unit-test * fix warning * implement mock API * implement restore schema command * add tests (and a dump debug command) * improve test * remove temporary logs * fix tests

GuyAv46 · 2025-10-23T17:13:40Z

/backport

* layout basic schema command generation * implement IndexSpec_Serialize and IndexSpec_Deserialize * added a unit-test * fix warning * implement mock API * implement restore schema command * add tests (and a dump debug command) * improve test * remove temporary logs * fix tests (cherry picked from commit 5feebc4)

redisearch-backport-pull-request · 2025-10-23T17:14:02Z

Successfully created backport PR for 8.4:

[8.4] Serialize and Restore Schema command - [MOD-11803] #7142

Serialize and Restore Schema command - [MOD-11803] (#7032) * layout basic schema command generation * implement IndexSpec_Serialize and IndexSpec_Deserialize * added a unit-test * fix warning * implement mock API * implement restore schema command * add tests (and a dump debug command) * improve test * remove temporary logs * fix tests (cherry picked from commit 5feebc4) Co-authored-by: GuyAv46 <[email protected]>

github-actions bot added the size:M label Oct 12, 2025

GuyAv46 added 7 commits October 12, 2025 18:32

layout basic schema command generation

b2f9970

implement IndexSpec_Serialize and IndexSpec_Deserialize

daa5ed4

added a unit-test

e7bc904

fix warning

266840e

implement mock API

1180b9b

implement restore schema command

fd0e796

add tests (and a dump debug command)

b950314

GuyAv46 force-pushed the guyav-serialize_schema_to_command branch from 963f5c3 to b950314 Compare October 12, 2025 17:24

GuyAv46 changed the base branch from guyav-ASM_notifications to master October 12, 2025 17:24

github-actions bot added the size:XL label Oct 12, 2025

GuyAv46 requested a review from alonre24 October 12, 2025 17:26

GuyAv46 changed the title ~~Guyav serialize schema to command~~ Serialize and Restore Schema command - [MOD-11803] Oct 12, 2025

GuyAv46 marked this pull request as ready for review October 12, 2025 17:39

This comment was marked as outdated.

Sign in to view

GuyAv46 added 2 commits October 12, 2025 20:46

improve test

ae06f86

remove temporary logs

ffc36ae

This comment was marked as outdated.

Sign in to view

GuyAv46 added 2 commits October 13, 2025 00:47

Merge remote-tracking branch 'origin/master' into guyav-serialize_sch…

cd66fc2

…ema_to_command

fix tests

17b4fe3

alonre24 reviewed Oct 12, 2025

View reviewed changes

GuyAv46 force-pushed the guyav-serialize_schema_to_command branch from cd4e361 to 17b4fe3 Compare October 13, 2025 06:23

alonre24 approved these changes Oct 13, 2025

View reviewed changes

GuyAv46 mentioned this pull request Oct 13, 2025

Handle ASM Notifications - [MOD-11120] #7022

Merged

2 tasks

raz-mon reviewed Oct 15, 2025

View reviewed changes

src/spec.c Show resolved Hide resolved

raz-mon approved these changes Oct 15, 2025

View reviewed changes

GuyAv46 added this pull request to the merge queue Oct 15, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 15, 2025

GuyAv46 added this pull request to the merge queue Oct 15, 2025