Repository navigation
Description
Delta Snapshot Reindex with RFS
RFC Proposal
What / Why
We propose Delta Snapshot Reindex (DSR), an enhancement to Reindex‑from‑Snapshot (RFS) that applies only the changes between two snapshots to a target cluster, rather than full reingestion. This reduces unnecessary I/O, network, and compute overhead when updating historical data already backfilled via RFS or Snapshot/Restore (see OpenSearch Migration Assistant docs: https://opensearch.org/docs/latest/migration-assistant/).
Who Wants This?
- Operators maintaining standby clusters who need periodic, incremental updates.
- Data-lake integrators: managing cold or archived OpenSearch clusters who periodically snapshot data for long-term storage and need to apply only incremental changes (deltas) to downstream systems or refreshed clusters, avoiding full reprocessing.
- Migration Operators: optimizing backfill resource costs while reducing lag between source and target clusters.
Problem Statement
Updating a target index currently requires deleting it and re‑ingesting all documents to guarantee consistency. This:
- Wastes network/CPU on unchanged docs.
- Causes write amplification on the target cluster.
- Causes a high RTO for the migration without other solutions such as Capture and Replay
Proposed Solution
- Inputs:
- Old snapshot name (already applied to target).
- New snapshot name (later/earlier point‑in‑time).
- Per‑shard, per‑segment diff:
- Parse Lucene segment files from both snapshots.
- Compare segment names and live‑docs bitsets.
- Delta determination:
| Segment Scenario | Action |
|---|---|
| In old but not in new | Delete all live docs from that segment. |
| In new but not in old | Add all live docs from that segment. |
| In both snapshots | Compare bitsets: - Old ∖ New → delete - New ∖ Old → add |
- Apply deltas:
- First execute all deletes, then all adds/updates.
- Merge‑optimization (Future enhancement):
- Track a hashmap of
_id→ hash(_source); if a delete and add target the same_idwith identical hash, skip both.
- Track a hashmap of
Limitations & Drawbacks
- Segment merges can trigger full‑segment deletes/adds unless optimized via the hashmap.
_source:falseindices is not compatible (same as RFS).- Plugin classes: custom plugins may need to be on the DSR application classpath.
Alternatives Considered
-
Log‑based CDC: Invasive change‑stream setup. Limited compatibility across older versions.
-
Reindex REST API: Burdens source cluster; no delete detection.
-
Incremental Snapshot/Restore: Unsupported beyond one version.
-
Native Delta Reindex in Snapshot/Restore API:
Embedding delta logic directly into theSnapshot/Restoremechanism (rather than building around Reindex-from-Snapshot) could offer a cleaner, lower-latency path for applying snapshot diffs. This would allow:- Repository-level segment diffing and blob reuse.
- Elimination of reindex overhead and custom coordination logic.
- Tighter integration with existing DR workflows.
Drawbacks: Requires deeper changes to core restore flows and compatibility handling across OpenSearch/Lucene versions. Slower to implement and validate across all repository types.
Feedback Requested
- Suggestions for efficiently processing segment merges
- Special data types needing extra care (e.g. nested docs, binary fields, geo‑shapes, custom analyzers, etc.).
Activity
kanatti commentedon May 15, 2025
Thanks, this is great.
I had question on Delta Determination case 1 -
In old but not in newwhen there is a segment merge.snapshot 1:
snapshot 2:
In this case will DSR delete all live docs from 2 and 3 and then re-ingest same docs from 4? And I believe applying deletes first before additions becomes important here? A reverse order of 4 and 2,3 will end up deleting docs without recreating.
AndreKurait commentedon May 15, 2025
@kanatti - Thanks for the feedback!
Yes, that is understood, i've specified that behavior as "First execute all deletes, then all adds/updates" in the original plan.
Does that alleviate any concerns?
kanatti commentedon May 15, 2025
No concerns. Just wanted to confirm if my understanding is right.
peternied commentedon May 15, 2025
Thanks for writing this proposal up @AndreKurait. I was wondering if you've got more details on details around the
Delta determination1/ Sounds like comparison happening between snapshot1 that was previously deployed with RFS and snapshot2 which is the delta from snapshot1. What about the state of the cluster itself, can we verify that the cluster is in 'good standing' compared to the last execution of RFS tasks?
2/ It also sounds like there will be a considerable calculation between delta snapshots, is that state stored anywhere, can you pause/resume DSR?