Skip to content

Delta Snapshot - Delete processing with new schema - #1784

Merged
AndreKurait merged 16 commits into
opensearch-project:mainfrom
AndreKurait:DeleteProcessingWithNewSchema
Aug 26, 2025
Merged

AndreKurait merged 16 commits into
opensearch-project:mainfrom
AndreKurait:DeleteProcessingWithNewSchema

Conversation

@AndreKurait

@AndreKurait AndreKurait commented Aug 21, 2025 •

Copy link
Copy Markdown
Member

Description

This PR introduces delete operation support for delta snapshot migrations and significantly matures the RFS transformation schema to support a more complete set of bulk operation capabilities.

Key Changes:

  1. Delete Operations Now Functional
    • Documents deleted between snapshots are now properly removed from the target cluster
    • DeltaMode.UPDATES_AND_DELETES is fully enabled
    • DeltaMode.DELETES_ONLY is fully enabled
    • End-to-end tests confirm deletions return 404 (not found) instead of persisting
  2. Enhanced Bulk Operation Schema
    • Replaces simple BulkDocSection string-based approach with structured operation objects
    • New schema supports:
      • External versioning
      • Version types (internal, external, external_gte)
      • Separate IndexOp and DeleteOp implementations
    • Sets foundation for future transformation capabilities
  3. Improved Architecture
    • Type-safe builder patterns for operation construction
    • Clear separation between operation metadata and document content
    • Better extensibility for future operations (updates, conditional deletes, scripts, etc.)

Impact:

This refactoring enables complete delta migration functionality while establishing a robust schema that can support advanced transformations and version control strategies during the migration process.

Issues Resolved

https://opensearch.atlassian.net/browse/MIGRATIONS-2582

Testing

Functional tests added for DeltaMode.UPDATES_AND_DELETES

Check List

  • New functionality includes testing
  • Public documentation issue/PR created, if applicable.
    • RFS Custom Transformations and Delta Snapshots are not yet documented

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ter model for the different operations

Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
@AndreKurait
AndreKurait marked this pull request as ready for review August 22, 2025 22:05
Signed-off-by: Andre Kurait <[email protected]>

@peternied peternied left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a couple thinks are still in flight, seeing good progress here, a couple of comments on the current stand of the PR

throw new IllegalArgumentException("Failed to parse source doc: " + e.getMessage(), e);
}
}
// TODO: We can get a performance improvement by keeping doc.source as a byte array until here

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you want to handle these todos, in this PR or as follow up jiras?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is documenting existing behavior. Let's leave it here until we determine there's value in that refactor

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please create jira ticket(s) so we can prioritize this work on the backlog or as OE tasks

Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>

@peternied peternied left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments, feel free to merge after they are addressed or deferred.

Signed-off-by: Andre Kurait <[email protected]>
@AndreKurait
AndreKurait merged commit cbea2bf into opensearch-project:main Aug 26, 2025
@AndreKurait
AndreKurait deleted the DeleteProcessingWithNewSchema branch August 26, 2025 16:41
@codecov

codecov Bot commented Aug 26, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (39a0ddb) to head (4a9f59b).
⚠️ Report is 20 commits behind head on main.

Additional details and impacted files
@@     Coverage Diff      @@
##   main   #1784   +/-   ##
============================
============================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AndreKurait AndreKurait changed the title Delete processing with new schema Delta Snapshot - Delete processing with new schema Sep 4, 2025

This branch was previously deployed

1 inactive deployment
migrations-cicd — 4a9f59bf Deployed Aug 26, 2025 by AndreKurait
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants