Skip to content

Implement Backup Integrity Verification for Data Corruption Detection #9187

Open

Description

@shubham-pampattiwar

Describe the problem/challenge you have

Velero currently lacks mechanisms to detect if backup data has been tampered with or corrupted after storage. If an attacker gains access to the backup storage (S3
bucket, etc.) or if data corruption occurs, there is no way to verify the integrity of backup files before attempting a restore operation.

Current State:

  • Velero supports encryption at rest (SSE-C) but this only protects against unauthorized reading, not modification
  • No checksums or signatures are generated for backup artifacts
  • No verification process exists during backup retrieval or restoration
  • Users must trust that storage systems maintain data integrity
  • Reliance solely on storage systems to maintain data integrity
  • Silent failures possible during cross-region transfers or storage migrations

Relationship to WORM Storage
This feature complements WORM (Write Once Read Many) storage solutions:

  • WORM storage: Prevents intentional tampering at the storage level
  • Integrity verification: Detects unintentional corruption at the application level/transmission level

Use Cases

  1. Disaster Recovery: Verify backup integrity before restoring critical workloads in a new cluster
  2. Compliance Auditing: Provide cryptographic proof that backup data has not been tampered with
  3. Storage Migration: Detect corruption when moving backups between storage systems
  4. Security Incident Response: Verify backup authenticity when primary infrastructure is compromised
  5. Routine Operations: Periodic integrity checking to catch storage issues early

Describe the solution you'd like

This feature addresses critical user requirements where data tampering protection is essential for enterprise security policies and compliance frameworks.

Constraints:

  • Must maintain Velero's core value proposition of portable, cross-cluster backups
  • Should not significantly impact backup/restore performance
  • Must be backward compatible with existing backup workflows

Anything else you would like to add:

Related Issues:

Environment:

  • Velero version (use velero version):
  • Kubernetes version (use kubectl version):
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • 👍 for "The project would be better with this feature added"
  • 👎 for "This feature will not enhance the project in a meaningful way"

Activity

  1. Lyndon-Li commented on Aug 18, 2025

    @Lyndon-Li
    Contributor

    Backup integrity verification may be working but the most popular solution for data tampering protection or ransomeware protection is WORM (Write Once Read Many) flavored data storage.
    Most popular backup solutions are adopting WORM as the ransomeware protection solution, Velero is also aiming the same way.

    Then for backup integrity verification, we need to clarify the scenarios that WORM cannot solve and so the verification is required. E.g., we can use checksum to detect network data transfer errors.

  2. shubham-pampattiwar commented on Aug 21, 2025

    @shubham-pampattiwar
    MemberAuthor

    @Lyndon-Li You're right that WORM storage is great for ransomware protection. But I think integrity verification covers different scenarios:

    Network transfer corruption: Like you mentioned, we need to catch data corruption during backup upload/download, especially with flaky networks.

    For example: Users may be using AWS S3 with Object Lock (WORM) for ransomware protection. But lets say during a DR test, they might find that the backups were corrupted due to network issues during upload. WORM prevented tampering but couldn't detect the corruption that happened before the data hit storage.

    Other scenarios which we should consider:

    • Storage hardware failures
    • Cross-region migrations where we need to verify data integrity
    • Compliance requirements that mandate cryptographic proof of integrity

    Wouldn't implementing checksums/signatures alongside WORM storage give us better coverage for these cases?

  3. Lyndon-Li commented on Aug 22, 2025

    @Lyndon-Li
    Contributor

    @shubham-pampattiwar
    I agree on the use cases you mentioned above.
    Merely, data integrity check will not for ransomware protection/data tampering attack, because ransomeware could embed the same checksum during tampering data.
    That is to say, they are two different directions, which are both needed regardless the priority.

    Therefore I suggest we change the title and description of this issue, ransomware protection will be handled by integration to WORM and the current proposal are primarily for data integrity assurance during transmission, etc.
    This way could better reveal our development direction.

  4. changed the title [-]Backup Integrity Verification to Prevent Data Tampering Attacks[/-] [+]Implement Backup Integrity Verification for Data Corruption Detection[/+] on Aug 25, 2025
  5. shubham-pampattiwar commented on Aug 25, 2025

    @shubham-pampattiwar
    MemberAuthor

    @Lyndon-Li Done, update the title and description, added a section Relationship to WORM Storage

  6. moved this to New / Design in OADPon Sep 2, 2025
  7. anshulahuja98 commented on Sep 16, 2025

    @anshulahuja98
    Collaborator

    Is there any high level thought process on how we wish to achieve this?
    is the focus only the PV data or also the tarballs?

  8. shubham-pampattiwar commented on Sep 16, 2025

    @shubham-pampattiwar
    MemberAuthor

    @anshulahuja98 I am currently working on the complete design proposal and will post a PR soon for community review.

  9. Lyndon-Li commented on Sep 18, 2025

    @Lyndon-Li
    Contributor

    @shubham-pampattiwar
    I am revisiting this issue and come out a new idea --- since some modern crypto methods, e.g, AES-GCM, support both encryption and authentication (integrity assurance) at the same time, why would we add a dedicated functionality for integrity verification?

    Instead, we can directly create Velero's encryption support (for resource data) solution with these modern crypto methods and integrity verification will be there as a side benefit.

    And to answer @anshulahuja98's question --- we only need to do this for resource backup data. Volume data which is managed by backup repositories, and data encryption/authentication is an internal feature from backup repositories and almost must-have for all backup repositories (e.g., both Restic and Kopia support it); and on the other hand, we cannot do this outside because it correlates to other features (e.g., deduplication).

  10. shubham-pampattiwar commented on Sep 18, 2025

    @shubham-pampattiwar
    MemberAuthor

    @Lyndon-Li That's a very good point about AES-GCM providing both encryption and integrity in one solution. I've been exploring that approach alongside traditional checksum/HMAC methods.

    However, I'm leaning toward preserving backup data transparency, many users rely on being able to directly inspect backup contents for debugging, compliance, and emergency recovery scenarios.

    Regarding volume data, you're right that Kopia have encryption and authentication capabilities. However, I discovered that Velero currently disables Kopia's verification features: https://github.com/vmware-tanzu/velero/blob/1ec281a64eae4dbd37c7ddf5f6b0ba4a9f5678da/pkg/uploader/kopia/shim.go#L73

    And repository maintenance only does pruning (manager.PruneRepo()), not verification. So while the underlying repositories have verification capabilities, Velero users currently have no way to verify volume backup integrity.

    My design proposes:

    1. Resource data integrity - checksums/HMAC for backup tarballs
    2. Volume data integrity - enabling Kopia's existing verification capabilities that are currently blocked

    This would provide complete backup integrity verification while letting repositories handle the underlying crypto as they already do.

    cc: @weshayutin @pradeepkchaturvedi

  11. Lyndon-Li commented on Sep 19, 2025

    @Lyndon-Li
    Contributor

    I'm leaning toward preserving backup data transparency

    I don't quite understand this.
    For any method, either data encryption or authentication, we don't modify the payload data (the data backed up), but we must modify the final data stored, because we need to re-produce them.
    So for data transparency, if you are referring not changing the payload data, it is always true; but if you say keep the final data format/content the same as the payload, it is not possible for any method.

  12. Lyndon-Li commented on Sep 19, 2025

    @Lyndon-Li
    Contributor

    Velero users currently have no way to verify volume backup integrity

    Here I think you have mixed the data verification capability in the storage/repository and the snapshot verification CLI from UX.

    For any data authentication method, AES-GCM or checksum, the verification is done itself when the data is read. If the verification fails, the data read will fail, then users know this.

    While, from the upper level, the read could happen in many occasions, e.g., when restoring the data.
    And for sure, we can provide a CLI called something like snapshot verification to users, which simulates a restore process and read all the snapshot data, so that any problems, including the data integrity problem could be detected.
    That is to say, the snapshot verification CLI is a UX functionality and is implemented by the various capabilities from the storage/repository, i.e., decryption, integrity, etc. The CLI is not only for integrity but to detect any problem for the snapshot data.

    By this I mean, these are different topics. First of all, we need to have all the capabilities from the storage/repository. Then:

    1. We only need to consider resource data, because the the backup repository which stores the volume data already has this capability
    2. For resource data, we should build encryption and authentication capability together because they come from one solution, we don't need to build them separately.

    Eventually, after we have 2, we will have equal capabilities for resource data and volume data. Then we can consider to provide a CLI for users to do backup/snapshot verification, but for sure, this is a different topic.

  13. shubham-pampattiwar commented on Sep 19, 2025

    @shubham-pampattiwar
    MemberAuthor

    @Lyndon-Li Thanks for breaking down the storage-level vs UX-level verification, that really helps clarify things for me.

    On data transparency: You're right that any integrity method changes the stored format. What I meant was keeping users able to examine backup contents with regular tools without needing decryption keys.

    Here's what I'm worried about with AES-GCM:

    Current state:

    s3://bucket/backups/my-backup/
    ├── my-backup.tar.gz           # Users can: tar -tzf, examine contents
    ├── my-backup-logs.gz          # Users can: gunzip, read logs
    └── my-backup-podvolumebackups.json.gz  # Users can: jq, debug volume data
    

    With checksum approach:

    s3://bucket/backups/my-backup/
    ├── my-backup.tar.gz           # Users can: tar -tzf, examine contents
    ├── my-backup-logs.gz          # Users can: gunzip, read logs
    ├── my-backup-podvolumebackups.json.gz  # Users can: jq, debug volume data
    └── my-backup-integrity.json.gz         # Added: integrity metadata
    

    With AES-GCM encryption:

    s3://bucket/backups/my-backup/
    ├── my-backup.tar.gz.encrypted    # Encrypted blob - no standard tools work
    ├── my-backup-logs.gz.encrypted   # Cannot debug without Velero + key
    └── my-backup-podvolumebackups.json.gz.encrypted  # No inspection possible
    

    Real scenarios this breaks:

    • Compliance teams who need to audit backup contents directly
    • Emergency recovery when Velero itself is down
    • Debugging what actually got backed up when restores fail
    • Moving backups to different tools if needed

    I get that AES-GCM gives us encryption + integrity in one shot, but we'd be trading away a lot of operational flexibility that users rely on today.

    Should we offer both approaches - encrypted integrity for high-security environments, and checksum integrity for transparency needs? Or do you think the encryption benefits outweigh these operational use cases?

  14. Lyndon-Li commented on Sep 23, 2025

    @Lyndon-Li
    Contributor

    Should we offer both approaches - encrypted integrity for high-security environments, and checksum integrity for transparency needs? Or do you think the encryption benefits outweigh these operational use cases?

    • For a backup solution, encryption is an essential functionality, instead of a good-to-have.
    • From the users scenarios, encryption will be most probably adopted by production environments, because users don't want to face a security leak; For dev/testing envs, users may want to see more flexibility without encryption, and data integrity is even less required by them; another possibility is that users may have hardware encryption from their storage or they don't want client-side encryption, then they will also have hardware data authentication or server side authentication.
    • From the implementation, as you can see, encryption/authentication implementation heavily couples with the final data format, so their design/implementation is regarded as a native part of the backup storage/repository. Therefore, as the ideal solution, with or without encryption/authentication are regarded as DIFFERENT data formats of the backup storage/repository.

    Then conclusively:

    • Many backup tools don't even allow users to disable encryption, because they only want to keep one data format, e.g., Restic and Kopia both don't allow this.
    • For Velero, we could make some flexibility, because Velero has the unencrypted/unauthorized data format first, so we can keep this format and add another new format with encryption/authentication. Then we most probably focus on the new format. And it is not worthy to add authentication to the old format --- if users care about the data very much and they don't have alternative methods, they most probably want encryption+authentication, otherwise, they don't need either.

    Compliance teams who need to audit backup contents directly
    Emergency recovery when Velero itself is down
    Debugging what actually got backed up when restores fail

    Directly accessing the backed up data is NEVER a validate case in my opinion. The data are probably reorganized by the backup tool, and changed without any compatibility consideration, so users are not recommended to interpret the backup data directly from the backup storage. E.g., think about how is it possible for users to interpret the volume backup data without the repository. For resource backup data, it is just because the data format for now is simple and unencrypted so users could happen to do it, it doesn't mean they can do this all the time in future.
    On the other hand, it is not a valuable support either. Users could bring up a Velero instance very easily, they don't have to consider how to access the data without Velero.

    Moving backups to different tools if needed

    This could be done by Velero if Velero supports data replication to those tools or by those tools themselves which support Velero's data format.

  15. shubham-pampattiwar commented on Sep 23, 2025

    @shubham-pampattiwar
    MemberAuthor

    @Lyndon-Li I appreciate the guidance, you are making good points about encryption being essential for production environments and the industry trend toward encrypted-by-default, plus the precedent with Restic/Kopia not allowing unencrypted storage.

    However, I think we still need to support both approaches rather than just encrypted. Here's why:

    Real enterprise adoption blockers:

    • Many large enterprises have change approval processes that take months for encryption changes
    • Compliance teams often require separate approval cycles for encryption vs integrity checking
    • Some organizations have regulatory requirements that mandate being able to audit backup contents directly
    • Dev/test environments in regulated industries still need integrity checking but can't use production encryption keys

    Current user workflows that would break:

    • Support teams who debug failed restores by examining backup contents with tar -tzf
    • Compliance auditors who verify backup contents as part of SOC2/ISO27001 audits
    • Operations teams who use jq to examine volume backup metadata during troubleshooting
    • Disaster recovery scenarios where primary Velero is down but backups need emergency access

    I understand your reasoning that "if users care about data, they want encryption+authentication, otherwise they don't need either." But there are legitimate middle-ground scenarios where enterprises need corruption detection during their lengthy encryption adoption processes.

    On direct data access being "never valid": You're right that backup formats can change and volume data is already uninterpretable. But there's a difference between "format might change" vs "format is encrypted and completely inaccessible." Many enterprises have built operational processes around the current readable format.

    I think we should support both approaches, but keep them separate - each backup uses exactly one integrity method:

    type BackupSpec struct {
        // existing fields...
    
        // IntegrityVerification configures backup integrity verification
        // +optional
        IntegrityVerification *IntegrityVerificationSpec `json:"integrityVerification,omitempty"`
    }
    
    type IntegrityVerificationSpec struct {
        // Mode: "disabled", "checksum", or "aes-gcm"
        // +kubebuilder:validation:Enum=disabled;checksum;aes-gcm
        Mode IntegrityVerificationMode `json:"mode,omitempty"`
    
        // Required for AES-GCM mode - references secret with encryption key
        // +optional
        AESSecretRef *corev1.LocalObjectReference `json:"aesSecretRef,omitempty"`
    }

    How the storage would look:

    Checksum mode backup:

    s3://bucket/backups/my-backup/
    ├── my-backup.tar.gz           # Still readable
    ├── my-backup-logs.gz          # Still readable
    └── my-backup-integrity.json.gz # Checksums added
    

    AES-GCM mode backup:

    s3://bucket/backups/my-backup/
    ├── my-backup.tar.gz.encrypted    # Encrypted with built-in integrity
    ├── my-backup-logs.gz.encrypted   # Encrypted with built-in integrity
    └── backup-metadata.json          # Unencrypted (key references, etc.)
    

    This gives us:

    • AES-GCM as the recommended, modern approach (following Restic/Kopia model)
    • Checksum as bridge solution for enterprise adoption constraints
    • Same verification infrastructure and UX for both
    • Clear adoption path: users start with checksum integrity, migrate to encrypted when processes allow

    What do you think? Are there specific concerns about supporting both that I'm missing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Needs triageWe need discussion to understand problem and decide the priorityReviewed Q3 2025

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions