Repository navigation
Description
Describe the problem/challenge you have
Velero currently lacks mechanisms to detect if backup data has been tampered with or corrupted after storage. If an attacker gains access to the backup storage (S3
bucket, etc.) or if data corruption occurs, there is no way to verify the integrity of backup files before attempting a restore operation.
Current State:
- Velero supports encryption at rest (SSE-C) but this only protects against unauthorized reading, not modification
- No checksums or signatures are generated for backup artifacts
- No verification process exists during backup retrieval or restoration
- Users must trust that storage systems maintain data integrity
- Reliance solely on storage systems to maintain data integrity
- Silent failures possible during cross-region transfers or storage migrations
Relationship to WORM Storage
This feature complements WORM (Write Once Read Many) storage solutions:
- WORM storage: Prevents intentional tampering at the storage level
- Integrity verification: Detects unintentional corruption at the application level/transmission level
Use Cases
- Disaster Recovery: Verify backup integrity before restoring critical workloads in a new cluster
- Compliance Auditing: Provide cryptographic proof that backup data has not been tampered with
- Storage Migration: Detect corruption when moving backups between storage systems
- Security Incident Response: Verify backup authenticity when primary infrastructure is compromised
- Routine Operations: Periodic integrity checking to catch storage issues early
Describe the solution you'd like
This feature addresses critical user requirements where data tampering protection is essential for enterprise security policies and compliance frameworks.
Constraints:
- Must maintain Velero's core value proposition of portable, cross-cluster backups
- Should not significantly impact backup/restore performance
- Must be backward compatible with existing backup workflows
Anything else you would like to add:
Related Issues:
- Support for data integrity and data trust of the backups #3875 - Data integrity and trust tests for backups
- Backup checksums #1072 - Store checksums for backup files to ensure integrity
Environment:
- Velero version (use
velero version): - Kubernetes version (use
kubectl version): - Kubernetes installer & version:
- Cloud provider or hardware configuration:
- OS (e.g. from
/etc/os-release):
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
- 👍 for "The project would be better with this feature added"
- 👎 for "This feature will not enhance the project in a meaningful way"
Activity
Lyndon-Li commentedon Aug 18, 2025
Backup integrity verification may be working but the most popular solution for data tampering protection or ransomeware protection is WORM (Write Once Read Many) flavored data storage.
Most popular backup solutions are adopting WORM as the ransomeware protection solution, Velero is also aiming the same way.
Then for backup integrity verification, we need to clarify the scenarios that WORM cannot solve and so the verification is required. E.g., we can use checksum to detect network data transfer errors.
shubham-pampattiwar commentedon Aug 21, 2025
@Lyndon-Li You're right that WORM storage is great for ransomware protection. But I think integrity verification covers different scenarios:
Network transfer corruption: Like you mentioned, we need to catch data corruption during backup upload/download, especially with flaky networks.
For example: Users may be using AWS S3 with Object Lock (WORM) for ransomware protection. But lets say during a DR test, they might find that the backups were corrupted due to network issues during upload. WORM prevented tampering but couldn't detect the corruption that happened before the data hit storage.
Other scenarios which we should consider:
Wouldn't implementing checksums/signatures alongside WORM storage give us better coverage for these cases?
Lyndon-Li commentedon Aug 22, 2025
@shubham-pampattiwar
I agree on the use cases you mentioned above.
Merely, data integrity check will not for ransomware protection/data tampering attack, because ransomeware could embed the same checksum during tampering data.
That is to say, they are two different directions, which are both needed regardless the priority.
Therefore I suggest we change the title and description of this issue, ransomware protection will be handled by integration to WORM and the current proposal are primarily for data integrity assurance during transmission, etc.
This way could better reveal our development direction.
[-]Backup Integrity Verification to Prevent Data Tampering Attacks[/-][+]Implement Backup Integrity Verification for Data Corruption Detection[/+]shubham-pampattiwar commentedon Aug 25, 2025
@Lyndon-Li Done, update the title and description, added a section
Relationship to WORM Storageanshulahuja98 commentedon Sep 16, 2025
Is there any high level thought process on how we wish to achieve this?
is the focus only the PV data or also the tarballs?
shubham-pampattiwar commentedon Sep 16, 2025
@anshulahuja98 I am currently working on the complete design proposal and will post a PR soon for community review.
Lyndon-Li commentedon Sep 18, 2025
@shubham-pampattiwar
I am revisiting this issue and come out a new idea --- since some modern crypto methods, e.g, AES-GCM, support both encryption and authentication (integrity assurance) at the same time, why would we add a dedicated functionality for integrity verification?
Instead, we can directly create Velero's encryption support (for resource data) solution with these modern crypto methods and integrity verification will be there as a side benefit.
And to answer @anshulahuja98's question --- we only need to do this for resource backup data. Volume data which is managed by backup repositories, and data encryption/authentication is an internal feature from backup repositories and almost must-have for all backup repositories (e.g., both Restic and Kopia support it); and on the other hand, we cannot do this outside because it correlates to other features (e.g., deduplication).
shubham-pampattiwar commentedon Sep 18, 2025
@Lyndon-Li That's a very good point about AES-GCM providing both encryption and integrity in one solution. I've been exploring that approach alongside traditional checksum/HMAC methods.
However, I'm leaning toward preserving backup data transparency, many users rely on being able to directly inspect backup contents for debugging, compliance, and emergency recovery scenarios.
Regarding volume data, you're right that Kopia have encryption and authentication capabilities. However, I discovered that Velero currently disables Kopia's verification features: https://github.com/vmware-tanzu/velero/blob/1ec281a64eae4dbd37c7ddf5f6b0ba4a9f5678da/pkg/uploader/kopia/shim.go#L73
And repository maintenance only does pruning
(manager.PruneRepo()), not verification. So while the underlying repositories have verification capabilities, Velero users currently have no way to verify volume backup integrity.My design proposes:
This would provide complete backup integrity verification while letting repositories handle the underlying crypto as they already do.
cc: @weshayutin @pradeepkchaturvedi
Lyndon-Li commentedon Sep 19, 2025
I don't quite understand this.
For any method, either data encryption or authentication, we don't modify the payload data (the data backed up), but we must modify the final data stored, because we need to re-produce them.
So for data transparency, if you are referring not changing the payload data, it is always true; but if you say keep the final data format/content the same as the payload, it is not possible for any method.
Lyndon-Li commentedon Sep 19, 2025
Here I think you have mixed the data verification capability in the storage/repository and the snapshot verification CLI from UX.
For any data authentication method, AES-GCM or checksum, the verification is done itself when the data is read. If the verification fails, the data read will fail, then users know this.
While, from the upper level, the read could happen in many occasions, e.g., when restoring the data.
And for sure, we can provide a CLI called something like snapshot verification to users, which simulates a restore process and read all the snapshot data, so that any problems, including the data integrity problem could be detected.
That is to say, the snapshot verification CLI is a UX functionality and is implemented by the various capabilities from the storage/repository, i.e., decryption, integrity, etc. The CLI is not only for integrity but to detect any problem for the snapshot data.
By this I mean, these are different topics. First of all, we need to have all the capabilities from the storage/repository. Then:
Eventually, after we have 2, we will have equal capabilities for resource data and volume data. Then we can consider to provide a CLI for users to do backup/snapshot verification, but for sure, this is a different topic.
shubham-pampattiwar commentedon Sep 19, 2025
@Lyndon-Li Thanks for breaking down the storage-level vs UX-level verification, that really helps clarify things for me.
On data transparency: You're right that any integrity method changes the stored format. What I meant was keeping users able to examine backup contents with regular tools without needing decryption keys.
Here's what I'm worried about with AES-GCM:
Current state:
With checksum approach:
With AES-GCM encryption:
Real scenarios this breaks:
I get that AES-GCM gives us encryption + integrity in one shot, but we'd be trading away a lot of operational flexibility that users rely on today.
Should we offer both approaches - encrypted integrity for high-security environments, and checksum integrity for transparency needs? Or do you think the encryption benefits outweigh these operational use cases?
Lyndon-Li commentedon Sep 23, 2025
Then conclusively:
Directly accessing the backed up data is NEVER a validate case in my opinion. The data are probably reorganized by the backup tool, and changed without any compatibility consideration, so users are not recommended to interpret the backup data directly from the backup storage. E.g., think about how is it possible for users to interpret the volume backup data without the repository. For resource backup data, it is just because the data format for now is simple and unencrypted so users could happen to do it, it doesn't mean they can do this all the time in future.
On the other hand, it is not a valuable support either. Users could bring up a Velero instance very easily, they don't have to consider how to access the data without Velero.
This could be done by Velero if Velero supports data replication to those tools or by those tools themselves which support Velero's data format.
shubham-pampattiwar commentedon Sep 23, 2025
@Lyndon-Li I appreciate the guidance, you are making good points about encryption being essential for production environments and the industry trend toward encrypted-by-default, plus the precedent with Restic/Kopia not allowing unencrypted storage.
However, I think we still need to support both approaches rather than just encrypted. Here's why:
Real enterprise adoption blockers:
Current user workflows that would break:
I understand your reasoning that "if users care about data, they want encryption+authentication, otherwise they don't need either." But there are legitimate middle-ground scenarios where enterprises need corruption detection during their lengthy encryption adoption processes.
On direct data access being "never valid": You're right that backup formats can change and volume data is already uninterpretable. But there's a difference between "format might change" vs "format is encrypted and completely inaccessible." Many enterprises have built operational processes around the current readable format.
I think we should support both approaches, but keep them separate - each backup uses exactly one integrity method:
How the storage would look:
Checksum mode backup:
AES-GCM mode backup:
This gives us:
What do you think? Are there specific concerns about supporting both that I'm missing?