-
Notifications
You must be signed in to change notification settings - Fork 634
Preserve Incomplete Output and Overwritten Files #3351
Copy link
Copy link
Open
1 / 11 of 1 issue completedLabels
Description
Is your feature request related to a problem? Please describe.
I often find it frustrating when a Snakemake job fails because Snakemake:
- Removes any broken or incomplete output files.
- Cleans up shadow directories regardless of job success.
- Automatically deletes files that need to be regenerated.
In most cases, rerunning the job is quick, or I can check logs to diagnose the issue. However, sometimes I make mistakes, and large files generated after intensive computation are lost—without backups.
Describe the solution you'd like
Instead of discarding these files outright, Snakemake could move them to subfolders within .snakemake/, preserving them for potential recovery. These subfolders could be named based on the lock file, ensuring consistency at each moment.
Proposed behavior:
- Failed or overwritten files should be moved to
.snakemake/cache/1-recycle/, preserving the original directory structure. - Shadow directories could follow a consistent naming scheme, such as
.snakemake/shadow/1-xxxx, and should not be removed if the job fails. - If a new workflow attempts to create
.snakemake/lock/1.out, it can assume that the cached broken files are no longer needed and remove them before execution.
This approach would provide a safeguard against accidental data loss while keeping workflow execution manageable.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Ready