Skip to content

[Bug] Intermittent volume backup failures when Docker cleanup is enabled - race condition with ubuntu image pull #3307

Closed

Description

@Yuri-NagaSaki

To Reproduce

Describe the bug

When both volume backup and Docker cleanup features are enabled, backups fail because the cleanup removes the ubuntu image that volume backups depend on.

Error message
Unable to find image 'ubuntu:latest' locally
docker: Error response from daemon: lease does not exist: not found

Cause analysis

  1. Volume backup uses docker run --rm ubuntu to create tar archives
  2. Docker cleanup runs docker image prune --all --force at UTC 00:00
  3. Volume backup is scheduled at exactly 0 0 * * * (same time as cleanup.)
  4. Race condition: When both run simultaneously:
  • Cleanup deletes containerd's content store and leases
  • Backup tries to pull ubuntu image at the same time
  • containerd ends up in inconsistent state → pull fails with various errors
  1. Success case: When there's enough time gap between cleanup completion and backup start, the pull succeeds

Current vs. Expected behavior

When there's enough time gap between cleanup completion and backup start, the pull succeeds

Task Cron Expression Execution Time
Docker Cleanup 0 0 * * * UTC 00:00
Volume Backup 0 0 * * * UTC 00:00

Both tasks executed at precisely the same time, causing a race condition:

  • Cleanup deleted the Ubuntu image and containerd's content store
  • Backup simultaneously attempted to pull the Ubuntu image
  • containerd entered an inconsistent state → Pulled image failed intermittently
Initializing volume backup
Volume name: service-crowdsec-k5ws6c_crowdsec-config
Backup file name: service-crowdsec-k5ws6c_crowdsec-config-2025-12-19T00:00:00.219Z.tar
Turning off volume backup: No
Starting volume backup
Dir: /etc/dokploy/volume-backups/volumeBackup-calculate-neural-program-n4jw7g
Unable to find image 'ubuntu:latest' locally
docker: Error response from daemon: lease does not exist: not found
Run 'docker run --help' for more information

Provide environment information

Ubuntu 24.04 
DokPloy Version v0.26.2

Which area(s) are affected? (Select all that apply)

Application, Docker

Are you deploying the applications where Dokploy is installed or on a remote server?

Same server where Dokploy is installed

Additional context

No response

Will you send a PR to fix it?

Yes

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions