Skip to content

Devmapper snapshotter not cleaning up after failed parent suspend #5110

@ctrlaltdel121

Description

@ctrlaltdel121

I am seeing an issue with the devmapper snapshotter that causes the metadata db to be left in a seemingly corrupt state.

First, an image pull failed because the parent snapshot was not found. I think this is related to #3787:

failed to suspend device \"docker--fc-thinpool--devmapper-snap-78199\": no such device or address" msg="failed to create snapshot device from parent docker--fc-thinpool--devmapper-snap-78199"

error="failed to prepare extraction snapshot \"extract-291310233-tINQ sha256:548debedae4b2c03c1dd3abb3cb9a2ecab28447a6ff4ed7f446cfab2b992648d\": failed to suspend device \"docker--fc-thinpool--devmapper-snap-78199\": no such device or address: unknown"

The device that was trying to snapshot from 78199 was given ID 78200, and after it errored out here, the 78200 device seems to have been left in the metadata.

This is causing future devices to try and re-use 78200 and fail:

error="failed to create container, cleaning up: failed to save metadata for device \"docker--fc-thinpool--devmapper-snap-78200\" (parent: \"docker--fc-thinpool--devmapper-snap-78179\"): device \"docker--fc-thinpool--devmapper-snap-78200\" is already there {DeviceID:548 Size:3221225472 Name:docker--fc-thinpool--devmapper-snap-78200 ParentName:docker--fc-thinpool--devmapper-snap-78199 State:unknown 0 Error:}: object already exists: unknown"

In the CreateSnapshotDevice function, the device is properly rolled back if the activation or creation of the device errors, but nothing is done to roll back if the parent suspend errors, even though it is after it is added to metadata.

Steps to reproduce the issue:

  1. Force the devmapper snapshotter to fail suspending a parent device in CreateSnapshotDevice
  2. Try and create a new snapshot
  3. ID is re-used and creation fails

Describe the results you received:
Failed device is not removed from metadata

Describe the results you expected:
Something is done to clean up the metadata so the ID returned by the sequence is not in use

Output of containerd --version:
1.3-8

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions