@dmcgowan, @AkihiroSuda, @cpuguy83, @fuweid

Could you please take a look when you get a chance? :)

@aadhar-agarwal Would you mind updating the code with dmverity? To fix linting issues, I have updated some API names in go-dmverity: https://github.com/containerd/go-dmverity/pull/5/files

Updated!

Open Question for reviewers:

Before attempting to optimize this, I wanted to get guidance on whether it's the right approach.

Summary
When using EROFS with dm-verity, I'm seeing duplicate device setup/teardown during container creation. Each layer pays ~50ms overhead that could potentially be avoided.

Observation
During CreateContainer, containerd creates a temporary mount (via withReadonlyFS()) to read passwd and group. This triggers:

1. Create dm-verity device (~22ms)
2. Mount EROFS
3. Read files
4. Unmount EROFS
5. Close dm-verity device (~26ms)

Then during StartContainer, for the same layer:

6. Create dm-verity device again (~22ms)
7. Mount EROFS

The second dm-verity setup is identical to the first - same image, same verification.

Logs (alpine container with single layer):

# CreateContainer - temp mount for reading /etc/passwd, /etc/group
23:06:02.662  activating mount name=readonly-fs-662671502-xSx3
23:06:02.663  detected dm-verity metadata, setting up dm-verity device
23:06:02.663  opening dm-verity device device-name=containerd-erofs-1
23:06:02.685  dm-verity device created successfully                    # +22ms
23:06:02.691  attempting to close dm-verity device device=containerd-erofs-1
23:06:02.717  dm-verity device closed successfully                     # +26ms

# StartContainer - same layer, same device recreated
23:06:02.795  activating mount name=6ef600517a9df...
23:06:02.795  detected dm-verity metadata, setting up dm-verity device
23:06:02.795  opening dm-verity device device-name=containerd-erofs-1  # same!
23:06:02.816  dm-verity device created successfully                    # +21ms

Scale
With ~10 layers, this adds 500-700ms to container startup.

Questions
This seems like a design-level question rather than something I should hack around:

1. Is this expected behavior? The temp mount is supposed to be temporary, so full cleanup makes sense.
2. Should dm-verity-backed snapshotters handle this differently? Or is this just the cost of verification?

I don't want to add complexity (like keeping devices open after temp unmount) without understanding the intended behavior.

I tested the code, but found the following error when trying to run the container:

# ctr run --snapshotter erofs -t ubuntu:22.04 erofs-test sh 
ctr: failed to create erofs mount for parent 1: dm-verity mode is 'on' but .dmverity metadata not found for layer /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs"

Did we miss anything?

Hi @ChengyuZhu6, thanks for testing! I tried this and it works for me.

$ sudo ctr run --snapshotter erofs -t docker.io/library/ubuntu:22.04 erofs-test sh
#

A couple different reasons this may be happening:

1. Missing config in differ/snapshotter

[plugins."io.containerd.differ.v1.erofs"]
  enable_dmverity = true

[plugins."io.containerd.snapshotter.v1.erofs"]
  dmverity_mode = "on"

2. The existing snapshots could have been created before dm-verity was enabled. Then, the dmverity_mode = "on" setting was enabled. This setting requires .dmverity metadata for each layer, but layers unpacked earlier (when dm-verity was disabled) don't have this metadata. Could you try cleaning up the snapshots and trying again? I’ll update the error message to point out this possibility and add a note to the documentation describing the behavior. Failing in this case is appropriate because the "on" mode is explicitly strict.

I tested the code, but found the following error when trying to run the container:

# ctr run --snapshotter erofs -t ubuntu:22.04 erofs-test sh 
ctr: failed to create erofs mount for parent 1: dm-verity mode is 'on' but .dmverity metadata not found for layer /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs"

Did we miss anything?

@ChengyuZhu6 Can you confirm if it works?
@dmcgowan @fuweid @AkihiroSuda could we take a look at this for v2.3?

I tested the code, but found the following error when trying to run the container:

# ctr run --snapshotter erofs -t ubuntu:22.04 erofs-test sh 
ctr: failed to create erofs mount for parent 1: dm-verity mode is 'on' but .dmverity metadata not found for layer /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs"

Did we miss anything?

@ChengyuZhu6 Can you confirm if it works? @dmcgowan @fuweid @AkihiroSuda could we take a look at this for v2.3?

Kind ping @dmcgowan, @fuweid, @AkihiroSuda, @cpuguy83

Could I please get another review on this PR in order to land this for the 2.3 milestone? :)

@ChengyuZhu6 can we tag a release for github.com/containerd/go-dmverity anyway since it's merged.