Skip to content

Creating diff of a container running on overlayfs generates duplicate mounts #6077

@foamkeen

Description

@foamkeen

Description

When we try to create a diff of a container running on overlayfs with ctr utility, we immediately see the following in the kernel log:

kernel: [42256.430385] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
kernel: [42256.430389] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.

And later for that container we observe this "undefined behavior" as the RootFS of that container becoming partially not-writable (an attempt to create a file results in "No such file or directory" error).

My understanding is that this happens because Compare function of walkingDiff does temporary mounts:

if err := mount.WithTempMount(ctx, lower, func(lowerRoot string) error {

And it does it for an 'Active' snapshot using the same workdir and upperdir, but for overlayfs you are not supposed to do this.

As a solution I see a conversion of a mount to a read-only one before passing it to Compare. It can be done by altering the mount options like so: workdir and upperdir options are removed and a path from upperdir option is moved to the leftmost in lowerdir option.

Steps to reproduce the issue

  1. Run ctr snapshot diff <container id> for a container running on overlayfs and observe overlayfs complains about upperdir and workdir in the kernel log.

Describe the results you received and expected

Running ctr snapshot diff on overlayfs should not corrupt the RootFS of a running container.

What version of containerd are you using?

containerd github.com/containerd/containerd 1.4.8+azure 7eba593

Any other relevant information

runc --version:

runc version 1.0.0-rc95
commit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.5.1

uname -a:

Linux aks-nodepool1-24028517-vmss000002 5.4.0-1056-azure #58~18.04.1-Ubuntu SMP Wed Jul 28 23:14:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions