Description
As the tmpfs man page tells us it is not permitted to use user extended attributes:
The tmpfs filesystem supports extended attributes (see xattr(7)), but user extended attributes are not permitted.
containerd does not check the backing filesystem in the NeedsUserXAttr function.
The issue only will only occur when running in a userns, otherwise it will not use the userxattr mount option.
If containerd is executed in userns and its root is a tmpfs, it might run into issues at applyNaive function, while trying to extracting layers.
It will fail if it runs into return false, os.RemoveAll(originalPath) with the error that is reproduced below(the error will be thrown inside the go os package at removeAllFrom):
This example error was produced when pulling the docker.io/jupyterhub/configurable-http-proxy:4.5.3 image:
mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount926207991: failed to convert whiteout file "usr/local/lib/node_modules/.wh.npm": unlinkat /var/lib/containerd/tmpmounts/containerd-mount926207991/usr/local/lib/node_modules/npm/node_modules/yallist: input/output error: unknown
Steps to reproduce the issue
Test util for reproduction
This little script will create a overlay filesystem in the specified path.
One time with the userxattr mount option and one time without.
For each of these it will try to delete a folder with a file in it.
function execute_overlay_test_in {
local target="$1"
local lower="$target/test/lower"
local upper="$target/test/upper"
local merged="$target/test/merged"
local work="$target/test/work"
mkdir -p $lower/a $upper $merged $work
# Create file to be deleted
uname -a > $lower/a/a
# mount overlay
local overlayMountOptionsWorking="rw,relatime,lowerdir=$lower,upperdir=$upper,workdir=$work"
local overlayMountOptionsFailing="$overlayMountOptionsWorking,userxattr"
echo "Running mount with userxattr"
echo "Executing: mount -t overlay -o $overlayMountOptionsFailing overlay $merged"
mount -t overlay -o $overlayMountOptionsFailing overlay $merged
find $target/test -ls
echo "Try to delete $merged/a"
rm -rf $merged/a
find $target/test -ls
echo ""
echo "Unmounting and resetting test dirs"
umount $merged
rm -rf $target/test
mkdir -p $lower/a $upper $merged $work
# Create file to be deleted
uname -a > $lower/a/a
echo ""
echo ""
echo "Running mount without userxattr"
echo "Executing: mount -t overlay -o $overlayMountOptionsWorking overlay $merged"
mount -t overlay -o $overlayMountOptionsWorking overlay $merged
find $target/test -ls
echo "Try to delete $merged/a"
rm -rf $merged/a
find $target/test -ls
echo ""
echo "Unmounting and cleaning up test dirs"
umount $merged
rm -rf $target/test
}Test overlay with backing tmpfs
This tests will create the overlay using a backing tmpfs, as the man page of tmpfs specifies, it should have some unexpected behaviour to use the userxattr mount option:
The tmpfs filesystem supports extended attributes (see xattr(7)), but user extended attributes are not permitted.
sudo -s
mkdir /mnt/tmpfs_test
mount -t tmpfs -o size=200M none /mnt/tmpfs_test
execute_overlay_test_in /mnt/tmpfs_test
Result
When the userxattr mount option is used it will fail with:
rm: cannot remove '/mnt/tmpfs_test/test/merged/a': Input/output error
Otherwise it will work.
Test overlay with backing ext4
When we try this not using a tmpfs, in my case ext4, user extended attributes should be working:
sudo -s
mkdir /mnt/ext4_test
mount /dev/sda2 /mnt/ext4_test
execute_overlay_test_in /mnt/ext4_test
Result
File deletion works with userxattr mount option.
Describe the results you received and expected
Containerd will not use the userxattr for mounting overlay filesystem when using a tmpfs as backing fs.
What version of containerd are you using?
commit 021beb5
Any other relevant information
root@containerdtest:/mnt# runc --version
runc version 1.1.0-0ubuntu1.1
spec: 1.0.2-dev
go: go1.18.1
libseccomp: 2.5.3
root@containerdtest:/mnt# uname -a
Linux containerdtest 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Show configuration if it is related to CRI plugin.
No response
Description
As the tmpfs man page tells us it is not permitted to use user extended attributes:
containerd does not check the backing filesystem in the NeedsUserXAttr function.
The issue only will only occur when running in a userns, otherwise it will not use the
userxattrmount option.If containerd is executed in userns and its root is a tmpfs, it might run into issues at applyNaive function, while trying to extracting layers.
It will fail if it runs into
return false, os.RemoveAll(originalPath)with the error that is reproduced below(the error will be thrown inside the go os package at removeAllFrom):This example error was produced when pulling the
docker.io/jupyterhub/configurable-http-proxy:4.5.3image:Steps to reproduce the issue
Test util for reproduction
This little script will create a overlay filesystem in the specified path.
One time with the
userxattrmount option and one time without.For each of these it will try to delete a folder with a file in it.
Test overlay with backing tmpfs
This tests will create the overlay using a backing tmpfs, as the man page of tmpfs specifies, it should have some unexpected behaviour to use the
userxattrmount option:Result
When the
userxattrmount option is used it will fail with:Otherwise it will work.
Test overlay with backing ext4
When we try this not using a tmpfs, in my case ext4, user extended attributes should be working:
Result
File deletion works with
userxattrmount option.Describe the results you received and expected
Containerd will not use the
userxattrfor mounting overlay filesystem when using a tmpfs as backing fs.What version of containerd are you using?
commit 021beb5
Any other relevant information
root@containerdtest:/mnt# uname -a Linux containerdtest 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/LinuxShow configuration if it is related to CRI plugin.
No response