Description

containerd does not respect the soft limit for RLIMIT_NOFILE, containers inherit a soft limit matching the hard limit.

During a lengthy investigation into decisions behind LimitNOFILE=infinity and it's impact, I have since verified that LimitNOFILE=2048:262144 (soft:hard) is not respected.

This appears to be due to a change in Go 1.19 (Aug 2022) to implicitly raise the RLIMIT_NOFILE soft limit to the hard limit value. On many Linux systems running systemd v240 or newer (since 2019), this translates into a hard limit of 2^30 (1073741816) when the limit is configured to infinity.

On a typical host system the soft limit would be 1024. Go 1.19 maintainers did not consider the change to negatively impact the majority of software built with it, but this has introduced a regression of containers inheriting the raised soft limit, despite explicitly requesting a low soft limit.

This now requires containers to more explicitly configure the limit, or for containerd to internally lower it back before running a container.

Steps to reproduce the issue

1. LimitNOFILE=2048:262144 in containerd.service should provide a distro agnostic soft + hard limit to observe this, or you can ensure this does not exist in containerd.service and it should match the systemd default hard limit of 2^19 (524288) which on any modern distro should be below fs.nr_open (infinity).
2. systemctl daemon-reload && systemctl restart containerd
3. cat /proc/$(pidof containerd)/limits shows the soft limit is the same as the hard limit for "Max open files".
4. docker run --rm -it alpine ash -c 'ulimit -Sn; ulimit -Hn' will output both soft and hard limits as the hard limit value.

Describe the results you received and expected

I expected the soft limit assigned to continue to apply to containers, not for it to be raised to the hard limit assigned.

It does not seem appropriate as a workaround to set same desired limits explicitly to each container config.

What version of containerd are you using?

1.6.19

Any other relevant information

# runc --version
runc version 1.1.4
spec: 1.0.2-dev
go: go1.19
libseccomp: 2.5.4

# uname -a
6.2.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 03 Mar 2023 15:58:31 +0000 x86_64 GNU/Linux
# Arch Linux, Docker v23, systemd 253

Show configuration if it is related to CRI plugin.

No response