[release/2.3] runc-shim: don't hold the service lock across runc create#13512
Merged
mxpv merged 1 commit intoJun 2, 2026
Conversation
The task service guards its containers map with s.mu, and getContainer()
takes it on behalf of effectively every task RPC (State, Connect, Stats,
Wait, Pause, Kill, ...). Create() held s.mu for its whole duration,
including runc.NewContainer(), which runs the actual `runc create`.
`runc create` can be slow on a loaded host. While it runs, any concurrent
task RPC blocks on s.mu. The tasks service applies a 2s timeout to State
(io.containerd.timeout.task.state), so a concurrent State waits on s.mu,
exceeds the deadline, and the ttrpc call is abandoned -- the late shim
reply then shows up as:
ttrpc: received message on inactive stream stream=3
Since deadline errors are now surfaced to clients, this is treated as a
fatal failure and the just-created container is torn down right after
start (observed on Lima/vz: nginx -> Exited (1)).
Move runc.NewContainer() out of the s.mu critical section, mirroring the
runtime v1 shim lock optimization. s.mu is taken only once the container
exists, to guard the map and the remaining (fast) setup, so a slow create
no longer blocks concurrent State and other lookups.
preStart/handleStarted/cleanup only use s.lifecycleMu, so early-exit
handling is unchanged.
See lima-vm/lima#5030.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Akihiro Suda <[email protected]>
estesp
approved these changes
Jun 1, 2026
austinvazquez
approved these changes
Jun 2, 2026
github-project-automation
Bot
moved this from Needs Triage
to Review In Progress
in Pull Request Review
github-project-automation
Bot
moved this from Review In Progress
to Done
in Pull Request Review
jaredledvina
added a commit
to DataDog/containerd
that referenced
this pull request
Jul 7, 2026
containerd 2.3.2 Welcome to the v2.3.2 release of containerd! The second patch release for containerd 2.3 contains various fixes and updates including security patches. * **containerd** * [**CVE-2026-50195**](GHSA-cvxm-645q-p574) * [**CVE-2026-53488**](GHSA-xhf5-7wjv-pqxp) * [**CVE-2026-53492**](GHSA-33vj-92qq-66hc) * [**CVE-2026-53489**](GHSA-rgh6-rfwx-v388) * [**CVE-2026-47262**](GHSA-jpcc-p29g-p8mq) * Fix a data race when reading shim logs on Windows ([containerd#13522](containerd#13522)) * Allow the last host to retry on transient network errors ([containerd#13591](containerd#13591)) * Fix container startup failures caused by concurrent task RPC timeouts during slow container creation ([containerd#13512](containerd#13512)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. * Samuel Karp * Chris Henzie * Akihiro Suda * Derek McGowan * Akhil Mohan * Austin Vazquez * Ben Cressey * Brian Goff * Maksym Pavlenko * Sebastiaan van Stijn * Sergey Kanzhelev <details><summary>30 commits</summary> <p> * Prepare release notes for v2.3.2 ([containerd#13627](containerd#13627)) * [`fb8ca00b0`](containerd@fb8ca00) Prepare release notes for v2.3.2 * [`9c69960ba`](containerd@9c69960) Merge commit from fork * [`0f6251520`](containerd@0f62515) Merge commit from fork * [`91d7471e2`](containerd@91d7471) cri: filter CDI annotations on checkpoint restore * [`7c2e086bf`](containerd@7c2e086) Merge commit from fork * [`dae67765f`](containerd@dae6776) cri: do not re-tag restored checkpoints * [`94aa1e2c1`](containerd@94aa1e2) Merge commit from fork * [`09599078f`](containerd@0959907) cri: make checkpoint restore robust to unexpected archive content * [`e1fdb8d22`](containerd@e1fdb8d) Merge commit from fork * [`ff1d116ef`](containerd@ff1d116) Bound user-database file reads in openUserFile * [`d156e07cb`](containerd@d156e07) Merge commit from fork * [`f99aad54a`](containerd@f99aad5) Do not propagate reserved labels from image configs * vendor: golang.org/x/crypto v0.53.0 ([containerd#13608](containerd#13608)) * [`0b9469501`](containerd@0b94695) [release/2.3] vendor: golang.org/x/crypto v0.53.0 * resolver: retry on transient network errors ([containerd#13591](containerd#13591)) * [`983bbddc1`](containerd@983bbdd) resolver: retry on transient network errors * update runc binary to v1.4.3 ([containerd#13601](containerd#13601)) * [`3f76f2dc1`](containerd@3f76f2d) update runc binary to v1.4.3 * update go to 1.26.4 ([containerd#13580](containerd#13580)) * [`8a49dfe85`](containerd@8a49dfe) update go to 1.26.4 * [`5aa6bb2b7`](containerd@5aa6bb2) remove 1.26.2 from CI builds as it is not supported any longer due to the dependency * Configure udevd children-max for root-test ([containerd#13568](containerd#13568)) * [`bfb8aebc0`](containerd@bfb8aeb) Configure udevd children-max for root-test * core/runtime/v2: fix race on Windows deferredPipeConnection.c in Read ([containerd#13522](containerd#13522)) * [`62ceafff0`](containerd@62ceaff) core/runtime/v2: fix race on Windows deferredPipeConnection.c in Read * runc-shim: don't hold the service lock across runc create ([containerd#13512](containerd#13512)) * [`9b0c0dc58`](containerd@9b0c0dc) runc-shim: don't hold the service lock across runc create * contrib/checkpoint: increase timeouts to 30s ([containerd#13459](containerd#13459)) * [`f588bc6fb`](containerd@f588bc6) contrib/checkpoint: increase timeouts to 30s </p> </details> * **golang.org/x/crypto** v0.49.0 -> v0.53.0 * **golang.org/x/mod** v0.35.0 -> v0.36.0 * **golang.org/x/net** v0.52.0 -> v0.55.0 * **golang.org/x/sync** v0.20.0 -> v0.21.0 * **golang.org/x/sys** v0.43.0 -> v0.46.0 * **golang.org/x/term** v0.41.0 -> v0.44.0 * **golang.org/x/text** v0.35.0 -> v0.38.0 Previous release can be found at [v2.3.1](https://github.com/containerd/containerd/releases/tag/v2.3.1) * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automated cherry-pick of #13483
/assign AkihiroSuda