runc-shim: don't hold the service lock across runc create by AkihiroSuda · Pull Request #13483 · containerd/containerd

AkihiroSuda · 2026-05-27T04:15:28Z

The task service guards its containers map with s.mu, and getContainer() takes it on behalf of effectively every task RPC (State, Connect, Stats, Wait, Pause, Kill, ...). Create() held s.mu for its whole duration, including runc.NewContainer(), which runs the actual runc create.

runc create can be slow on a loaded host. While it runs, any concurrent task RPC blocks on s.mu. The tasks service applies a 2s timeout to State (io.containerd.timeout.task.state), so a concurrent State waits on s.mu, exceeds the deadline, and the ttrpc call is abandoned -- the late shim reply then shows up as:

ttrpc: received message on inactive stream stream=3

Since deadline errors are now surfaced to clients, this is treated as a fatal failure and the just-created container is torn down right after start (observed on Lima/vz: nginx -> Exited (1)).

Move runc.NewContainer() out of the s.mu critical section, mirroring the runtime v1 shim lock optimization. s.mu is taken only once the container exists, to guard the map and the remaining (fast) setup, so a slow create no longer blocks concurrent State and other lookups. preStart/handleStarted/cleanup only use s.lifecycleMu, so early-exit handling is unchanged.

See:

Integration tests (vz) (default.yaml) is failing after updating nerdctl to v2.3.1 lima-vm/lima#5030.

The regression is caused by f078cebbd

fix: propagate context deadline exceeded error properly #12821

f078cebbd https://github.com/lima-vm/lima/actions/runs/26487350835/job/77997535035?pr=5049

+ limactl shell default nerdctl ps -a
time="2026-05-27T02:56:35Z" level=warning msg="treating lima version \"0c93a37\" from \"/Users/runner/.lima/default/lima-version\" as very latest release"
CONTAINER ID    IMAGE                                              COMMAND                   CREATED           STATUS                      PORTS                     NAMES
3f67f52a5931    ghcr.io/stargz-containers/nginx:1.19-alpine-org    "/docker-entrypoint.…"    10 seconds ago    Exited (1) 5 seconds ago    127.0.0.1:8080->80/tcp    nginx
+ limactl shell default nerdctl logs nginx
time="2026-05-27T02:56:36Z" level=warning msg="treating lima version \"0c93a37\" from \"/Users/runner/.lima/default/lima-version\" as very latest release"
+ [[ -z '' ]]
+ limactl shell default journalctl --user -u containerd
[...]
May 27 02:55:49 lima-default containerd-rootless.sh[1173]: time="2026-05-27T02:55:49.956586096Z" level=info msg="containerd successfully booted in 7.782954s"
May 27 02:56:25 lima-default containerd-rootless.sh[1173]: time="2026-05-27T02:56:25.445537685Z" level=info msg="connecting to shim 3f67f52a59314ed6fc4e1048b0e2c826ab26df575cec4cbf38619beb754dbebd" address="unix:///run/containerd/s/56dfcbff8ef161f104b965a6f02b0158fca28ed1beba7df4c255935edf3a9122" namespace=default protocol=ttrpc version=3
May 27 02:56:29 lima-default containerd-rootless.sh[1173]: time="2026-05-27T02:56:29.753250886Z" level=error msg="ttrpc: received message on inactive stream" stream=3

f078cebbd~1 https://github.com/lima-vm/lima/actions/runs/26487350835/job/77997535032?pr=5049

Note: used Claude

Copilot

Pull request overview

This PR reduces lock contention in the runc v2 shim task service by avoiding holding the global service.mu (which guards the containers map) while executing runc create during Create(). This prevents unrelated task RPCs (e.g., State, Stats, Connect) from blocking behind a potentially slow container creation and timing out.

Changes:

Move runc.NewContainer() (which runs runc create) outside of the s.mu critical section in Create().
Add an explanatory comment documenting why s.mu should not be held across runc create.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

AkihiroSuda · 2026-05-27T04:55:14Z

Running the test in:

Test containerd PR 13483 (runc-shim: don't hold the service lock across runc create) lima-vm/lima#5053

AkihiroSuda · 2026-05-27T05:23:06Z

/cherry-pick release/2.3

k8s-infra-cherrypick-robot · 2026-05-27T05:23:09Z

@AkihiroSuda: once the present PR merges, I will cherry-pick it on top of release/2.3 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release/2.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

ningmingxiao · 2026-05-27T07:41:50Z

can you show the code container is torn down in nerdctl ? @AkihiroSuda

AkihiroSuda · 2026-05-27T08:07:22Z

can you show the code container is torn down in nerdctl ? @AkihiroSuda

See:

[DNM] bisect containerd lima-vm/lima#5049 (comment)

ningmingxiao · 2026-05-27T08:57:18Z

I add some sleep

func (s *service) Create(ctx context.Context, r *taskAPI.CreateTaskRequest) (_ *taskAPI.CreateTaskResponse, err error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.lifecycleMu.Lock()
	handleStarted, cleanup := s.preStart(nil)
	s.lifecycleMu.Unlock()
	defer cleanup()

	container, err := runc.NewContainer(ctx, s.platform, r)
	if err != nil {
		return nil, err
	}
	time.Sleep(time.Second * 3)

nerdctl run -d busybox sleep 1000 but I can't see container will be killed
only see

time="2026-05-27T16:48:10.057569440+08:00" level=error msg="ttrpc: received message on inactive stream" stream=3

@AkihiroSuda

AkihiroSuda · 2026-05-27T08:58:55Z

So far, the issue is only seen in Lima's CI:

[DNM] bisect containerd lima-vm/lima#5049 (comment)

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

The task service guards its containers map with s.mu, and getContainer() takes it on behalf of effectively every task RPC (State, Connect, Stats, Wait, Pause, Kill, ...). Create() held s.mu for its whole duration, including runc.NewContainer(), which runs the actual `runc create`. `runc create` can be slow on a loaded host. While it runs, any concurrent task RPC blocks on s.mu. The tasks service applies a 2s timeout to State (io.containerd.timeout.task.state), so a concurrent State waits on s.mu, exceeds the deadline, and the ttrpc call is abandoned -- the late shim reply then shows up as: ttrpc: received message on inactive stream stream=3 Since deadline errors are now surfaced to clients, this is treated as a fatal failure and the just-created container is torn down right after start (observed on Lima/vz: nginx -> Exited (1)). Move runc.NewContainer() out of the s.mu critical section, mirroring the runtime v1 shim lock optimization. s.mu is taken only once the container exists, to guard the map and the remaining (fast) setup, so a slow create no longer blocks concurrent State and other lookups. preStart/handleStarted/cleanup only use s.lifecycleMu, so early-exit handling is unchanged. See lima-vm/lima#5030. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda · 2026-06-01T16:05:57Z

/cherry-pick release/2.3

k8s-infra-cherrypick-robot · 2026-06-01T16:06:01Z

@AkihiroSuda: once the present PR merges, I will cherry-pick it on top of release/2.3 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release/2.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mikebrow

concern over possible unforeseen timing changes

mikebrow · 2026-06-01T16:00:29Z

 	handleStarted, cleanup := s.preStart(nil)
 	s.lifecycleMu.Unlock()
 	defer cleanup()



perhaps a mux for each r.ID, to hold lock for each, across the newcontainer + task create event + cgroup oom monitor + started tasks?

Not sure. s.send doesn't seem to alter the struct s

kk

FYI @fuweid ... might "enhance" the parallelism of sidecar/init/containers pls be on the lookout

The lock behaviour was there for 7 years. To be honest, it's not easy-review one.
BTW, @AkihiroSuda could you please remove the AI-co-author next time? I would like to update project-check to check this. I am not saying we can't use AI. just want the sign-off one is real person. Thanks

BTW, @AkihiroSuda could you please remove the AI-co-author next time? I would like to update project-check to check this. I am not saying we can't use AI. just want the sign-off one is real person. Thanks

I'm not sure. The sign-off was added by myself, not by AI.
(IIUC, Co-Authored-by does not count as a sign-off)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Akihiro Suda <[email protected]>

We should have a solid policy anyway.
Could you open an issue or a PR against https://github.com/containerd/project/blob/main/CONTRIBUTING.md#sign-your-work ?

k8s-infra-cherrypick-robot · 2026-06-01T17:23:19Z

@AkihiroSuda: new pull request created: #13512

Details

In response to this:

/cherry-pick release/2.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot AI review requested due to automatic review settings May 27, 2026 04:15

github-project-automation Bot added this to Pull Request Review May 27, 2026

k8s-ci-robot added the do-not-merge/work-in-progress label May 27, 2026

github-project-automation Bot moved this to Needs Triage in Pull Request Review May 27, 2026

k8s-ci-robot added the size/S label May 27, 2026

Copilot started reviewing on behalf of AkihiroSuda May 27, 2026 04:15 View session

AkihiroSuda mentioned this pull request May 27, 2026

Test containerd PR 13483 (runc-shim: don't hold the service lock across runc create) lima-vm/lima#5053

Closed

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread cmd/containerd-shim-runc-v2/task/service.go

AkihiroSuda force-pushed the fix-lima-5030 branch from 4c55365 to 64f7673 Compare May 27, 2026 04:52

AkihiroSuda added the cherry-pick/2.3.x Change to be cherry picked to release/2.3 label May 27, 2026

AkihiroSuda marked this pull request as ready for review May 27, 2026 05:22

Copilot AI review requested due to automatic review settings May 27, 2026 05:22

k8s-ci-robot removed the do-not-merge/work-in-progress label May 27, 2026

Copilot started reviewing on behalf of AkihiroSuda May 27, 2026 05:22 View session

AkihiroSuda mentioned this pull request May 27, 2026

Integration tests (vz) (default.yaml) is failing after updating nerdctl to v2.3.1 lima-vm/lima#5030

Closed

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread cmd/containerd-shim-runc-v2/task/service.go Outdated

AkihiroSuda force-pushed the fix-lima-5030 branch from 64f7673 to e29b724 Compare May 27, 2026 05:27

k8s-ci-robot added size/XS and removed size/S labels May 27, 2026

AkihiroSuda added the easy-to-review Easy to review label May 27, 2026

AkihiroSuda mentioned this pull request May 27, 2026

AlmaLinux CI fails to start: Error: Unable to find a match: kernel-modules-extra-6.12.0-124.56.5.el10_1.x86_64 #13486

Closed

unsuman mentioned this pull request May 27, 2026

v2.1.2 planning (for Ubuntu 26.04) lima-vm/lima#4814

Closed

Copilot AI review requested due to automatic review settings May 28, 2026 00:03

AkihiroSuda force-pushed the fix-lima-5030 branch from e29b724 to dbcaa50 Compare May 28, 2026 00:03

Copilot started reviewing on behalf of AkihiroSuda May 28, 2026 00:03 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

AkihiroSuda requested a review from a team May 29, 2026 15:57

estesp approved these changes Jun 1, 2026

View reviewed changes

austinvazquez approved these changes Jun 1, 2026

View reviewed changes

github-project-automation Bot moved this from Needs Triage to Review In Progress in Pull Request Review Jun 1, 2026

AkihiroSuda added this pull request to the merge queue Jun 1, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 1, 2026

AkihiroSuda added this pull request to the merge queue Jun 1, 2026

mikebrow reviewed Jun 1, 2026

View reviewed changes

Merged via the queue into containerd:main with commit c8802b6 Jun 1, 2026
50 checks passed

github-project-automation Bot moved this from Review In Progress to Done in Pull Request Review Jun 1, 2026

k8s-infra-cherrypick-robot mentioned this pull request Jun 1, 2026

[release/2.3] runc-shim: don't hold the service lock across runc create #13512

Merged

fuweid mentioned this pull request Jun 4, 2026

CONTRIBUTING.md: clarify AI usage in commit trailers containerd/.project#166

Open

claytono mentioned this pull request Jul 13, 2026

Upgrade containerd to 2.3.2+ when supported by the Ansible role claytono/infra#2550

Open

Uh oh!

Conversation

AkihiroSuda commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

AkihiroSuda commented May 27, 2026

Uh oh!

AkihiroSuda commented May 27, 2026

Uh oh!

k8s-infra-cherrypick-robot commented May 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

ningmingxiao commented May 27, 2026

Uh oh!

AkihiroSuda commented May 27, 2026

Uh oh!

ningmingxiao commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented May 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

AkihiroSuda commented Jun 1, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jun 1, 2026

Uh oh!

Uh oh!

mikebrow left a comment

Choose a reason for hiding this comment

Uh oh!

mikebrow Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

mikebrow Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fuweid Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

k8s-infra-cherrypick-robot commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

AkihiroSuda commented May 27, 2026 •

edited

Loading

ningmingxiao commented May 27, 2026 •

edited

Loading

mikebrow Jun 1, 2026 •

edited

Loading