Refresh containerd remotes on containerd restarted by cpuguy83 · Pull Request #36173 · moby/moby

cpuguy83 · 2018-01-31T22:39:17Z

Before this patch, when containerd is restarted (due to a crash, or
kill, whatever), the daemon would keep trying to process the event
stream against the old socket handles. This would lead to a CPU spin due
to the error handling when the client can't connect to containerd.

This change makes sure the containerd remote client is updated for all
registered libcontainerd clients.

This is not necessarily the ideal fix which would likely require a
major refactor, but at least gets things to a working state with a
minimal patch.

Fixes #36002

cpuguy83 · 2018-01-31T22:39:59Z

ping @thaJeztah

stevvooe · 2018-01-31T22:55:36Z

LGTM

vdemeester

LGTM 🐯

thaJeztah · 2018-01-31T23:31:48Z

Error response from daemon: driver "windowsfilter" failed to remove root filesystem for 2cdd9517ebbcff26868b4aaecbb61b6e9a9a201382645575954cea935cf8cfe4: rename D:\CI\CI-c0f59a531\daemon\windowsfilter\2cdd9517ebbcff26868b4aaecbb61b6e9a9a201382645575954cea935cf8cfe4 D:\CI\CI-c0f59a531\daemon\windowsfilter\2cdd9517ebbcff26868b4aaecbb61b6e9a9a201382645575954cea935cf8cfe4-removing: Access is denied.
23:16:29 
	Messages:   	failed to remove 2cdd9517ebbcff26868b4aaecbb61b6e9a9a201382645575954cea935cf8cfe4
23:16:29

mlaventure · 2018-02-01T15:52:42Z

@thaJeztah Windows doesn't use containerd yet :)

mlaventure

LGTM

cpuguy83 · 2018-02-01T16:08:20Z

Well, something here seems to not like CI....

mlaventure · 2018-02-01T16:14:25Z

The windows CI is definitely unrelated, since this code is not even built on windows.

I see that z failed too, that's something else unless z is known to be flaky sometimes? But it wasn't last time I paid attention.

cpuguy83 · 2018-02-01T16:36:40Z

Janky is also stuck waiting for the daemon to start, multiple times now.

mlaventure · 2018-02-01T16:59:18Z

The stack trace would help, I can't see how this is causing it so far :?

Before this patch, when containerd is restarted (due to a crash, or kill, whatever), the daemon would keep trying to process the event stream against the old socket handles. This would lead to a CPU spin due to the error handling when the client can't connect to containerd. This change makes sure the containerd remote client is updated for all registered libcontainerd clients. This is not neccessarily the ideal fix which would likely require a major refactor, but at least gets things to a working state with a minimal patch. Signed-off-by: Brian Goff <[email protected]>

cpuguy83 · 2018-02-07T16:57:25Z

Found the issue. It was due to recursive locking in Restore()

cpuguy83 · 2018-02-07T19:22:09Z

And green.

vdemeester

LGTM 🐸

Install docker 17.12.1-ce-rc2 to fix moby/moby#36173

… feature/merge-in-v2.3.5 to feature/add_latest_vault_plugin * commit 'bcb6fe5980595d336a08c9b16333c58e733a1758': (52 commits) use stable agent use ami that doesn't require usage agreement Update changelog for 2.3.5-rc2 Bump changelog for v2.3.5 Show docker logs if docker isn't running Fix linting issue in docker installer Install 17.12.1-ce-rc2 to fix moby/moby#36173 Revert "Merge pull request buildkite#360 from buildkite/vpc-rewrite-for-all-available-subnets" Update changelog for v2.3.1-v2.3.4 Add some troubleshooting tips to README Configure docker config in boothook to avoid race conditions Check docker is running in environment Remove du output, will be too slow Try cleaning up disk in the environment hook if needed Trim leading whitespace from disk space output 💅🏻 Make disk space output more human friendly Append to elastic-stack.log rather than overwrite Only redirect cron output if caller is root Tests should run cron as root Add logging to cron tasks and show disk usage in stack environment ...

… feature/add_latest_vault_plugin to use-vault-plugin * commit 'f50fdfc20190313bd80ddabe1b69fa21d09b3e29': (55 commits) use stable agent use ami that doesn't require usage agreement specify vault-backend branch for vault plugin update to latest vault plugin update to latest vault plugin Update changelog for 2.3.5-rc2 Bump changelog for v2.3.5 Show docker logs if docker isn't running Fix linting issue in docker installer Install 17.12.1-ce-rc2 to fix moby/moby#36173 Revert "Merge pull request buildkite#360 from buildkite/vpc-rewrite-for-all-available-subnets" Update changelog for v2.3.1-v2.3.4 Add some troubleshooting tips to README Configure docker config in boothook to avoid race conditions Check docker is running in environment Remove du output, will be too slow Try cleaning up disk in the environment hook if needed Trim leading whitespace from disk space output 💅🏻 Make disk space output more human friendly Append to elastic-stack.log rather than overwrite ...

In case we are killing and restarting containerd, do try to reconnect and use the new connection. Loosely based on moby#36173 Might help https://bugzilla.redhat.com/show_bug.cgi?id=1746435 Signed-off-by: Kir Kolyshkin <[email protected]>

GordonTheTurtle added the status/0-triage label Jan 31, 2018

cpuguy83 added status/2-code-review and removed status/0-triage labels Jan 31, 2018

cpuguy83 requested a review from mlaventure January 31, 2018 22:40

vdemeester approved these changes Jan 31, 2018

View reviewed changes

thaJeztah added the rebuild/z label Jan 31, 2018

GordonTheTurtle removed the rebuild/z label Jan 31, 2018

thaJeztah added area/runtime Runtime impact/changelog labels Jan 31, 2018

cpuguy83 added the rebuild/windowsRS1 label Feb 1, 2018

GordonTheTurtle removed the rebuild/windowsRS1 label Feb 1, 2018

mlaventure approved these changes Feb 1, 2018

View reviewed changes

JennyLawrance mentioned this pull request Feb 1, 2018

Stop/Start of containers fail with "/var/run/docker/containerd/docker-containerd.sock: connect: connection refused" #36117

Closed

cpuguy83 force-pushed the fix_containerd_crash_spin branch 2 times, most recently from d11d95b to 2247756 Compare February 6, 2018 03:45

cpuguy83 force-pushed the fix_containerd_crash_spin branch from 2247756 to 400126f Compare February 7, 2018 16:57

vdemeester approved these changes Feb 8, 2018

View reviewed changes

yongtang merged commit 384ff69 into moby:master Feb 8, 2018

cpuguy83 deleted the fix_containerd_crash_spin branch February 8, 2018 14:27

thaJeztah mentioned this pull request Feb 8, 2018

[17.12] Refresh containerd remotes on containerd restarted docker-archive/docker-ce#417

Merged

gugahoi mentioned this pull request Feb 15, 2018

Builds failing with docker errors buildkite/elastic-ci-stack-for-aws#381

Closed

benbc mentioned this pull request Feb 19, 2018

Synchronization issue between containerd and Docker docker/for-linux#155

Open

3 tasks

cpuguy83 mentioned this pull request Feb 21, 2018

Killing containerd breaks interaction with running containers #36367

Closed

lox added a commit to buildkite/elastic-ci-stack-for-aws that referenced this pull request Feb 23, 2018

Install 17.12.1-ce-rc2 to fix moby/moby#36173

14e2d86

lox added a commit to buildkite/elastic-ci-stack-for-aws that referenced this pull request Feb 23, 2018

Merge pull request #385 from buildkite/bump-docker-to-17.12.1-ce-rc2

ce24abf

Install docker 17.12.1-ce-rc2 to fix moby/moby#36173

lox added a commit to buildkite/elastic-ci-stack-for-aws that referenced this pull request Feb 24, 2018

Install 17.12.1-ce-rc2 to fix moby/moby#36173

1aa4c8b

lox added a commit to buildkite/elastic-ci-stack-for-aws that referenced this pull request Feb 26, 2018

Install 17.12.1-ce-rc2 to fix moby/moby#36173

e3224c9

thaJeztah mentioned this pull request Mar 5, 2018

Killing docker-containerd breaks interaction with containers #36002

Closed

seven-eleven mentioned this pull request Mar 12, 2018

docker exec fail after kill docker-containerd #36558

Closed

mikeknox pushed a commit to mikeknox/elastic-ci-stack-for-aws that referenced this pull request Apr 10, 2018

Install 17.12.1-ce-rc2 to fix moby/moby#36173

dad9fcd

pdcastro mentioned this pull request Feb 4, 2019

If containerd dies, engine tries to reuse the old connection balena-os/balena-engine#142

Open

robertgzr mentioned this pull request Feb 25, 2019

[backport] Refresh containerd remotes on containerd restarted balena-os/balena-engine#149

Closed

kolyshkin mentioned this pull request Jul 30, 2020

Docker 1.13.1 rhel reconnect projectatomic/docker#377

Merged

Conversation

cpuguy83 commented Jan 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpuguy83 commented Jan 31, 2018

Uh oh!

stevvooe commented Jan 31, 2018

Uh oh!

vdemeester left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Jan 31, 2018

Uh oh!

mlaventure commented Feb 1, 2018

Uh oh!

mlaventure left a comment

Choose a reason for hiding this comment

Uh oh!

cpuguy83 commented Feb 1, 2018

Uh oh!

mlaventure commented Feb 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpuguy83 commented Feb 1, 2018

Uh oh!

mlaventure commented Feb 1, 2018

Uh oh!

cpuguy83 commented Feb 7, 2018

Uh oh!

cpuguy83 commented Feb 7, 2018

Uh oh!

vdemeester left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cpuguy83 commented Jan 31, 2018 •

edited

Loading

mlaventure commented Feb 1, 2018 •

edited

Loading