Skip to content

container/state.go: fix race in StateString()#28912

Closed
LK4D4 wants to merge 1 commit into
moby:masterfrom
LK4D4:fix_state_string_race
Closed

container/state.go: fix race in StateString()#28912
LK4D4 wants to merge 1 commit into
moby:masterfrom
LK4D4:fix_state_string_race

Conversation

@LK4D4

@LK4D4 LK4D4 commented Nov 29, 2016

Copy link
Copy Markdown
Contributor

Overall State struct is extremely weird because it exposes Lock() and Unlock() which actively used by outside code. But I'm not sure how to fix that.
ping @mlaventure

@AkihiroSuda

AkihiroSuda commented Nov 29, 2016

Copy link
Copy Markdown
Member

cc @cpuguy83

IIRC previously similar discussion was done in #26668

@AkihiroSuda

Copy link
Copy Markdown
Member

CI hanging

00:14:46.880 --- FAIL: Test (335.68s)
00:14:46.882 panic: DockerSuite.TestAPINetworkConnectDisconnect test timed out after 5m0s [recovered]
00:14:46.883 	panic: DockerSuite.TestAPINetworkConnectDisconnect test timed out after 5m0s

@AkihiroSuda AkihiroSuda added status/failing-ci Indicates that the PR in its current state fails the test suite rebuild/experimental labels Nov 29, 2016
@LK4D4

LK4D4 commented Nov 29, 2016

Copy link
Copy Markdown
Contributor Author

Looks valid. That stuff is really bad.

@AkihiroSuda

Copy link
Copy Markdown
Member

BTW Gordon not recognizing rebuild/experimental label... 😕

@duglin

duglin commented Nov 29, 2016

Copy link
Copy Markdown
Contributor

Just wondering - why was it racey w/o the use of the lock? Is it because the s.* properties could change values 1/2 thru the processing of that func?

@cpuguy83

Copy link
Copy Markdown
Member

@duglin Yes, and could potentially even cause a panic.

@cpuguy83

Copy link
Copy Markdown
Member

@LK4D4 #28483 is the start of removing this lock altogether.
See master...cpuguy83:container_memdb -- this is a bit out of date but still the same basic idea of what I want to do to remove this lock.

@duglin

duglin commented Nov 29, 2016

Copy link
Copy Markdown
Contributor

could we not remove the lock and instead just copy the state into a local variable and then examine that? That should make it thread safe, no?

@cpuguy83

Copy link
Copy Markdown
Member

@duglin It's racey to look at the state without holding the lock, and just reading the state string is super fast. It's not about lock contention here, it's an issue of deadlocks.... in this case due to a goroutine trying to double-lock the container.

This lock is doing way way too much and it's causing lots of problems.

@LK4D4

LK4D4 commented Nov 29, 2016

Copy link
Copy Markdown
Contributor Author

@cpuguy83 I've added lock in specific place with race. I understand that it's bad, but lock is still there :/

@mlaventure mlaventure removed the status/failing-ci Indicates that the PR in its current state fails the test suite label Nov 29, 2016
@mlaventure

Copy link
Copy Markdown
Contributor

@LK4D4 is there a way to add a test for that race?

@LK4D4

LK4D4 commented Nov 30, 2016

Copy link
Copy Markdown
Contributor Author

@mlaventure there is no state tests at all :/ I can write one specifically for that race, but seems like we didn't run with -race anyway.

@cpuguy83

cpuguy83 commented Dec 6, 2016

Copy link
Copy Markdown
Member

I think the main place where this is a problem is in /info, right?
Can we just keep a running tally of states when the state change happens?
Then /info doesn't need to look at the container list at all.

@AkihiroSuda

Copy link
Copy Markdown
Member

@cpuguy83
The last revision of #26668 seems to have implemented that design.

However, the PR is closed because we found that we should cover docker ps as well.

@LK4D4

LK4D4 commented Dec 19, 2016

Copy link
Copy Markdown
Contributor Author

@cpuguy83 can we merge this in the meantime?

@cpuguy83

Copy link
Copy Markdown
Member

I really don't like locking the container mutex here.
Opened #29554 which implements a counter for states. This is at least a stop-gap until the container object is reigned in.

@AkihiroSuda

Copy link
Copy Markdown
Member

what's the conclusion?
Related: #30225

@LK4D4

LK4D4 commented Jan 27, 2017

Copy link
Copy Markdown
Contributor Author

I'm going to close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants