Skip to content

awslogs: Prevent close from being blocked on log#47748

Merged
thompson-shaun merged 1 commit intomoby:masterfrom
cpuguy83:logjam
Jan 17, 2025
Merged

awslogs: Prevent close from being blocked on log#47748
thompson-shaun merged 1 commit intomoby:masterfrom
cpuguy83:logjam

Conversation

@cpuguy83
Copy link
Copy Markdown
Member

@cpuguy83 cpuguy83 commented Apr 23, 2024

Before this change a call to Close could be blocked if the the channel used to buffer logs is full.
When this happens the container state will end up wedged causing a deadlock on anything that needs to lock the container state.

This removes the use of a channel which has semantics which are difficult to manage to something more suitable for the situation.

Closes #39523
I can't say for sure if this resolves every report in #39523 but with the limited information provided I think this is the best we can do.
If others experience an issue then they'll need to open a new issue with the needed details to track the problem down.

@cpuguy83 cpuguy83 requested a review from neersighted April 23, 2024 18:43
@cpuguy83 cpuguy83 requested a review from samuelkarp as a code owner April 23, 2024 18:43
@cpuguy83 cpuguy83 force-pushed the logjam branch 6 times, most recently from 9f0a44a to 7f2b744 Compare April 23, 2024 19:29
@cpuguy83 cpuguy83 marked this pull request as draft April 23, 2024 19:56
@cpuguy83 cpuguy83 force-pushed the logjam branch 2 times, most recently from 27b7ef5 to 3821810 Compare May 2, 2024 21:47
@cpuguy83 cpuguy83 marked this pull request as ready for review May 2, 2024 21:47
Copy link
Copy Markdown
Contributor

@austinvazquez austinvazquez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cpuguy83 nice refactor here! Overall changes LGTM. Just had one or two questions but nothing blocking.

"sync"

"github.com/docker/docker/daemon/logger"
"github.com/pkg/errors"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor-nit: should we use built-in errors package here instead?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use pkg/errors everywhere in moby, mostly because it handles attaching stack traces.

Comment thread daemon/logger/loggerutils/queue.go Outdated
Comment thread daemon/logger/awslogs/cloudwatchlogs.go
Copy link
Copy Markdown
Contributor

@austinvazquez austinvazquez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Great change.

Copy link
Copy Markdown
Member

@neersighted neersighted left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we need the closeWait channel as the entire Close() method takes the mutex, and therefore concurrent close should be impossible. It seems like we can just guard the method with a check that the object has not been already closed, and early-return if it has.

Otherwise, LGTM; I particularly like the Close vs. blocked Enqueue semantics.

Comment thread daemon/logger/loggerutils/queue.go Outdated
@cpuguy83 cpuguy83 force-pushed the logjam branch 2 times, most recently from 1314bde to a0d04eb Compare May 23, 2024 18:58
Before this change a call to `Close` could be blocked if the the channel
used to buffer logs is full.
When this happens the container state will end up wedged causing a
deadlock on anything that needs to lock the container state.

This removes the use of a channel which has semantics which are
difficult to manage to something more suitable for the situation.

Signed-off-by: Brian Goff <[email protected]>
@neersighted neersighted added this to the 27.0.0 milestone May 24, 2024
@vvoland
Copy link
Copy Markdown
Contributor

vvoland commented Jun 14, 2024

@neersighted LGTY?

@vvoland vvoland modified the milestones: 27.0.0, 28.0.0 Jun 14, 2024
@thompson-shaun thompson-shaun merged commit 41f4d91 into moby:master Jan 17, 2025
@cpuguy83 cpuguy83 deleted the logjam branch January 17, 2025 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docker stats hangs

6 participants