Skip to content

#42316 Wait for run goroutine to exit before Close#42320

Merged
thaJeztah merged 1 commit into
moby:masterfrom
anujva:fix_moby_ring_logger
May 27, 2021
Merged

#42316 Wait for run goroutine to exit before Close#42320
thaJeztah merged 1 commit into
moby:masterfrom
anujva:fix_moby_ring_logger

Conversation

@anujva

@anujva anujva commented Apr 25, 2021

Copy link
Copy Markdown
Contributor

Fixes #42316

The underlying Loggers Close(), function can be called with the the
run() goroutine still writing to the driver.
This is causing the fluentd-golang-logger to panic cause it doesn't
defensively check for the closing of the channel before writing to it.
It relies on the docker daemon to keep the contract of not calling Log()
if Close() has already been called.

- What I did
Synchronize the close of the underlying logger only after the run goroutine has exited.

- How I did it
Added a sync.Waitgroup to be instantiated withe the RingLogger and then added a wait condition of the goroutine to exit,
before Close is called on the underlying logger.

- How to verify it
I am not sure how best to verify it. Let me know if I should add a unit test for this change or something.

- Description for the changelog

Fixes the race condition between RingLogger Close() and run() goroutine.

- A picture of a cute animal (not mandatory but encouraged)
image

@anujva

anujva commented Apr 25, 2021

Copy link
Copy Markdown
Contributor Author

#42316 should be fixed by this. This panic is being caused by the fluentd-golang-logger not being defensive about the close of its channel. A comparable implementation of splunk actually uses a RWMutex to solve this problem, But this fix would solve the problem for both underlying drivers not needing any synchronization primitives.

@anujva
anujva force-pushed the fix_moby_ring_logger branch 3 times, most recently from 40cefda to 13802e5 Compare April 25, 2021 11:38
@anujva anujva changed the title Wait for run goroutine to exit before Close #42316 Wait for run goroutine to exit before Close Apr 25, 2021
Comment thread daemon/logger/ring.go Outdated
@anujva
anujva force-pushed the fix_moby_ring_logger branch from 13802e5 to d2fadac Compare April 30, 2021 23:07
Comment thread daemon/logger/ring.go Outdated
@anujva
anujva force-pushed the fix_moby_ring_logger branch from d2fadac to 7109e4e Compare April 30, 2021 23:42
The underlying Loggers Close() function can be called with the the
run() goroutine still writing to the driver. This is causing the
fluentd-golang-logger to panic cause it doesn't defensively check
for the closing of the channel before writing to it.
It relies on the docker daemon to keep the contract of not calling Log()
if Close() has already been called.

Contributions by: James Johnston <[email protected]>
                  Nathan Wong <[email protected]>

Signed-off-by: Anuj Varma <[email protected]>
@anujva
anujva force-pushed the fix_moby_ring_logger branch from 7109e4e to cf259eb Compare May 1, 2021 00:23

@cpuguy83 cpuguy83 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@thaJeztah
thaJeztah merged commit 08096e3 into moby:master May 27, 2021
@anujva
anujva deleted the fix_moby_ring_logger branch May 27, 2021 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Possible race condition between RingLogger's Close() and run()

3 participants