Skip to content
This repository was archived by the owner on Aug 30, 2019. It is now read-only.

{cmd/trace-agent,writer}: fix race conditions#495

Merged
gbbr merged 2 commits into
masterfrom
gbbr/race
Oct 16, 2018
Merged

{cmd/trace-agent,writer}: fix race conditions#495
gbbr merged 2 commits into
masterfrom
gbbr/race

Conversation

@gbbr

@gbbr gbbr commented Oct 16, 2018

Copy link
Copy Markdown
Contributor

Fixes some (non-fatal) race conditions detected by the -race flag.

@gbbr gbbr added the bug label Oct 16, 2018
@gbbr gbbr added this to the 6.6.0 milestone Oct 16, 2018
totalTraceCount uint64

mu sync.Mutex // guards lastFlush
lastFlush time.Time

@gbbr gbbr Oct 16, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is sampler.Sampler thread-safe? No errors thrown nor shown in the wild, just making sure.

Comment thread writer/fixtures_test.go
Comment thread cmd/trace-agent/sampler.go Outdated
Comment thread writer/fixtures_test.go

@AlexJF AlexJF left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's how I think a Start/Stop should look for correctness and no races:

func (w *Something) Start() {
  go func() {
    w.exitWG.Add(1)
    w.Run()
    w.exitWG.Done()
  }
}

func (w *Something) Stop() {
  close(exit)
  w.exitWG.Wait()
}

This should solve any races you might have observed no? Feel free to replace the wait group with another channel. Any other option, I fear, introduces the chances for weird corner cases.

// Add samples a trace and returns true if trace was sampled (should be kept), false otherwise
func (s *Sampler) Add(t processedTrace) bool {
s.totalTraceCount++
atomic.AddUint64(&s.totalTraceCount, 1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add can indeed be called from multiple routines simultaneously so 👍

@gbbr

gbbr commented Oct 16, 2018

Copy link
Copy Markdown
Contributor Author

Here's how I think a Start/Stop should look for correctness and no races:

Actually that's the original race right there. Calling Wait while Add-ing will cause a race condition. Check the documentation for sync.WaitGroup.Add

@gbbr

gbbr commented Oct 16, 2018

Copy link
Copy Markdown
Contributor Author

@AlexJF @bmermet PTAL

@AlexJF AlexJF left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok LGTM.

@AlexJF

AlexJF commented Oct 16, 2018

Copy link
Copy Markdown

Right. Although having moved all the waitgroup interactions to just Start and Stop, triggering the race condition would require a concurrent Start and Stop which is by definition racy.

@gbbr

gbbr commented Oct 16, 2018

Copy link
Copy Markdown
Contributor Author

Thanks for looking. I prefer to have a race-proof API, rather than an API which assumes that the user will not do anything dodgy, which they usually do 😄My sentiment is that a "wait group" should be used with a "group", which is not the case here. I'm open to simpler solutions, this one seems quite safe to me and we've experimented a lot with it within the Go tracer where we had problems of synchronous Start/Stop calls among other abusive behaviour.

@gbbr
gbbr merged commit 24d4c76 into master Oct 16, 2018
@gbbr
gbbr deleted the gbbr/race branch October 16, 2018 09:43
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants