Skip to content
This repository was archived by the owner on Aug 30, 2019. It is now read-only.

cmd/trace-agent: reduce histogram calls in concentrator#522

Merged
gbbr merged 2 commits into
masterfrom
gbbr/stats-reduce
Nov 8, 2018
Merged

cmd/trace-agent: reduce histogram calls in concentrator#522
gbbr merged 2 commits into
masterfrom
gbbr/stats-reduce

Conversation

@gbbr

@gbbr gbbr commented Nov 7, 2018

Copy link
Copy Markdown
Contributor

Attemtp to reduce UDP hits and flooding.

We had reports of excessive UDP packets (>3000 per 10 seconds) for metric datadog.trace_agent.err_distribution.len, all with values of zero. Seems uncalled for. This change should hopefully alleviate.

@gbbr
gbbr requested a review from LotharSee November 7, 2018 15:24
@gbbr
gbbr force-pushed the gbbr/stats-reduce branch 2 times, most recently from 4875242 to a782fe6 Compare November 7, 2018 16:10
@gbbr

gbbr commented Nov 7, 2018

Copy link
Copy Markdown
Contributor Author

I reduced the rate to 0.2, but not sure if this will alleviate the problem enough.

@gbbr gbbr added this to the 6.7.0 milestone Nov 7, 2018
@LotharSee

Copy link
Copy Markdown

Good catch, contrary to most metrics which we batch internally/report periodically, this one was done once per distribution.

In practice, we never look at it. I'd suggest to simply remove it.

gbbr added 2 commits November 8, 2018 09:53
These metrics were causing excessive UDP hits, errors and flooding.
This change removes them.
This change moves the `bucket.Export` call to after the precondition
where it is needed in an attempt to save some memory and CPU.
@gbbr
gbbr force-pushed the gbbr/stats-reduce branch from a782fe6 to 47278f4 Compare November 8, 2018 08:56
@gbbr

gbbr commented Nov 8, 2018

Copy link
Copy Markdown
Contributor Author

Great! That's probably the best thing to do. Note that I've added another commit with a small improvement. PTAL.

@gbbr
gbbr merged commit 4a56f12 into master Nov 8, 2018
@gbbr
gbbr deleted the gbbr/stats-reduce branch November 8, 2018 09:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants