[Tracer] Refactor stats computation into StatsAggregator#3133
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
bouwkast
left a comment
There was a problem hiding this comment.
I like the refactorings done to the samplers makes it a lot cleaner and easier to understand :)
This comment has been minimized.
This comment has been minimized.
1ed8df4 to
414732d
Compare
This comment has been minimized.
This comment has been minimized.
…default value `true`. This should make comprehension easier
- The span sampled by the RareSampler receives the metric _dd.rare=1 - A trace is kept if any span in the trace has an error - A trace is kept if any span is sampled by its analytic events sample rate
…sampling is done at the AgentWriter level, not the TraceContext level
414732d to
6885ce6
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
andrewlock
left a comment
There was a problem hiding this comment.
LGTM, I like the refactoring
| } | ||
| } | ||
|
|
||
| return false; |
There was a problem hiding this comment.
Just to confirm, HandlePriorityTrace always returns false? Also, _keys seems to have unbounded growth unless I'm mistaken?
There was a problem hiding this comment.
Regarding the unbounded keys, we said last time that as we already have clients with more than 1000 resources it could be worth it to limit the size of the set indeed (as it is done in Java IIRC what Zach had answered).
There was a problem hiding this comment.
Just to confirm, HandlePriorityTrace always returns false?
Yes, according to the agent logic we should return false when the trace has a positive sampling priority.
Also, _keys seems to have unbounded growth unless I'm mistaken?
Yes, I didn't fix this in the last PR but now would be a good time to implement this. I'll think on the implementation and get back to this point.
| // TODO: Add ability to disable the RareSampler, which the trace agent has | ||
| // Run the RareSampler early to make sure the signature gets counted | ||
| var rare = _rareSampler.Sample(trace); |
There was a problem hiding this comment.
Given the rare sampler is the only one with state, had we may as well just return immediately if this is true? If so, then FWIW, you could short-circuit here if preferable for readability:
return _rareSampler.Sample(trace)
|| _prioritySampler.Sample(trace)
|| _errorSampler.Sample(trace)
|| _analyticsEventSampler.Sample(trace);There was a problem hiding this comment.
What you propose has really great readability. I'll try to do something similar in my coming changes, but note that the RareSampler must always run first (unless disabled), but its result is used as the very last fallback
There was a problem hiding this comment.
OK I made a refactor similar to what you suggest in 69b3115 . The difference is that the last OR operand is the result of the _rareSampler.Sample(trace), which was run earlier in the method. The reason for this is documented in the code, so hopefully that's clear
There was a problem hiding this comment.
OK I made a refactor similar to what you suggest in 69b3115
Cool.
The reason for this is documented in the code, so hopefully that's clear
The reasoning there is clear, thanks. And I understand keeping it like this for consistency with other traceers, but...
but note that the RareSampler must always run first (unless disabled), but its result is used as the very last fallback
This makes sense to me if the other samplers have side-effects. But as far as I can see, they don't. so if the rareSpanFound is true, the result is always true, and running the other samplers is unnecessary?
I'm not passionate about it, especially as if the other samplers do have side effects (or might in the future), then the order is important
There was a problem hiding this comment.
Oh I see what you're saying about side effects, I wasn't fully understanding what you were saying that earlier. Yes since the other samplers have no side effects (right now) then we could possibly return early. I'll keep this as-is for now since we might eventually want to make more robust samplers (closer to what the agent has) that would need to keep some state
pierotibou
left a comment
There was a problem hiding this comment.
Nice refacto, thank you for doing it. I mainly have comments on RareSampling. And to answer your questions in the description:
IStatsAggregator.RunSamplers mimics the name in the trace agent code, but should we use another name? Perhaps IStatsAggregator.SampleTrace?
I think it should be named ShouldKeepTrace personally.
Does the sampling belong in the IStatsAggregator or in the AgentWriter? We only have to do the sampling when stats computation is enabled so that's why I originally stuck it in IStatsAggregator
I think it's fine here for now. What I challenge more is to have all this in AgentWriter personally. I think those steps should happen beforehand. But clearly we can come back to this later (if we agree).
Is the ITraceSampler interface necessary?
I think it makes sense to have dedicated classes for the various behaviours, it's easier to read. That said, I'm not sure all samplers fit the interface currently, I think the RareSampler should implement 2 distinct methods (cf comments)
| } | ||
| } | ||
|
|
||
| public bool RunSamplers(ArraySegment<Span> trace) |
There was a problem hiding this comment.
As it returns a boolean, I believe it would be better to call this ShouldKeepTrace
| public bool RunSamplers(ArraySegment<Span> trace) | ||
| { | ||
| // TODO: Add ability to disable the RareSampler, which the trace agent has | ||
| // Run the RareSampler early to make sure the signature gets counted |
There was a problem hiding this comment.
Maybe we could split the RareSampler in 2 parts:
- Span? Dectect(ArraySegment) that will register keys and return the first rare span if any
- bool Sample(Span?) that will add the tag if there's a span
The sampling part would be called only after all other samplers have been called.
I believe that would simplify the RareSampler code as you wouldn't have to handle differently spans with positive or negative priorities.
There was a problem hiding this comment.
The sampling part would be called only after all other samplers have been called.
So after a bit of reading the agent code and testing, this isn't quite true. Here's the algorithm we need:
- Run the RareSampler first. If the sampling priority is > 0, the RareSampler will not mark any spans with "_dd.rare = 1". Otherwise, the first rare span in the chunk that's determined to be rare is marked as "_dd.rare = 1"
- If no other samplers decide to sample the trace chunk, return the result of the RareSampler as the decision for the trace
There was a problem hiding this comment.
In commit 69b3115 I updated the RareSampler slightly. I documented the reasons for the algorithm so the algorithm itself didn't change too much, but I tried to refactor and add comments so it's clearer. Let me know what you think
There was a problem hiding this comment.
Run the RareSampler first. If the sampling priority is > 0, the RareSampler will not mark any spans with "_dd.rare = 1". Otherwise, the first rare span in the chunk that's determined to be rare is marked as "_dd.rare = 1"
If no other samplers decide to sample the trace chunk, return the result of the RareSampler as the decision for the trace
So to make sure IUC, the difference in behaviour with what I suggested is the fact that a span can be marked as _dd.rare = 1 even though it is sampled by another sampler (eg we have a span with prio <=0, it is rare, we add the tag, but it also contains an error, so it is returned anyway). Whereas in my suggestion, we wouldn't add the _dd.rare = 1 in that case. Right?
| } | ||
| } | ||
|
|
||
| return false; |
There was a problem hiding this comment.
Regarding the unbounded keys, we said last time that as we already have clients with more than 1000 resources it could be worth it to limit the size of the set indeed (as it is done in Java IIRC what Zach had answered).
|
|
||
| namespace Datadog.Trace.Util | ||
| { | ||
| internal class SamplingHelpers |
There was a problem hiding this comment.
Should we add a helper method like MarkedAsKept (not the best name but you see what I mean) that would contain that code
trace.Array[trace.Offset].Context.TraceContext.SamplingPriority is int p && p > 0;
There was a problem hiding this comment.
Implemented in 45c2227 with name IsKeptBySamplingPriority
…regator.ShouldKeepTrace
…it to helper method SamplingHelpers.IsKeptBySamplingPriority
… main design points: 1) Before running other samplers, if a trace is already kept by sampling priority, send the trace to the RareSampler first only to mark the stats points (but do not return the sampling decision yet) 2) Return the sampling decision of the RareSampler as the last sampler in the sampler chain.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…0 items in the set of seen spans at all times.
Benchmarks Report 🐌Benchmarks for #3133 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️
|
| Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
|---|---|---|---|---|
| Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 | 1.137 | 1,042.33 | 916.74 |
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | StartFinishSpan |
net472 | 1.21μs | 0.172ns | 0.642ns | 0.128 | 0 | 0 | 810 B |
| master | StartFinishSpan |
netcoreapp3.1 | 1.04μs | 0.599ns | 2.32ns | 0.0103 | 0 | 0 | 760 B |
| master | StartFinishScope |
net472 | 1.36μs | 0.637ns | 2.47ns | 0.141 | 0 | 0 | 891 B |
| master | StartFinishScope |
netcoreapp3.1 | 1.09μs | 1.01ns | 3.93ns | 0.012 | 0 | 0 | 880 B |
| #3133 | StartFinishSpan |
net472 | 1.16μs | 2.91ns | 10.9ns | 0.128 | 0 | 0 | 810 B |
| #3133 | StartFinishSpan |
netcoreapp3.1 | 917ns | 0.289ns | 1.08ns | 0.0101 | 0 | 0 | 760 B |
| #3133 | StartFinishScope |
net472 | 1.42μs | 0.908ns | 3.52ns | 0.141 | 0 | 0 | 891 B |
| #3133 | StartFinishScope |
netcoreapp3.1 | 1.09μs | 0.293ns | 1.1ns | 0.0119 | 0 | 0 | 880 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | RunOnMethodBegin |
net472 | 1.51μs | 0.417ns | 1.61ns | 0.141 | 0 | 0 | 891 B |
| master | RunOnMethodBegin |
netcoreapp3.1 | 1.15μs | 0.232ns | 0.898ns | 0.0121 | 0 | 0 | 880 B |
| #3133 | RunOnMethodBegin |
net472 | 1.53μs | 0.972ns | 3.77ns | 0.141 | 0 | 0 | 891 B |
| #3133 | RunOnMethodBegin |
netcoreapp3.1 | 1.11μs | 0.423ns | 1.52ns | 0.0117 | 0 | 0 | 880 B |
Code Coverage Report 📊✔️ Merging #3133 into master will not change line coverage
View the full report for further details: Datadog.Trace Breakdown ✔️
The following classes have significant coverage changes.
The following classes were added in #3133:
View the full reports for further details: |
Summary of changes
In #3048, the logic for keeping a trace chunk occurred twice: once in the TraceContext and then again in the StatsAggregator. This PR moves all of the logic into the StatsAggregator.
Reason for change
Simplifies the logic.
Implementation details
Main changes
bool IStatsAggregator.ShouldKeepTrace(ArraySegment<Span> trace)Datadog.Trace.Agent.ITraceSamplerbool Sample(ArraySegment<Span> trace)AnalyticsEventSamplerErrorSamplerPrioritySamplerRareSampler_dd.rare=1Other changes
IStatsAggregator.Add/IStatsAggregator.AddRangefrombooltovoidsince this is no longer making a sampling decisionbool shouldSerializeSpans, which used to be calculated inTraceContextand propagated through to theAgentWriterTraceContextTests, since now theTraceContextshould be unchangedSamplingHelpers.SampleByRateTraceContextTests.SetAASContextever so slightlyTest coverage
Other details
Some thoughts I'd like some feedback on:
IStatsAggregator.RunSamplersmimics the name in the trace agent code, but should we use another name? PerhapsIStatsAggregator.SampleTrace?IStatsAggregatoror in theAgentWriter? We only have to do the sampling when stats computation is enabled so that's why I originally stuck it inIStatsAggregatorITraceSamplerinterface necessary? We could easily in-line all of the logic, but the separation makes it very easy to improve our sampling logic in the future so that we don't send spans that the trace agent will end up dropping.