[wip] track sampling mechanism and sampling rate by lucaspimentel · Pull Request #2459 · DataDog/dd-trace-dotnet

lucaspimentel · 2022-02-17T14:10:43Z

Summary of changes

In this PR, we start tracking the sampling mechanism and sampling rate in addition to the sampling priority. In another PR, we will start propagating this information both vertically (as tags sent to the Agent) and horizontally (as upstream/downstream headers).

Reason for change

Implementation details

We introduce the SamplingDecision, which includes all the sampling information we need to track (priority, mechanism, and rate if used):

struct SamplingDecision
{
   int Priority;
   int Mechanism;
   double? Rate;
}

In most places where we were using SamplingPriority, we now use SamplingDecision instead.

Test coverage

Other details

Note that while we now track sampling mechanism and rate, we're still not using this additional information in new ways. My next PR will combine this work and #2432 to add the new _dd.p.upstream_services trace-level tag. The first two PRs were independent of each other and could be merged in any order.

TODO:

propagate sampling decision to "distributed tracer" (cross-version)

lucaspimentel · 2022-02-18T17:33:10Z


                        Span span = tracer.StartSpan("crank.test", startTime: minTimeStamp);

-                        span.SetTraceSamplingPriority(SamplingPriorityValues.AutoKeep);


What was the goal here by setting this? To force all the traces to be sampled?

lucaspimentel · 2022-02-18T17:33:26Z

                _buildSpan = _tracer.StartSpan(BuildTags.BuildOperationName);
                _buildSpan.SetMetric(Tags.Analytics, 1.0d);
-                _buildSpan.SetTraceSamplingPriority(SamplingPriorityValues.AutoKeep);
+                _buildSpan.SetTraceSamplingDecision(SamplingPriorityValues.AutoKeep, SamplingMechanism.CiApp);


@tonyredondo: What was the goal here by setting this? To force all the traces to be sampled?

lucaspimentel · 2022-02-18T17:34:43Z

+                        if (setSamplingPriority)
                        {
-                            scope.Span.SetTraceSamplingPriority(existingSpanContext.SamplingPriority.Value);
+                            // TODO: figure out SamplingMechanism, do we propagate that as well for special cases like this?


TODO: figure out SamplingMechanism, do we propagate that as well for special cases like this?

lucaspimentel · 2022-02-18T17:34:57Z

                        if (setSamplingPriority && spanContext?.SamplingPriority is not null)
                        {
-                            scope.Span.SetTraceSamplingPriority(spanContext.SamplingPriority.Value);
+                            // TODO: figure out SamplingMechanism, do we propagate that as well for special cases like this?


TODO: figure out SamplingMechanism, do we propagate that as well for special cases like this?

lucaspimentel · 2022-02-18T17:50:00Z

+        /// Used to propagate sampling mechanism across tracer versions in the same process,
+        /// not for propagation across process boundaries.
+        /// </summary>
+        internal const string SamplingMechanism = "x-datadog-sampling-mechanism";


I think we should come up with special internal keys for internal propagation. The keys used in headers need to be switchable/customizable (e.g. Datadog headers vs B3 headers), but these are no included there.

andrewlock · 2022-02-18T19:02:48Z

Code Coverage Report 📊

✔️ Merging #2459 into master will not change line coverage
✔️ Merging #2459 into master will not change branch coverage
⛔ Merging #2459 into master will will increase complexity by 47

	master	#2459	Change
Lines	`11610` / `15971`	`11657` / `16012`
Lines %	`73%`	`73%`	`0%` ✔️
Branches	`6235` / `9336`	`6276` / `9376`
Branches %	`67%`	`67%`	`0%` ✔️
Complexity	`10546`	`10593`	`47` ⛔

View the full report for further details:

Datadog.Trace Breakdown ✔️

	master	#2459	Change
Lines %	`73%`	`73%`	`0%` ✔️
Branches %	`67%`	`67%`	`0%` ✔️
Complexity	`10546`	`10593`	`47` ⛔

The following classes have significant coverage changes.

File	Line coverage change	Branch coverage change	Complexity change
Datadog.Trace.ClrProfiler.AutoInstrumentation.MongoDb.MongoDbIntegration	`-7%` ⛔	`0%` ✔️	`0` ✔️
Datadog.Trace.Ci.Agent.CIAgentWriter	`0%` ✔️	`-21%` ⛔	`0` ✔️
Datadog.Trace.ClrProfiler.CommonTracer	`0%` ✔️	`-11%` ⛔	`2` ⛔
Datadog.Trace.SpanContext	`3%` ✔️	`-7%` ⛔	`32` ⛔
Datadog.Trace.Telemetry.JsonWebRequestTelemetryTransport	`28%` ✔️	`57%` ✔️	`0` ✔️

The following classes were added in #2459:

File	Line coverage	Branch coverage	Complexity
Datadog.Trace.Sampling.SamplingDecision	`67%`	`100%`	`5`

View the full reports for further details:

andrewlock · 2022-02-18T23:34:02Z

Benchmarks Report 🐌

Benchmarks for #2459 compared to master:

2 benchmarks are slower, with geometric mean 1.184
24 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net472	986μs	7.17μs	67.3μs	3.16 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	755μs	3.72μs	23.3μs	2.57 KB
#2459	`WriteAndFlushEnrichedTraces`	net472	848μs	4.01μs	15.5μs	3.16 KB
#2459	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	2.5μs	9.7μs	2.57 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
master	`SendRequest`	netcoreapp3.1	310μs	1.07μs	3.99μs	0.154	19.71 KB
#2459	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#2459	`SendRequest`	netcoreapp3.1	298μs	1.41μs	5.96μs	0.147	19.74 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net472	586 B	610 B	24 B	4.10%
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑netcoreapp3.1	624 B	648 B	24 B	3.85%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net472	1.81μs	10ns	60.8ns	0.0929	0.000894	586 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.52μs	7.44ns	41.4ns	0.00838	0	624 B
#2459	`ExecuteNonQuery`	net472	1.74μs	5.72ns	20.6ns	0.0961	0.000866	610 B
#2459	`ExecuteNonQuery`	netcoreapp3.1	1.57μs	6.19ns	24ns	0.00924	0	648 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net472	794 B	819 B	25 B	3.15%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑netcoreapp3.1	784 B	808 B	24 B	3.06%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑netcoreapp3.1	904 B	928 B	24 B	2.65%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net472	931 B	955 B	24 B	2.58%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`CallElasticsearch`	net472	2.8μs	15.3ns	83.9ns	0.125	0	0	794 B
master	`CallElasticsearch`	netcoreapp3.1	1.77μs	2.69ns	9.7ns	0.0108	0	0	784 B
master	`CallElasticsearchAsync`	net472	2.95μs	9.33ns	34.9ns	0.148	0.00298	0.00149	931 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.9μs	8.99ns	35.9ns	0.0127	0	0	904 B
#2459	`CallElasticsearch`	net472	2.76μs	14.6ns	74.4ns	0.129	0	0	819 B
#2459	`CallElasticsearch`	netcoreapp3.1	1.81μs	10.4ns	74ns	0.0109	0	0	808 B
#2459	`CallElasticsearchAsync`	net472	2.77μs	9.94ns	38.5ns	0.151	0	0	955 B
#2459	`CallElasticsearchAsync`	netcoreapp3.1	2μs	10.9ns	60.8ns	0.0135	0	0	928 B

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑netcoreapp3.1	1.02 KB	1.05 KB	24 B	2.34%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net472	1.05 KB	1.08 KB	24 B	2.28%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteAsync`	net472	3.37μs	15.7ns	60.8ns	0.166	0.00158	1.05 KB
master	`ExecuteAsync`	netcoreapp3.1	2.19μs	7.39ns	28.6ns	0.014	0	1.02 KB
#2459	`ExecuteAsync`	net472	3.13μs	8.61ns	32.2ns	0.17	0.00156	1.08 KB
#2459	`ExecuteAsync`	netcoreapp3.1	2.11μs	10.6ns	47.5ns	0.0145	0	1.05 KB

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.HttpClientBenchmark.SendAsync‑netcoreapp3.1	2.2 KB	2.22 KB	24 B	1.09%
Benchmarks.Trace.HttpClientBenchmark.SendAsync‑net472	2.3 KB	2.33 KB	24 B	1.04%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net472	7.38μs	15.2ns	56.7ns	0.363	2.3 KB
master	`SendAsync`	netcoreapp3.1	5.02μs	11.6ns	43.5ns	0.0296	2.2 KB
#2459	`SendAsync`	net472	7μs	32.2ns	125ns	0.367	2.33 KB
#2459	`SendAsync`	netcoreapp3.1	5.06μs	24.5ns	97.8ns	0.0323	2.22 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net472	1.44 KB	1.46 KB	24 B	1.67%
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑netcoreapp3.1	1.52 KB	1.54 KB	24 B	1.58%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	3.34μs	12.9ns	48.3ns	0.225	1.44 KB
master	`EnrichedLog`	netcoreapp3.1	3.2μs	7.81ns	30.2ns	0.0208	1.52 KB
#2459	`EnrichedLog`	net472	3.42μs	17.8ns	85.6ns	0.228	1.46 KB
#2459	`EnrichedLog`	netcoreapp3.1	3.27μs	13.5ns	53.9ns	0.0212	1.54 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Log4netBenchmark.EnrichedLog‑netcoreapp3.1	4.19 KB	4.22 KB	25 B	0.60%
Benchmarks.Trace.Log4netBenchmark.EnrichedLog‑net472	4.31 KB	4.33 KB	23 B	0.53%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	294μs	859ns	3.33μs	0.447	0.149	4.31 KB
master	`EnrichedLog`	netcoreapp3.1	239μs	1.16μs	4.93μs	0	0	4.19 KB
#2459	`EnrichedLog`	net472	289μs	1.26μs	4.71μs	0.441	0.147	4.33 KB
#2459	`EnrichedLog`	netcoreapp3.1	231μs	1.12μs	4.47μs	0	0	4.22 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.NLogBenchmark.EnrichedLog‑net472	3.21 KB	3.23 KB	24 B	0.75%
Benchmarks.Trace.NLogBenchmark.EnrichedLog‑netcoreapp3.1	3.58 KB	3.61 KB	24 B	0.67%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	7.66μs	37.1ns	148ns	0.502	3.21 KB
master	`EnrichedLog`	netcoreapp3.1	6.61μs	36.8ns	224ns	0.0495	3.58 KB
#2459	`EnrichedLog`	net472	7.27μs	34.5ns	134ns	0.505	3.23 KB
#2459	`EnrichedLog`	netcoreapp3.1	6.71μs	33.7ns	143ns	0.0495	3.61 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.RedisBenchmark.SendReceive‑netcoreapp3.1	1 KB	1.02 KB	24 B	2.40%
Benchmarks.Trace.RedisBenchmark.SendReceive‑net472	1 KB	1.03 KB	24 B	2.39%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net472	2.49μs	10.4ns	39ns	0.159	0.00128	1 KB
master	`SendReceive`	netcoreapp3.1	2.1μs	9.7ns	37.6ns	0.0135	0	1 KB
#2459	`SendReceive`	net472	2.44μs	13.8ns	96.4ns	0.161	0	1.03 KB
#2459	`SendReceive`	netcoreapp3.1	2.13μs	9.99ns	39.9ns	0.0135	0	1.02 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.SerilogBenchmark.EnrichedLog‑netcoreapp3.1	1.47 KB	1.5 KB	24 B	1.63%
Benchmarks.Trace.SerilogBenchmark.EnrichedLog‑net472	1.85 KB	1.88 KB	25 B	1.35%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	6.01μs	23.4ns	90.5ns	0.29	1.85 KB
master	`EnrichedLog`	netcoreapp3.1	5.24μs	25.8ns	139ns	0.0202	1.47 KB
#2459	`EnrichedLog`	net472	6.18μs	33.4ns	186ns	0.294	1.88 KB
#2459	`EnrichedLog`	netcoreapp3.1	5.44μs	25.1ns	100ns	0.0219	1.5 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #2459

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.205	898.97	1,083.68
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.164	873.66	1,016.61

More allocations ⚠️ in #2459

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	448 B	472 B	24 B	5.36%
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	449 B	473 B	24 B	5.35%
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	530 B	554 B	24 B	4.53%
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	568 B	592 B	24 B	4.23%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net472	901ns	4.33ns	16.8ns	0.0706	449 B
master	`StartFinishSpan`	netcoreapp3.1	876ns	2.4ns	9.29ns	0.00625	448 B
master	`StartFinishScope`	net472	1.09μs	5.59ns	26.8ns	0.0829	530 B
master	`StartFinishScope`	netcoreapp3.1	1.07μs	2.99ns	11.2ns	0.00801	568 B
#2459	`StartFinishSpan`	net472	1.09μs	5.25ns	21.7ns	0.0741	473 B
#2459	`StartFinishSpan`	netcoreapp3.1	1.02μs	3.2ns	12ns	0.00678	472 B
#2459	`StartFinishScope`	net472	1.19μs	6.05ns	27.7ns	0.0869	554 B
#2459	`StartFinishScope`	netcoreapp3.1	1.14μs	3.94ns	14.7ns	0.00789	592 B

lucaspimentel · 2022-03-28T21:53:14Z

Closing this PR because we are re-thinking how to handle this feature.

lucaspimentel added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) type:new-feature status:work-in-progress Actively worked on. If this is a PR, no review needed yet. WIP. labels Feb 17, 2022

lucaspimentel force-pushed the lpimentel/sampling-decision branch 5 times, most recently from 82f845d to 4873317 Compare February 17, 2022 22:21

This comment has been minimized.

Sign in to view

lucaspimentel force-pushed the lpimentel/sampling-decision branch 4 times, most recently from dae8a5f to 500fef1 Compare February 18, 2022 17:32

lucaspimentel commented Feb 18, 2022

View reviewed changes

lucaspimentel force-pushed the lpimentel/sampling-decision branch 2 times, most recently from 8a814aa to 4035890 Compare February 18, 2022 17:48

lucaspimentel commented Feb 18, 2022

View reviewed changes

lucaspimentel force-pushed the lpimentel/sampling-decision branch 4 times, most recently from ebceaa5 to af775d5 Compare February 24, 2022 20:05

lucaspimentel force-pushed the lpimentel/sampling-decision branch from af775d5 to 77bbbbc Compare March 3, 2022 17:45

lucaspimentel force-pushed the lpimentel/sampling-decision branch from 77bbbbc to af775d5 Compare March 25, 2022 17:00

lucaspimentel added 2 commits March 25, 2022 13:01

add SamplingMechanism and SamplingDecision

4de9c5c

update sampling rules to return sampling decisions

56008df

lucaspimentel added 3 commits March 25, 2022 13:01

store the full sampling decision in TraceContext

4843af5

switch to sampling decision

422ddb8

remove unused APIs [skip ci]

be18e60

lucaspimentel force-pushed the lpimentel/sampling-decision branch from af775d5 to be18e60 Compare March 25, 2022 17:13

lucaspimentel closed this Mar 28, 2022

lucaspimentel removed the status:work-in-progress Actively worked on. If this is a PR, no review needed yet. WIP. label Jul 14, 2022

lucaspimentel mentioned this pull request Jul 14, 2022

[Tracer] Tracking sampling mechanism with _dd.p.dm tag #2982

Merged

lucaspimentel deleted the lpimentel/sampling-decision branch August 23, 2022 14:08


		Span span = tracer.StartSpan("crank.test", startTime: minTimeStamp);

		span.SetTraceSamplingPriority(SamplingPriorityValues.AutoKeep);

Uh oh!

Conversation

lucaspimentel commented Feb 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

This comment has been minimized.

lucaspimentel Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

lucaspimentel Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

lucaspimentel Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

lucaspimentel Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

lucaspimentel Feb 18, 2022

Choose a reason for hiding this comment

Uh oh!

andrewlock commented Feb 18, 2022

Code Coverage Report 📊

Datadog.Trace Breakdown ✔️

Uh oh!

andrewlock commented Feb 18, 2022

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

More allocations ⚠️ in #2459

Raw results

Slower ⚠️ in #2459

More allocations ⚠️ in #2459

Raw results

Uh oh!

lucaspimentel commented Mar 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lucaspimentel commented Feb 17, 2022 •

edited

Loading