Add `SpanRateLimiter` for Single Span Ingestion by bouwkast · Pull Request #3282 · DataDog/dd-trace-dotnet

bouwkast · 2022-09-30T12:51:25Z

Summary of changes

Adds a SpanRateLimiter for use with Single Span Ingestion.

Reason for change

Each span sampling rule will have its own rate limit.

Implementation details

Mainly copied the TracerRateLimiter

Test coverage

😬 I didn't add anything to RateLimiterTests, but I definitely can do some refactoring there to get this tested, but wasn't sure if it was needed as it doesn't really do anything different.

Other details

This is for the Single Span Ingestion feature and will have more PRs on top of it (trying to break them up into small as possible).

andrewlock · 2022-09-30T13:09:06Z

+        public override void OnDisallowed(Span span, int count, int intervalMs, int maxTracesPerInterval)
+        {
+            // TODO - copied from "TracerRateLimiter" - should this get logged?
+            Log.Warning<ulong, int, int>("Dropping span id {SpanId} with count of {Count} for last {Interval}ms.", span.SpanId, count, intervalMs);


I don't feel like this should be a warning (I'm not convinced the TraceRateLimiter one should be either), Debug seems like it would good enough to me 🤔

Swapped it over to Debug 7ba0b8e

andrewlock · 2022-10-13T13:27:15Z

Benchmarks Report 🐌

Benchmarks for #3282 compared to master:

All benchmarks have the same speed
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net472	798μs	454ns	1.64μs	0.393	3.22 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	534μs	226ns	846ns	0	2.62 KB
#3282	`WriteAndFlushEnrichedTraces`	net472	799μs	649ns	2.43μs	0.401	3.22 KB
#3282	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	533μs	432ns	1.61μs	0	2.62 KB

Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net472	196ns	0.0691ns	0.249ns	0.0727	0	457 B
master	`AllCycleSimpleBody`	netcoreapp3.1	254ns	0.0646ns	0.242ns	0.00632	0	456 B
master	`AllCycleMoreComplexBody`	net472	194ns	0.102ns	0.353ns	0.0688	0	433 B
master	`AllCycleMoreComplexBody`	netcoreapp3.1	251ns	0.134ns	0.518ns	0.00592	0	432 B
master	`BodyExtractorSimpleBody`	net472	271ns	0.223ns	0.865ns	0.0574	0	361 B
master	`BodyExtractorSimpleBody`	netcoreapp3.1	235ns	0.0611ns	0.22ns	0.0038	0	272 B
master	`BodyExtractorMoreComplexBody`	net472	16.1μs	15.3ns	59.1ns	1.2	0.0161	7.62 KB
master	`BodyExtractorMoreComplexBody`	netcoreapp3.1	13.1μs	3.57ns	13.8ns	0.0918	0	6.75 KB
#3282	`AllCycleSimpleBody`	net472	197ns	0.124ns	0.464ns	0.0727	0	457 B
#3282	`AllCycleSimpleBody`	netcoreapp3.1	255ns	0.116ns	0.435ns	0.00617	0	456 B
#3282	`AllCycleMoreComplexBody`	net472	195ns	0.0331ns	0.119ns	0.0688	0	433 B
#3282	`AllCycleMoreComplexBody`	netcoreapp3.1	251ns	0.124ns	0.428ns	0.00584	0	432 B
#3282	`BodyExtractorSimpleBody`	net472	271ns	0.135ns	0.524ns	0.0573	0	361 B
#3282	`BodyExtractorSimpleBody`	netcoreapp3.1	231ns	0.0778ns	0.301ns	0.00372	0	272 B
#3282	`BodyExtractorMoreComplexBody`	net472	16μs	14.4ns	55.8ns	1.2	0.0159	7.62 KB
#3282	`BodyExtractorMoreComplexBody`	netcoreapp3.1	13.1μs	3.58ns	13.4ns	0.0917	0	6.75 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
master	`SendRequest`	netcoreapp3.1	183μs	251ns	970ns	0.184	20.68 KB
#3282	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#3282	`SendRequest`	netcoreapp3.1	182μs	247ns	925ns	0.272	20.68 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net472	1.8μs	1.21ns	4.67ns	0.155	0.000936	979 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.48μs	1.23ns	4.43ns	0.0134	0	968 B
#3282	`ExecuteNonQuery`	net472	1.82μs	0.652ns	2.44ns	0.155	0.000907	979 B
#3282	`ExecuteNonQuery`	netcoreapp3.1	1.39μs	0.492ns	1.84ns	0.0127	0	968 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net472	2.51μs	0.988ns	3.7ns	0.188	1.19 KB
master	`CallElasticsearch`	netcoreapp3.1	1.62μs	1.16ns	4.35ns	0.0153	1.13 KB
master	`CallElasticsearchAsync`	net472	2.74μs	0.925ns	3.58ns	0.21	1.32 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	1.61μs	0.891ns	3.09ns	0.0169	1.25 KB
#3282	`CallElasticsearch`	net472	2.59μs	0.84ns	3.25ns	0.188	1.19 KB
#3282	`CallElasticsearch`	netcoreapp3.1	1.6μs	0.736ns	2.76ns	0.0152	1.13 KB
#3282	`CallElasticsearchAsync`	net472	2.63μs	0.941ns	3.64ns	0.21	1.32 KB
#3282	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.63ns	2.44ns	0.0171	1.25 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net472	2.66μs	1.71ns	6.39ns	0.23	1.45 KB
master	`ExecuteAsync`	netcoreapp3.1	1.79μs	0.453ns	1.63ns	0.0188	1.38 KB
#3282	`ExecuteAsync`	net472	2.76μs	0.787ns	2.95ns	0.23	1.45 KB
#3282	`ExecuteAsync`	netcoreapp3.1	1.71μs	0.7ns	2.62ns	0.0187	1.38 KB

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net472	6.01μs	1.96ns	7.08ns	0.443	2.8 KB
master	`SendAsync`	netcoreapp3.1	3.71μs	0.801ns	2.89ns	0.0354	2.63 KB
#3282	`SendAsync`	net472	5.88μs	1.78ns	6.43ns	0.442	2.8 KB
#3282	`SendAsync`	netcoreapp3.1	3.72μs	1.14ns	4.4ns	0.0353	2.63 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	3.2μs	1.72ns	6.64ns	0.297	1.88 KB
master	`EnrichedLog`	netcoreapp3.1	2.53μs	1.4ns	4.86ns	0.0252	1.91 KB
#3282	`EnrichedLog`	net472	3.25μs	5.93ns	23ns	0.298	1.88 KB
#3282	`EnrichedLog`	netcoreapp3.1	2.63μs	1.82ns	7.05ns	0.0251	1.91 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	151μs	94.7ns	367ns	0.686	0.229	4.72 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	313ns	1.17μs	0	0	4.55 KB
#3282	`EnrichedLog`	net472	153μs	284ns	1.1μs	0.678	0.226	4.72 KB
#3282	`EnrichedLog`	netcoreapp3.1	121μs	138ns	536ns	0.0604	0	4.55 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	5.86μs	2.32ns	8.99ns	0.579	0.00292	3.65 KB
master	`EnrichedLog`	netcoreapp3.1	4.7μs	1.39ns	5.22ns	0.0534	0	3.98 KB
#3282	`EnrichedLog`	net472	5.82μs	2ns	7.76ns	0.58	0.00293	3.65 KB
#3282	`EnrichedLog`	netcoreapp3.1	4.58μs	1.49ns	5.57ns	0.0527	0	3.98 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net472	2.26μs	1.86ns	7.21ns	0.222	1.4 KB
master	`SendReceive`	netcoreapp3.1	1.83μs	3.8ns	13.7ns	0.0182	1.35 KB
#3282	`SendReceive`	net472	2.32μs	2.75ns	10.6ns	0.222	1.4 KB
#3282	`SendReceive`	netcoreapp3.1	1.76μs	1.03ns	3.99ns	0.0185	1.35 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	5.21μs	2.33ns	9.04ns	0.363	2.3 KB
master	`EnrichedLog`	netcoreapp3.1	4.26μs	1.43ns	5.35ns	0.0236	1.86 KB
#3282	`EnrichedLog`	net472	5.09μs	2.54ns	9.17ns	0.364	2.3 KB
#3282	`EnrichedLog`	netcoreapp3.1	4.28μs	1.58ns	6.11ns	0.0255	1.86 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net472	1.1μs	0.344ns	1.29ns	0.134	842 B
master	`StartFinishSpan`	netcoreapp3.1	972ns	0.271ns	1.05ns	0.0107	792 B
master	`StartFinishScope`	net472	1.34μs	0.393ns	1.47ns	0.146	923 B
master	`StartFinishScope`	netcoreapp3.1	1.09μs	0.451ns	1.69ns	0.0125	912 B
#3282	`StartFinishSpan`	net472	1.17μs	0.448ns	1.68ns	0.133	842 B
#3282	`StartFinishSpan`	netcoreapp3.1	930ns	0.997ns	3.73ns	0.0107	792 B
#3282	`StartFinishScope`	net472	1.35μs	0.58ns	2.25ns	0.146	923 B
#3282	`StartFinishScope`	netcoreapp3.1	1.05μs	0.442ns	1.65ns	0.012	912 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net472	1.48μs	0.626ns	2.34ns	0.147	923 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.16μs	1.12ns	4.03ns	0.0122	912 B
#3282	`RunOnMethodBegin`	net472	1.53μs	0.323ns	1.12ns	0.146	923 B
#3282	`RunOnMethodBegin`	netcoreapp3.1	1.17μs	0.427ns	1.6ns	0.0123	912 B

andrewlock · 2022-10-13T14:45:31Z

Code Coverage Report 📊

✔️ Merging #3282 into master will not change line coverage
✔️ Merging #3282 into master will not change branch coverage
✔️ Merging #3282 into master will will decrease complexity by 143

	master	#3282	Change
Lines	`19500` / `26924`	`19395` / `26727`
Lines %	`72%`	`73%`	`0%` ✔️
Branches	`11588` / `16747`	`11545` / `16629`
Branches %	`69%`	`69%`	`0%` ✔️
Complexity	`18113`	`17970`	`-143` ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

	master	#3282	Change
Lines %	`72%`	`73%`	`0%` ✔️
Branches %	`69%`	`69%`	`0%` ✔️
Complexity	`18113`	`17970`	`-143` ✔️

The following classes have significant coverage changes.

File	Line coverage change	Branch coverage change	Complexity change

The following classes were added in #3282:

File	Line coverage	Branch coverage	Complexity
Datadog.Trace.Sampling.SpanRateLimiter	`0%`	`100%`	`4`

16 classes were removed from Datadog.Trace in #3282

View the full reports for further details:

bouwkast self-assigned this Sep 30, 2022

bouwkast requested a review from a team as a code owner September 30, 2022 12:51

github-actions Bot added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Sep 30, 2022

bouwkast mentioned this pull request Sep 30, 2022

Add SpanSamplingRule for Single Span Ingestion #3283

Merged

1 task

andrewlock approved these changes Sep 30, 2022

View reviewed changes

This comment has been minimized.

Sign in to view

bouwkast mentioned this pull request Sep 30, 2022

Implement the SpanSampler into the Tracer #3286

Merged

2 tasks

pierotibou approved these changes Oct 1, 2022

View reviewed changes

This comment has been minimized.

Sign in to view

zacharycmontoya approved these changes Oct 3, 2022

View reviewed changes

lucaspimentel changed the title ~~Add SpanRateLimiter for Single Span Ingestion~~ Add SpanRateLimiter for Single Span Ingestion Oct 4, 2022

bouwkast added 2 commits October 13, 2022 08:40

Add SpanRateLimiter for Single Span Ingestion

a7e6cea

Swap log statement from Warning to Debug

e17b72b

bouwkast force-pushed the steven/single-span/limiter branch from 7ba0b8e to e17b72b Compare October 13, 2022 12:40

bouwkast merged commit 8993897 into master Oct 13, 2022

bouwkast deleted the steven/single-span/limiter branch October 13, 2022 14:59

github-actions Bot added this to the vNext milestone Oct 13, 2022

Uh oh!

Conversation

bouwkast commented Sep 30, 2022

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

andrewlock Sep 30, 2022

Choose a reason for hiding this comment

Uh oh!

bouwkast Oct 3, 2022

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

andrewlock commented Oct 13, 2022

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

andrewlock commented Oct 13, 2022

Code Coverage Report 📊

Datadog.Trace Breakdown ✔️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants