Skip to content

Added integration tests to verify that the tracer still sends traces if the agent doesn't work properly.#3211

Merged
NachoEchevarria merged 15 commits into
masterfrom
nacho/TracerInitNoAgent
Sep 16, 2022
Merged

Added integration tests to verify that the tracer still sends traces if the agent doesn't work properly.#3211
NachoEchevarria merged 15 commits into
masterfrom
nacho/TracerInitNoAgent

Conversation

@NachoEchevarria

Copy link
Copy Markdown
Collaborator

Summary of changes

Some new integration tests have been added to verify that the tracer still sends traces even if the agent doesn't answer, answers with a significative delay, sends 404 or 500 codes in the response or sends wrong data.

Reason for change

Being able to make sure that the tracer can handle these agent malfunctions.

Implementation details

Test coverage

Other details

@NachoEchevarria
NachoEchevarria requested review from a team as code owners September 13, 2022 09:03
@github-actions github-actions Bot added the area:tests unit tests, integration tests label Sep 13, 2022

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for this. A couple of nits - main thing I wonder is if we should test over UDS and named pipes too (See TransportTests for an example)

Comment thread tracer/test/Datadog.Trace.TestHelpers/MockTracerAgent.cs Outdated
Comment thread tracer/test/Datadog.Trace.TestHelpers/AgentBehaviour.cs Outdated
Comment thread tracer/test/Datadog.Trace.TestHelpers/AgentBehaviour.cs Outdated
@andrewlock

This comment has been minimized.

@OmerRaviv

Copy link
Copy Markdown
Collaborator

Agree with Andrew's point, but otherwise LGTM. Thanks a lot for doing this!


if (behaviour == AgentBehaviour.SlowAnswer)
{
System.Threading.Thread.Sleep(10000);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider using a higher value than 10 seconds here, so that it would exceed the timeout, just to verify that scenario works OK?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I will set it to 120 secs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about the test case, I think that a big delay would be equivalent to the no_answer case, what do you think?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not exactly sure, I assumed that there is a difference in that the "No Answer" case would return immediately with an error such as "No connection could be made because the target machine actively refused it", whereas the "Slow Answer" would keep the request going until the timeout occurs. Does that make sense to you?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, the "no answer" case will keep the connection open until timeout (I named it "no aswer" because it keeps the connection open but never really answers). The 404 and 500 cases return error codes. The slowAnswer case answers with a big delay, which also causes a timeout, but in real time, when the agent answers with a big delay, the tracer already has sent the spans and the sample program has already exited, so we would be testing the same case as "no answer". I think that it makes sense to will keep only one timeout case.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, got it! So yes, perhaps it makes sense to keep just one of them, since in practice they are equivalent. @andrewlock WDYT?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree.

The "no answer" case is really a special case, where we can't make a connection to the agent (i.e. there's no agent listening on that port etc). We probably still want to test that (including for named pipes + system tests), but it's fundamentally different to these tests, as we won't get traces then either. For that case we just want to confirm that the app can run without crashing, and exits in a timely matter.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the "no answer" case we still receive the traces in the test since the tracer seems to send them even if the agent does not answer.

If you think that the PR is ready, please approve and I will merge it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the "no answer" case we still receive the traces in the test since the tracer seems to send them even if the agent does not answer.

I think we need to:

  • Remove either the "no answer" case or the timeout case (as they're functionally the same), which you've already done ✅
  • Create a new test, that doesn't set up a mock agent, so the app tries to send to a non-existent agent. Confirm that the app exists quickly (normally) and there's no crash.

That second one could be a separate PR if you prefer 🙂

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, if it is ok, I would prefer to implement this test in a separate PR.

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

Comment thread tracer/test/Datadog.Trace.TestHelpers/MockTracerAgent.cs Outdated
Comment thread tracer/test/Datadog.Trace.TestHelpers/MockTracerAgent.cs
Comment thread tracer/test/Datadog.Trace.TestHelpers/MockTracerAgent.cs Outdated
await stream.WriteAsync(GetResponseBytes(body: mockTracerResponse.Response));
}

handler.Shutdown(SocketShutdown.Both);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically I think this closes the handler, so this is a different test from the "no answer" one in TCP (premature disconnect vs timeout) but I don't know if that matters too much

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Comment thread tracer/test/Datadog.Trace.TestHelpers/MockTracerAgent.cs Outdated

if (behaviour == AgentBehaviour.SlowAnswer)
{
System.Threading.Thread.Sleep(10000);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the "no answer" case we still receive the traces in the test since the tracer seems to send them even if the agent does not answer.

I think we need to:

  • Remove either the "no answer" case or the timeout case (as they're functionally the same), which you've already done ✅
  • Create a new test, that doesn't set up a mock agent, so the app tries to send to a non-existent agent. Confirm that the app exists quickly (normally) and there's no crash.

That second one could be a separate PR if you prefer 🙂

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

Comment thread tracer/test/Datadog.Trace.TestHelpers/MockTracerAgent.cs Outdated

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks 👍

@NachoEchevarria

Copy link
Copy Markdown
Collaborator Author

LGTM, thanks 👍

Thanks to you for your support and advices!!!

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

@andrewlock

This comment has been minimized.

@andrewlock

Copy link
Copy Markdown
Member

Benchmarks Report 🐌

Benchmarks for #3211 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 719μs 769ns 2.98μs 0.359 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 453μs 174ns 675ns 0 0 0 2.59 KB
#3211 WriteAndFlushEnrichedTraces net472 722μs 698ns 2.61μs 0.357 0 0 3.18 KB
#3211 WriteAndFlushEnrichedTraces netcoreapp3.1 465μs 160ns 618ns 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 182ns 0.148ns 0.534ns 0.0675 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 238ns 0.252ns 0.978ns 0.00588 0 0 424 B
master AllCycleMoreComplexBody net472 184ns 0.213ns 0.767ns 0.0638 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 235ns 0.214ns 0.771ns 0.00552 0 0 400 B
master BodyExtractorSimpleBody net472 257ns 0.244ns 0.944ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 231ns 0.263ns 1.02ns 0.00369 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.6μs 12.7ns 49.2ns 1.21 0.0218 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 13.2ns 49.3ns 0.0892 0 0 6.75 KB
#3211 AllCycleSimpleBody net472 182ns 0.138ns 0.515ns 0.0676 0 0 425 B
#3211 AllCycleSimpleBody netcoreapp3.1 235ns 0.338ns 1.31ns 0.00565 0 0 424 B
#3211 AllCycleMoreComplexBody net472 180ns 0.203ns 0.786ns 0.0637 0 0 401 B
#3211 AllCycleMoreComplexBody netcoreapp3.1 234ns 0.218ns 0.815ns 0.00544 0 0 400 B
#3211 BodyExtractorSimpleBody net472 267ns 0.339ns 1.27ns 0.0573 0 0 361 B
#3211 BodyExtractorSimpleBody netcoreapp3.1 213ns 0.13ns 0.468ns 0.00366 0 0 272 B
#3211 BodyExtractorMoreComplexBody net472 14.3μs 7.08ns 26.5ns 1.21 0.0214 0 7.62 KB
#3211 BodyExtractorMoreComplexBody netcoreapp3.1 11.7μs 10.2ns 39.4ns 0.0878 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 180μs 130ns 487ns 0.269 0 0 20.6 KB
#3211 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3211 SendRequest netcoreapp3.1 184μs 225ns 871ns 0.183 0 0 20.6 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.86μs 0.747ns 2.79ns 0.15 0.000937 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.45μs 0.863ns 3.34ns 0.0123 0 0 936 B
#3211 ExecuteNonQuery net472 1.9μs 2.72ns 10.2ns 0.15 0 0 947 B
#3211 ExecuteNonQuery netcoreapp3.1 1.44μs 0.714ns 2.67ns 0.0122 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.55μs 1.8ns 6.96ns 0.183 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.48μs 0.907ns 3.51ns 0.0147 0 0 1.1 KB
master CallElasticsearchAsync net472 2.66μs 1.87ns 7.23ns 0.204 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.64μs 0.451ns 1.75ns 0.0164 0 0 1.22 KB
#3211 CallElasticsearch net472 2.56μs 1.28ns 4.97ns 0.183 0 0 1.16 KB
#3211 CallElasticsearch netcoreapp3.1 1.53μs 0.596ns 2.31ns 0.0145 0 0 1.1 KB
#3211 CallElasticsearchAsync net472 2.72μs 2.79ns 10.8ns 0.204 0 0 1.29 KB
#3211 CallElasticsearchAsync netcoreapp3.1 1.69μs 0.757ns 2.93ns 0.016 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.73μs 3ns 10.8ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.69μs 3.84ns 14.9ns 0.0183 0 0 1.34 KB
#3211 ExecuteAsync net472 2.72μs 6.93ns 26.8ns 0.225 0 0 1.42 KB
#3211 ExecuteAsync netcoreapp3.1 1.69μs 4.05ns 15.7ns 0.0184 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.69μs 10.8ns 42ns 0.439 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.54μs 6.96ns 27ns 0.0353 0 0 2.6 KB
#3211 SendAsync net472 5.71μs 13.2ns 51.2ns 0.437 0 0 2.77 KB
#3211 SendAsync netcoreapp3.1 3.69μs 7.67ns 28.7ns 0.0344 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.17μs 2.57ns 9.95ns 0.288 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.55μs 1.8ns 6.72ns 0.0255 0 0 1.85 KB
#3211 EnrichedLog net472 3.19μs 4.06ns 15.7ns 0.288 0 0 1.81 KB
#3211 EnrichedLog netcoreapp3.1 2.54μs 1.17ns 4.38ns 0.0253 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 150μs 185ns 717ns 0.674 0.225 0 4.65 KB
master EnrichedLog netcoreapp3.1 119μs 156ns 604ns 0 0 0 4.49 KB
#3211 EnrichedLog net472 151μs 111ns 414ns 0.684 0.228 0 4.66 KB
#3211 EnrichedLog netcoreapp3.1 116μs 138ns 517ns 0.0577 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.82μs 9.81ns 38ns 0.568 0.00288 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.31μs 8.01ns 28.9ns 0.0542 0 0 3.91 KB
#3211 EnrichedLog net472 5.6μs 10.7ns 41.5ns 0.57 0.00282 0 3.59 KB
#3211 EnrichedLog netcoreapp3.1 4.35μs 10.6ns 40.9ns 0.0535 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.18μs 2.84ns 10.6ns 0.217 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.8μs 0.558ns 2.09ns 0.018 0 0 1.32 KB
#3211 SendReceive net472 2.32μs 1.77ns 6.84ns 0.218 0 0 1.37 KB
#3211 SendReceive netcoreapp3.1 1.85μs 0.718ns 2.69ns 0.0176 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.02μs 2.8ns 10.5ns 0.353 0.00249 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.27μs 3.15ns 12.2ns 0.0253 0 0 1.8 KB
#3211 EnrichedLog net472 5.04μs 4.4ns 17ns 0.352 0.00252 0 2.23 KB
#3211 EnrichedLog netcoreapp3.1 4.26μs 1.97ns 7.61ns 0.0234 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.09μs 0.672ns 2.6ns 0.129 0 0 810 B
master StartFinishSpan netcoreapp3.1 924ns 0.357ns 1.38ns 0.0102 0 0 760 B
master StartFinishScope net472 1.48μs 0.863ns 3.34ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.14μs 1.12ns 4.33ns 0.0118 0 0 880 B
#3211 StartFinishSpan net472 1.15μs 0.493ns 1.78ns 0.129 0 0 810 B
#3211 StartFinishSpan netcoreapp3.1 954ns 0.261ns 0.976ns 0.01 0 0 760 B
#3211 StartFinishScope net472 1.39μs 1.16ns 4.49ns 0.141 0 0 891 B
#3211 StartFinishScope netcoreapp3.1 1.07μs 0.738ns 2.86ns 0.0118 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.42μs 1.23ns 4.75ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.16μs 0.289ns 1.08ns 0.0121 0 0 880 B
#3211 RunOnMethodBegin net472 1.53μs 0.725ns 2.71ns 0.141 0 0 891 B
#3211 RunOnMethodBegin netcoreapp3.1 1.22μs 0.584ns 2.1ns 0.0122 0 0 880 B

@andrewlock

Copy link
Copy Markdown
Member

Code Coverage Report 📊

✔️ Merging #3211 into master will not change line coverage
✔️ Merging #3211 into master will not change branch coverage
✔️ Merging #3211 into master will not change complexity

master #3211 Change
Lines 17967 / 24564 17999 / 24564
Lines % 73% 73% 0% ✔️
Branches 10574 / 15054 10595 / 15054
Branches % 70% 70% 0% ✔️
Complexity 16280 16280 0 ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3211 Change
Lines % 73% 73% 0% ✔️
Branches % 70% 70% 0% ✔️
Complexity 16280 16280 0 ✔️

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Ci.CIVisibility 5% ✔️ 6% ✔️ 0 ✔️

View the full reports for further details:

@NachoEchevarria
NachoEchevarria merged commit 8417653 into master Sep 16, 2022
@NachoEchevarria
NachoEchevarria deleted the nacho/TracerInitNoAgent branch September 16, 2022 10:00
@github-actions github-actions Bot added this to the vNext milestone Sep 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tests unit tests, integration tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants