Added integration tests to verify that the tracer still sends traces if the agent doesn't work properly.#3211
Conversation
andrewlock
left a comment
There was a problem hiding this comment.
LGTM, thanks for this. A couple of nits - main thing I wonder is if we should test over UDS and named pipes too (See TransportTests for an example)
This comment has been minimized.
This comment has been minimized.
|
Agree with Andrew's point, but otherwise LGTM. Thanks a lot for doing this! |
|
|
||
| if (behaviour == AgentBehaviour.SlowAnswer) | ||
| { | ||
| System.Threading.Thread.Sleep(10000); |
There was a problem hiding this comment.
Should we consider using a higher value than 10 seconds here, so that it would exceed the timeout, just to verify that scenario works OK?
There was a problem hiding this comment.
Sure. I will set it to 120 secs.
There was a problem hiding this comment.
Thinking about the test case, I think that a big delay would be equivalent to the no_answer case, what do you think?
There was a problem hiding this comment.
Not exactly sure, I assumed that there is a difference in that the "No Answer" case would return immediately with an error such as "No connection could be made because the target machine actively refused it", whereas the "Slow Answer" would keep the request going until the timeout occurs. Does that make sense to you?
There was a problem hiding this comment.
Right now, the "no answer" case will keep the connection open until timeout (I named it "no aswer" because it keeps the connection open but never really answers). The 404 and 500 cases return error codes. The slowAnswer case answers with a big delay, which also causes a timeout, but in real time, when the agent answers with a big delay, the tracer already has sent the spans and the sample program has already exited, so we would be testing the same case as "no answer". I think that it makes sense to will keep only one timeout case.
There was a problem hiding this comment.
Ah, got it! So yes, perhaps it makes sense to keep just one of them, since in practice they are equivalent. @andrewlock WDYT?
There was a problem hiding this comment.
Yes, I agree.
The "no answer" case is really a special case, where we can't make a connection to the agent (i.e. there's no agent listening on that port etc). We probably still want to test that (including for named pipes + system tests), but it's fundamentally different to these tests, as we won't get traces then either. For that case we just want to confirm that the app can run without crashing, and exits in a timely matter.
There was a problem hiding this comment.
In the "no answer" case we still receive the traces in the test since the tracer seems to send them even if the agent does not answer.
If you think that the PR is ready, please approve and I will merge it.
There was a problem hiding this comment.
In the "no answer" case we still receive the traces in the test since the tracer seems to send them even if the agent does not answer.
I think we need to:
- Remove either the "no answer" case or the timeout case (as they're functionally the same), which you've already done ✅
- Create a new test, that doesn't set up a mock agent, so the app tries to send to a non-existent agent. Confirm that the app exists quickly (normally) and there's no crash.
That second one could be a separate PR if you prefer 🙂
There was a problem hiding this comment.
Yes, if it is ok, I would prefer to implement this test in a separate PR.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| await stream.WriteAsync(GetResponseBytes(body: mockTracerResponse.Response)); | ||
| } | ||
|
|
||
| handler.Shutdown(SocketShutdown.Both); |
There was a problem hiding this comment.
Technically I think this closes the handler, so this is a different test from the "no answer" one in TCP (premature disconnect vs timeout) but I don't know if that matters too much
|
|
||
| if (behaviour == AgentBehaviour.SlowAnswer) | ||
| { | ||
| System.Threading.Thread.Sleep(10000); |
There was a problem hiding this comment.
In the "no answer" case we still receive the traces in the test since the tracer seems to send them even if the agent does not answer.
I think we need to:
- Remove either the "no answer" case or the timeout case (as they're functionally the same), which you've already done ✅
- Create a new test, that doesn't set up a mock agent, so the app tries to send to a non-existent agent. Confirm that the app exists quickly (normally) and there's no crash.
That second one could be a separate PR if you prefer 🙂
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Thanks to you for your support and advices!!! |
This comment has been minimized.
This comment has been minimized.
158cbde to
b36d067
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Co-authored-by: Andrew Lock <[email protected]>
…tMalfunctionTests/AgentMalfunctionTests.cs Co-authored-by: Andrew Lock <[email protected]>
Co-authored-by: Andrew Lock <[email protected]>
Co-authored-by: Andrew Lock <[email protected]>
b36d067 to
5165047
Compare
Benchmarks Report 🐌Benchmarks for #3211 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️Raw results
|
Code Coverage Report 📊✔️ Merging #3211 into master will not change line coverage
View the full report for further details: Datadog.Trace Breakdown ✔️
The following classes have significant coverage changes.
View the full reports for further details: |
Summary of changes
Some new integration tests have been added to verify that the tracer still sends traces even if the agent doesn't answer, answers with a significative delay, sends 404 or 500 codes in the response or sends wrong data.
Reason for change
Being able to make sure that the tracer can handle these agent malfunctions.
Implementation details
Test coverage
Other details