[Exception Replay] Added test suite for ASP.NET Core by GreenMatan · Pull Request #5821 · DataDog/dd-trace-dotnet

GreenMatan · 2024-07-29T13:15:41Z

Summary of changes

Added the infrastructure of integration testing for Exception Replay in ASP.NET Core.

To add a test, one needs to simply create a class that inherits from IRun / IAsyncRun and decorated with ExceptionReplayTestData attribute, providing to the attribute the amount of snapshots anticipated both in default configuration and full stack trace capture configuration.

This PR also introduces the renaming of the configs of Exception Replay from Exception Debugging. In the early days we used to call Exception Replay internally Exception Debugging, thus some of the configs held this name. This PR renamed those leftovers to Exception Replay, including the enablement config of DD_EXCEPTION_REPLAY_ENABLED. Introduced a PR to dd-go too where these configs were added to the config_norm_rules.json file.

Reason for change

Laying the groundwork for integration testing of Exception Replay.

Implementation details

The testing infra bootstraps the web app, queries via reflection all the tests (classes inheriting from IRun/IAsyncRun and decorated with ExceptionReplayTestData attribute) and list them. Upon executing a test, a request will hit an endpoint of RunTest with the name of the test to run (FQN of the class name). The crafted test must throw an exception to be considered valid.
For a more complex test scenarios, more endpoints could be introduced.

Every test runs in 4 flavors:

With Exception Replay while Dynamic Instrumentation is enabled, default stack trace capturing limit.
With Exception Replay while Dynamic Instrumentation is enabled, full stack trace capturing limit.
With Exception Replay while Dynamic Instrumentation is disabled, default stack trace capturing limit.
With Exception Replay while Dynamic Instrumentation is disabled, full stack trace capturing limit.

Test coverage

This whole PR 🤓. I've added two tests that utilize the infra:

ExceptionWithNonSupportedFramesTest
RecursiveExceptionTest

Other details

Fixes DEBUG-2681, DEBUG-2732, DEBUG-1642.

datadog-ddstaging · 2024-07-29T13:39:13Z

Datadog Report

Branch report: matang/exception-replay-testing-suite
Commit report: fc591b4
Test service: dd-trace-dotnet

❌ 1 Failed (1 Known Flaky), 80346 Passed, 229 Skipped, 54m 19.22s Total Time

❌ Failed Tests (1)

Profiler_liveheap_cpu_walltime - scenarios - ❄️ Known flaky - Details

Expand for error

 
  ########### Finishing run at 8/30/2024 12:57:27 PM169eak 25 times with 1 thread(s).shared\bin\monitoring-home\win-x86\Datadog.Profiler.Native.dlltation 5=compute fibonacci 6=n sleeping threads 7=async calls 8=iterator calls 9=allocate array of Generic<int>> 10=threads competing for a lock 11=lunix signal handler 12=trigger garbage collections 13=memory leak 14=short lived threads] [--param <any number to pass to the scenario - used for contention duration for example>] [--timeout <duration in seconds> | --run-infinitely]uteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
    at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
    at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
    at System.Threading.Thread.StartCallback()

andrewlock · 2024-07-29T14:03:52Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

andrewlock · 2024-07-29T15:38:18Z

Benchmarks Report for tracer 🐌

Benchmarks for #5821 compared to master:

4 benchmarks are faster, with geometric mean 1.182
2 benchmarks are slower, with geometric mean 1.131
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.77μs	43.4ns	282ns	0.0153	0.00763	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	52.4ns	273ns	0.0198	0.00989	0	5.62 KB
master	`StartStopWithChild`	net472	16.1μs	45.3ns	175ns	1.03	0.326	0.103	6.06 KB
#5821	`StartStopWithChild`	net6.0	7.8μs	44.6ns	322ns	0.0194	0.00777	0	5.43 KB
#5821	`StartStopWithChild`	netcoreapp3.1	9.66μs	54.6ns	366ns	0.0243	0.00971	0	5.61 KB
#5821	`StartStopWithChild`	net472	15.9μs	32.4ns	125ns	1.02	0.303	0.0931	6.05 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	475μs	199ns	771ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	624μs	365ns	1.42μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	851μs	488ns	1.83μs	0.425	3.3 KB
#5821	`WriteAndFlushEnrichedTraces`	net6.0	485μs	407ns	1.58μs	0	2.7 KB
#5821	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	652μs	447ns	1.73μs	0	2.7 KB
#5821	`WriteAndFlushEnrichedTraces`	net472	836μs	552ns	2.07μs	0.417	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	186μs	1.07μs	7.68μs	0.183	18.45 KB
master	`SendRequest`	netcoreapp3.1	213μs	1.24μs	9.96μs	0.201	20.61 KB
master	`SendRequest`	net472	0.00408ns	0.000941ns	0.00352ns	0	0 b
#5821	`SendRequest`	net6.0	191μs	1.09μs	8.93μs	0.197	18.45 KB
#5821	`SendRequest`	netcoreapp3.1	207μs	1.16μs	8.09μs	0.204	20.61 KB
#5821	`SendRequest`	net472	0.00073ns	0.00033ns	0.00128ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	549μs	1.54μs	5.75μs	0.553	0	0	41.61 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	3.15μs	13.7μs	0.326	0	0	41.78 KB
master	`WriteAndFlushEnrichedTraces`	net472	866μs	3.95μs	15.3μs	8.36	2.64	0.44	53.29 KB
#5821	`WriteAndFlushEnrichedTraces`	net6.0	571μs	1.96μs	8.09μs	0.604	0	0	41.71 KB
#5821	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	664μs	1.98μs	7.4μs	0.338	0	0	41.85 KB
#5821	`WriteAndFlushEnrichedTraces`	net472	891μs	4.54μs	20.3μs	8.15	2.26	0.453	53.26 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.34μs	0.951ns	3.68ns	0.0141	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.61μs	1.24ns	4.81ns	0.0131	1.02 KB
master	`ExecuteNonQuery`	net472	2.03μs	2.26ns	8.74ns	0.157	987 B
#5821	`ExecuteNonQuery`	net6.0	1.26μs	0.972ns	3.76ns	0.0139	1.02 KB
#5821	`ExecuteNonQuery`	netcoreapp3.1	1.64μs	1.16ns	4.35ns	0.0139	1.02 KB
#5821	`ExecuteNonQuery`	net472	2.02μs	0.576ns	2.08ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.21μs	0.821ns	3.18ns	0.0139	0	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.49μs	1.44ns	5.56ns	0.0134	0	976 B
master	`CallElasticsearch`	net472	2.5μs	1.25ns	4.86ns	0.158	0	995 B
master	`CallElasticsearchAsync`	net6.0	1.36μs	0.94ns	3.64ns	0.0136	0	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	0.256ns	0.924ns	0.0137	0	1.02 KB
master	`CallElasticsearchAsync`	net472	2.5μs	1.95ns	7.56ns	0.166	0.00125	1.05 KB
#5821	`CallElasticsearch`	net6.0	1.31μs	0.968ns	3.75ns	0.0134	0	976 B
#5821	`CallElasticsearch`	netcoreapp3.1	1.51μs	0.643ns	2.49ns	0.0129	0	976 B
#5821	`CallElasticsearch`	net472	2.46μs	1.93ns	7.47ns	0.158	0	995 B
#5821	`CallElasticsearchAsync`	net6.0	1.24μs	3.27ns	12.7ns	0.0134	0	952 B
#5821	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.837ns	3.13ns	0.014	0	1.02 KB
#5821	`CallElasticsearchAsync`	net472	2.63μs	2.44ns	9.46ns	0.166	0	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5821

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0	1.122	1,177.64	1,321.82

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.18μs	1.12ns	4.33ns	0.0129	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.56μs	1.35ns	4.87ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.75μs	1.41ns	5.47ns	0.145	915 B
#5821	`ExecuteAsync`	net6.0	1.32μs	2.11ns	7.88ns	0.0138	952 B
#5821	`ExecuteAsync`	netcoreapp3.1	1.66μs	2.73ns	10.6ns	0.0132	952 B
#5821	`ExecuteAsync`	net472	1.77μs	1.27ns	4.75ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.2μs	18ns	67.5ns	0.0304	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.12μs	3.12ns	12.1ns	0.036	2.76 KB
master	`SendAsync`	net472	7.94μs	6.26ns	24.2ns	0.5	3.15 KB
#5821	`SendAsync`	net6.0	4.29μs	2.36ns	8.84ns	0.0301	2.22 KB
#5821	`SendAsync`	netcoreapp3.1	5.1μs	2.26ns	8.47ns	0.0359	2.76 KB
#5821	`SendAsync`	net472	7.78μs	1.5ns	5.82ns	0.496	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.52μs	0.61ns	2.36ns	0.0236	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.23μs	1.4ns	5.06ns	0.0222	1.64 KB
master	`EnrichedLog`	net472	2.69μs	2.15ns	8.06ns	0.249	1.57 KB
#5821	`EnrichedLog`	net6.0	1.63μs	1.13ns	4.22ns	0.0228	1.64 KB
#5821	`EnrichedLog`	netcoreapp3.1	2.29μs	0.765ns	2.76ns	0.0227	1.64 KB
#5821	`EnrichedLog`	net472	2.66μs	1.95ns	7.54ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	163ns	630ns	0.059	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	162ns	629ns	0.0598	0	4.28 KB
master	`EnrichedLog`	net472	150μs	150ns	583ns	0.674	0.225	4.46 KB
#5821	`EnrichedLog`	net6.0	116μs	295ns	1.14μs	0.0576	0	4.28 KB
#5821	`EnrichedLog`	netcoreapp3.1	120μs	211ns	818ns	0	0	4.28 KB
#5821	`EnrichedLog`	net472	148μs	274ns	1.02μs	0.659	0.22	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.18μs	0.988ns	3.7ns	0.0303	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.24μs	1.35ns	5.23ns	0.0298	2.2 KB
master	`EnrichedLog`	net472	4.89μs	1.76ns	6.59ns	0.319	2.02 KB
#5821	`EnrichedLog`	net6.0	3.09μs	1.23ns	4.77ns	0.0309	2.2 KB
#5821	`EnrichedLog`	netcoreapp3.1	4.26μs	1.58ns	6.11ns	0.0298	2.2 KB
#5821	`EnrichedLog`	net472	4.84μs	2.41ns	9.03ns	0.318	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.26μs	0.93ns	3.6ns	0.0158	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.75μs	1.23ns	4.75ns	0.0149	0	1.14 KB
master	`SendReceive`	net472	2.23μs	0.856ns	3.32ns	0.184	0.00113	1.16 KB
#5821	`SendReceive`	net6.0	1.38μs	0.859ns	3.33ns	0.0159	0	1.14 KB
#5821	`SendReceive`	netcoreapp3.1	1.84μs	0.481ns	1.8ns	0.0155	0	1.14 KB
#5821	`SendReceive`	net472	2.14μs	1.51ns	5.86ns	0.183	0.00108	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.82μs	1.12ns	4.04ns	0.0225	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.72μs	1.25ns	4.66ns	0.0224	1.65 KB
master	`EnrichedLog`	net472	4.2μs	3.55ns	13.8ns	0.324	2.04 KB
#5821	`EnrichedLog`	net6.0	2.73μs	0.743ns	2.78ns	0.0221	1.6 KB
#5821	`EnrichedLog`	netcoreapp3.1	3.97μs	0.978ns	3.79ns	0.0218	1.65 KB
#5821	`EnrichedLog`	net472	4.37μs	5.67ns	22ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5821

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.139	469.52	534.86

Faster 🎉 in #5821

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.284	509.46	396.71
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.165	697.28	598.47
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.115	749.58	672.00

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	509ns	0.43ns	1.67ns	0.00812	576 B
master	`StartFinishSpan`	netcoreapp3.1	588ns	0.641ns	2.4ns	0.00765	576 B
master	`StartFinishSpan`	net472	698ns	0.529ns	2.05ns	0.0917	578 B
master	`StartFinishScope`	net6.0	470ns	0.343ns	1.33ns	0.00971	696 B
master	`StartFinishScope`	netcoreapp3.1	750ns	0.404ns	1.57ns	0.00927	696 B
master	`StartFinishScope`	net472	819ns	1.51ns	5.84ns	0.104	658 B
#5821	`StartFinishSpan`	net6.0	397ns	0.218ns	0.817ns	0.00816	576 B
#5821	`StartFinishSpan`	netcoreapp3.1	588ns	0.493ns	1.91ns	0.00769	576 B
#5821	`StartFinishSpan`	net472	597ns	0.674ns	2.61ns	0.0917	578 B
#5821	`StartFinishScope`	net6.0	535ns	0.317ns	1.19ns	0.00961	696 B
#5821	`StartFinishScope`	netcoreapp3.1	671ns	0.427ns	1.65ns	0.00932	696 B
#5821	`StartFinishScope`	net472	907ns	0.971ns	3.76ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5821

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1	1.170	1,007.94	861.42

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	638ns	0.46ns	1.78ns	0.00997	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.01μs	1.65ns	6.39ns	0.00964	696 B
master	`RunOnMethodBegin`	net472	1.01μs	1.12ns	4.34ns	0.104	658 B
#5821	`RunOnMethodBegin`	net6.0	641ns	0.501ns	1.81ns	0.00961	696 B
#5821	`RunOnMethodBegin`	netcoreapp3.1	862ns	0.599ns	2.24ns	0.00951	696 B
#5821	`RunOnMethodBegin`	net472	1.09μs	0.9ns	3.37ns	0.104	658 B

andrewlock · 2024-07-30T22:45:56Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5821) (11.158M)   : 0, 11158295
    master (11.202M)   : 0, 11202233
    benchmarks/2.9.0 (11.711M)   : 0, 11710619

    section Automatic
    This PR (5821) (7.435M)   : 0, 7434561
    master (7.538M)   : 0, 7538081
    benchmarks/2.9.0 (8.265M)   : 0, 8264630

    section Trace stats
    master (7.758M)   : 0, 7757943

    section Manual
    master (11.213M)   : 0, 11212808

    section Manual + Automatic
    This PR (5821) (6.855M)   : 0, 6854724
    master (6.917M)   : 0, 6917068

    section DD_TRACE_ENABLED=0
    master (10.379M)   : 0, 10379306

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5821) (9.398M)   : 0, 9397785
    master (9.418M)   : 0, 9417716
    benchmarks/2.9.0 (9.717M)   : 0, 9716586

    section Automatic
    This PR (5821) (6.402M)   : 0, 6402472
    master (6.559M)   : 0, 6558521

    section Trace stats
    master (6.837M)   : 0, 6836938

    section Manual
    master (9.281M)   : 0, 9280678

    section Manual + Automatic
    This PR (5821) (6.053M)   : 0, 6053199
    master (6.154M)   : 0, 6153874

    section DD_TRACE_ENABLED=0
    master (8.911M)   : 0, 8910789

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5821) (9.771M)   : 0, 9770678
    master (9.803M)   : 0, 9802727

    section Automatic
    This PR (5821) (6.510M)   : 0, 6509709
    master (6.555M)   : 0, 6555282

    section Trace stats
    master (7.160M)   : 0, 7160457

    section Manual
    master (9.684M)   : 0, 9684169

    section Manual + Automatic
    This PR (5821) (5.971M)   : 0, 5971358
    master (6.171M)   : 0, 6171345

    section DD_TRACE_ENABLED=0
    master (9.305M)   : 0, 9305209

dromanol · 2024-08-23T09:54:25Z

+        /// In .NET 6+ there's a bug that prevents Rejit-related APIs to work properly when Edit and Continue feature is turned on.
+        /// See https://github.com/dotnet/runtime/issues/91963 for addiitonal details.
+        /// </summary>
+        private static bool IsEnCFeatureEanbled()


NIT: typo here

dromanol

LGTM with a NIT

andrewlock · 2024-08-27T08:06:18Z

+		tracer\test\test-applications\Samples.Shared\Samples.Shared.projitems*{021efba6-c4ba-4de5-bf3f-c263ee9e20db}*SharedItemsImports = 5
+		tracer\test\test-applications\Samples.Shared\Samples.Shared.projitems*{061ab58b-8235-4dae-8d56-5f081dd78f5e}*SharedItemsImports = 5


I don't think these should all be added here 🤔

Visual Studio rewrites 🤔, I've added a reference to the new sample (which in turn references the Samples.Shared proj)

lucaspimentel

I looked only at non-debugger files. I'm a bit concerned about Datadog.Trace.sln, but if VS did that automatically and it doesn't break anyone else, it's probably fine.

Added the infrastructure of integration testing for Exception Replay in ASP.NET Core. To add a test, one needs to simply create a class that inherits from `IRun` / `IAsyncRun` and decorated with `ExceptionReplayTestData` attribute, providing to the attribute the amount of snapshots anticipated both in default configuration and full stack trace capture configuration. This PR also introduces the renaming of the configs of Exception Replay from Exception Debugging. In the early days we used to call Exception Replay internally Exception Debugging, thus some of the configs held this name. This PR renamed those leftovers to Exception Replay, including the enablement config of `DD_EXCEPTION_REPLAY_ENABLED`. [Introduced a PR to `dd-go`](DataDog/dd-go#146180) too where these configs were added to the `config_norm_rules.json` file. Laying the groundwork for integration testing of Exception Replay. The testing infra bootstraps the web app, queries via reflection all the tests (classes inheriting from `IRun`/`IAsyncRun` and decorated with `ExceptionReplayTestData` attribute) and list them. Upon executing a test, a request will hit an endpoint of `RunTest` with the name of the test to run (FQN of the class name). The crafted test must throw an exception to be considered valid. For a more complex test scenarios, more endpoints could be introduced. Every test runs in 4 flavors: - With Exception Replay while Dynamic Instrumentation is **enabled**, default stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **enabled**, full stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **disabled**, default stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **disabled**, full stack trace capturing limit. This whole PR 🤓. I've added two tests that utilize the infra: - `ExceptionWithNonSupportedFramesTest` - `RecursiveExceptionTest` Fixes DEBUG-2681, DEBUG-2732, DEBUG-1642.

Added the infrastructure of integration testing for Exception Replay in ASP.NET Core. To add a test, one needs to simply create a class that inherits from `IRun` / `IAsyncRun` and decorated with `ExceptionReplayTestData` attribute, providing to the attribute the amount of snapshots anticipated both in default configuration and full stack trace capture configuration. This PR also introduces the renaming of the configs of Exception Replay from Exception Debugging. In the early days we used to call Exception Replay internally Exception Debugging, thus some of the configs held this name. This PR renamed those leftovers to Exception Replay, including the enablement config of `DD_EXCEPTION_REPLAY_ENABLED`. [Introduced a PR to `dd-go`](https://github.com/DataDog/dd-go/pull/146180) too where these configs were added to the `config_norm_rules.json` file. Laying the groundwork for integration testing of Exception Replay. The testing infra bootstraps the web app, queries via reflection all the tests (classes inheriting from `IRun`/`IAsyncRun` and decorated with `ExceptionReplayTestData` attribute) and list them. Upon executing a test, a request will hit an endpoint of `RunTest` with the name of the test to run (FQN of the class name). The crafted test must throw an exception to be considered valid. For a more complex test scenarios, more endpoints could be introduced. Every test runs in 4 flavors: - With Exception Replay while Dynamic Instrumentation is **enabled**, default stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **enabled**, full stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **disabled**, default stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **disabled**, full stack trace capturing limit. This whole PR 🤓. I've added two tests that utilize the infra: - `ExceptionWithNonSupportedFramesTest` - `RecursiveExceptionTest` Fixes DEBUG-2681, DEBUG-2732, DEBUG-1642.

…5970) Added the infrastructure of integration testing for Exception Replay in ASP.NET Core. To add a test, one needs to simply create a class that inherits from `IRun` / `IAsyncRun` and decorated with `ExceptionReplayTestData` attribute, providing to the attribute the amount of snapshots anticipated both in default configuration and full stack trace capture configuration. This PR also introduces the renaming of the configs of Exception Replay from Exception Debugging. In the early days we used to call Exception Replay internally Exception Debugging, thus some of the configs held this name. This PR renamed those leftovers to Exception Replay, including the enablement config of `DD_EXCEPTION_REPLAY_ENABLED`. [Introduced a PR to `dd-go`](DataDog/dd-go#146180) too where these configs were added to the `config_norm_rules.json` file. Laying the groundwork for integration testing of Exception Replay. The testing infra bootstraps the web app, queries via reflection all the tests (classes inheriting from `IRun`/`IAsyncRun` and decorated with `ExceptionReplayTestData` attribute) and list them. Upon executing a test, a request will hit an endpoint of `RunTest` with the name of the test to run (FQN of the class name). The crafted test must throw an exception to be considered valid. For a more complex test scenarios, more endpoints could be introduced. Every test runs in 4 flavors: - With Exception Replay while Dynamic Instrumentation is **enabled**, default stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **enabled**, full stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **disabled**, default stack trace capturing limit. - With Exception Replay while Dynamic Instrumentation is **disabled**, full stack trace capturing limit. This whole PR 🤓. I've added two tests that utilize the infra: - `ExceptionWithNonSupportedFramesTest` - `RecursiveExceptionTest` Fixes DEBUG-2681, DEBUG-2732, DEBUG-1642. Backport of #5821 Co-authored-by: Matan Green <[email protected]>

GreenMatan changed the title ~~[Exception Replay] Added tests for ASP .NET Core 5~~ [Exception Replay] Added test suite for ASP .NET Core 5 Jul 29, 2024

GreenMatan changed the title ~~[Exception Replay] Added test suite for ASP .NET Core 5~~ [Exception Replay] Added test suite for ASP.NET Core 5 Jul 29, 2024

github-actions Bot added the area:tests unit tests, integration tests label Jul 29, 2024

GreenMatan force-pushed the matang/exception-replay-testing-suite branch 2 times, most recently from ab0ae58 to b3495da Compare July 29, 2024 13:35

GreenMatan force-pushed the matang/exception-replay-testing-suite branch from b3495da to 216caf8 Compare July 30, 2024 19:53

GreenMatan changed the title ~~[Exception Replay] Added test suite for ASP.NET Core 5~~ [Exception Replay] DEBUG-2681 Added test suite for ASP.NET Core 5 Aug 7, 2024

GreenMatan changed the title ~~[Exception Replay] DEBUG-2681 Added test suite for ASP.NET Core 5~~ [Exception Replay] Added test suite for ASP.NET Core 5 Aug 13, 2024

GreenMatan force-pushed the matang/exception-replay-testing-suite branch 7 times, most recently from a3dd2e7 to 293af30 Compare August 14, 2024 22:04

GreenMatan marked this pull request as ready for review August 14, 2024 22:04

GreenMatan requested a review from a team as a code owner August 14, 2024 22:04

GreenMatan force-pushed the matang/exception-replay-testing-suite branch from 293af30 to 30250ae Compare August 15, 2024 11:18

GreenMatan requested a review from a team as a code owner August 15, 2024 11:18

GreenMatan force-pushed the matang/exception-replay-testing-suite branch 5 times, most recently from 9f68911 to ff1a613 Compare August 16, 2024 13:18

GreenMatan requested a review from a team as a code owner August 16, 2024 13:18

GreenMatan force-pushed the matang/exception-replay-testing-suite branch from ff1a613 to 061a568 Compare August 16, 2024 22:48

GreenMatan force-pushed the matang/exception-replay-testing-suite branch 2 times, most recently from f28607e to b38e64d Compare August 22, 2024 16:06

dromanol reviewed Aug 23, 2024

View reviewed changes

dromanol approved these changes Aug 23, 2024

View reviewed changes

GreenMatan force-pushed the matang/exception-replay-testing-suite branch from b38e64d to 11f4fd7 Compare August 25, 2024 09:22

andrewlock reviewed Aug 27, 2024

View reviewed changes

GreenMatan force-pushed the matang/exception-replay-testing-suite branch 2 times, most recently from 0d7f6a3 to 3137b56 Compare August 27, 2024 14:12

lucaspimentel approved these changes Aug 27, 2024

View reviewed changes

GreenMatan force-pushed the matang/exception-replay-testing-suite branch from 3137b56 to 10cbf1a Compare August 27, 2024 15:30

andrewlock approved these changes Aug 27, 2024

View reviewed changes

GreenMatan force-pushed the matang/exception-replay-testing-suite branch 2 times, most recently from 2db9eab to 9dc3cd5 Compare August 29, 2024 08:52

datadog-datadog-prod-us1 Bot reviewed Aug 29, 2024

View reviewed changes

GreenMatan force-pushed the matang/exception-replay-testing-suite branch from 9dc3cd5 to da4a63b Compare August 29, 2024 16:13

GreenMatan added 6 commits August 30, 2024 13:33

Added tests for Exception Replay

2043e12

Renamed Exception Replay configs from Exception Debugging

a944fb6

Simplified the nuke of exception replay

15d8bd5

Addressed PR suggestions

dc0da99

Clearing rented array after receiving it

1e484cf

Fixed failing tests + fine-tunings

fc591b4

andrewlock force-pushed the matang/exception-replay-testing-suite branch from da4a63b to fc591b4 Compare August 30, 2024 12:34

andrewlock merged commit 01c50c3 into master Aug 30, 2024

andrewlock deleted the matang/exception-replay-testing-suite branch August 30, 2024 14:15

github-actions Bot added this to the vNext-v3 milestone Aug 30, 2024

andrewlock added the area:debugger label Aug 30, 2024

andrewlock mentioned this pull request Aug 30, 2024

[Exception Replay] Added test suite for ASP.NET Core (#5821 -> v2) #5970

Merged

		tracer\test\test-applications\Samples.Shared\Samples.Shared.projitems{021efba6-c4ba-4de5-bf3f-c263ee9e20db}SharedItemsImports = 5
		tracer\test\test-applications\Samples.Shared\Samples.Shared.projitems{061ab58b-8235-4dae-8d56-5f081dd78f5e}SharedItemsImports = 5

Uh oh!

Conversation

GreenMatan commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

datadog-ddstaging Bot commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

❌ Failed Tests (1)

Uh oh!

andrewlock commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

andrewlock commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5821

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5821

Faster 🎉 in #5821

Raw results

Faster 🎉 in #5821

Raw results

Uh oh!

andrewlock commented Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Throughput/Crank Report ⚡

Uh oh!

dromanol Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

dromanol left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

GreenMatan Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucaspimentel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

GreenMatan commented Jul 29, 2024 •

edited

Loading

datadog-ddstaging Bot commented Jul 29, 2024 •

edited

Loading

andrewlock commented Jul 29, 2024 •

edited

Loading

andrewlock commented Jul 29, 2024 •

edited

Loading

andrewlock commented Jul 30, 2024 •

edited

Loading