Skip to content

[CI Visibility] Set TracerSettings using a configuration source instead of mutating#6399

Merged
andrewlock merged 1 commit into
masterfrom
andrew/ci-vis-use-configuration-source
Dec 9, 2024
Merged

[CI Visibility] Set TracerSettings using a configuration source instead of mutating#6399
andrewlock merged 1 commit into
masterfrom
andrew/ci-vis-use-configuration-source

Conversation

@andrewlock

@andrewlock andrewlock commented Dec 4, 2024

Copy link
Copy Markdown
Member

Summary of changes

Change how CI Visibility creates its TracerSettings object to avoid mutating settings afterwards

Reason for change

This stack of changes is about removing duplication (among other things) in the TracerSettings etc related objects. This is partially required in order to remove the "snapshot generator" complexity that was removed in #6370. Given TracerSettings are not exposed publicly in Datadog.Trace, we want to move to doing a "one shot" configuring of them, ultimately so that we can make the object immutable (and remove ImmutableTracerSettings and friends).

Implementation details

CI Visibility is one of the few places where we mutate settings after creating the TracerSettings. This is mostly because there's additional logic that CI Visibility wants to perform.

Ultimately, the "solution" to that issue in this PR is to move that logic into the TracerSettings constructor. I'm not entirely happy about it, but it's the only approach I could find that works.

  • Add a "dummy" configuration value to a configuration source when creating the TracerSettings. This is used purely as a "switch" to say "we're in CI Visibility mode".
    • We absolutely could just pass this in as a constructor parameter. I went that route first and then backed away, but can totally be swayed.
  • Add an additional configuration source to update settings that we want to change in CI Vis (e.g. logs injection).
  • In the TracerSettings ctor, add an additional ignored URL when in CI Vis mode, modify the default service name (if required) and add an additional GlobalTag.

Test coverage

I'd love to have some, but the CI Visibility configuration is kinda all over the place, so if you have any pointers @tonyredondo I'm all ears...

Other details

Part of stack

@andrewlock andrewlock added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) type:refactor area:ci-visibility labels Dec 4, 2024
@andrewlock
andrewlock requested review from a team as code owners December 4, 2024 18:15
andrewlock added a commit that referenced this pull request Dec 5, 2024
## Summary of changes

- Delete the tracer settings snapshot source generator
- Delete the snapshot objects

## Reason for change

We used these for tracking when customers use config in code so we know
which properties they're changing. However, in v3 we explicitly set the
values ourselves in the `ConfigureIntegration` so we may as well do all
the work there

## Implementation details

- Delete the source generator
- Delete usages of `[GenerateSnapshot]`, `[IgnoreForSnapshot]`, and
`[ConfigKey]`
- Delete `TracerSettingsSnapshot` and associated files and usages
- Record config changes in `ConfigureIntegration`

## Test coverage

Added a simple basic assertion to the manual instrumentation tests to
assert that we're still recording the `code` origin

## Other details

This is step 1 of a whole load of post-v3 code cleanup 😄 

- #6370 👈 This PR
- #6376
- #6385
- #6386
- #6397
- #6399
@andrewlock
andrewlock force-pushed the andrew/manual-instrumentation-config-source branch from e5ebb65 to 6d25385 Compare December 5, 2024 09:53
@andrewlock
andrewlock requested review from a team as code owners December 5, 2024 09:53
@andrewlock
andrewlock force-pushed the andrew/ci-vis-use-configuration-source branch from 197b83e to bcae049 Compare December 5, 2024 09:53

@kevingosse kevingosse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly :blindfold:, I'm leaving it to Tony to validate the CI visibility logic changes

HttpClientExcludedUrlSubstrings = !string.IsNullOrEmpty(urlSubstringSkips)
? TrimSplitString(urlSubstringSkips.ToUpperInvariant(), commaSeparator)
: Array.Empty<string>();
: [];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Living the dream

@datadog-ddstaging

datadog-ddstaging Bot commented Dec 5, 2024

Copy link
Copy Markdown

Datadog Report

Branch report: andrew/ci-vis-use-configuration-source
Commit report: 244bf25
Test service: dd-trace-dotnet

✅ 0 Failed, 461788 Passed, 2805 Skipped, 18h 56m 57.43s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

  • NoExceptions - Datadog.Trace.ClrProfiler.IntegrationTests.SmokeTests.AssemblyResolveMscorlibResourcesInfiniteRecursionCrashSmokeTest - Last Failure

    Expand for error
     Expected no errors in smoke test: ptrace(ATTACH, 21722) FAILED Operation not permitted
     
     Expected: True
     Actual:   False
    

@andrewlock
andrewlock force-pushed the andrew/manual-instrumentation-config-source branch from 6d25385 to 0dcb6ec Compare December 5, 2024 12:52
@andrewlock
andrewlock force-pushed the andrew/ci-vis-use-configuration-source branch from bcae049 to 8c4bbd5 Compare December 5, 2024 12:52
@andrewlock

andrewlock commented Dec 5, 2024

Copy link
Copy Markdown
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6399) - mean (69ms)  : 66, 72
     .   : milestone, 69,
    master - mean (70ms)  : 67, 72
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6399) - mean (982ms)  : 961, 1003
     .   : milestone, 982,
    master - mean (988ms)  : 963, 1014
     .   : milestone, 988,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6399) - mean (109ms)  : 107, 111
     .   : milestone, 109,
    master - mean (110ms)  : 107, 114
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (6399) - mean (680ms)  : 666, 693
     .   : milestone, 680,
    master - mean (687ms)  : 669, 704
     .   : milestone, 687,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6399) - mean (92ms)  : 90, 94
     .   : milestone, 92,
    master - mean (93ms)  : 91, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (6399) - mean (634ms)  : 616, 652
     .   : milestone, 634,
    master - mean (643ms)  : 626, 660
     .   : milestone, 643,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6399) - mean (191ms)  : 186, 196
     .   : milestone, 191,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6399) - mean (1,093ms)  : 1063, 1122
     .   : milestone, 1093,
    master - mean (1,091ms)  : 1063, 1119
     .   : milestone, 1091,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6399) - mean (277ms)  : 272, 282
     .   : milestone, 277,
    master - mean (276ms)  : 272, 280
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6399) - mean (871ms)  : 839, 903
     .   : milestone, 871,
    master - mean (869ms)  : 839, 900
     .   : milestone, 869,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6399) - mean (266ms)  : 263, 270
     .   : milestone, 266,
    master - mean (266ms)  : 262, 269
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (6399) - mean (850ms)  : 818, 881
     .   : milestone, 850,
    master - mean (850ms)  : 818, 882
     .   : milestone, 850,

Loading

@andrewlock

andrewlock commented Dec 5, 2024

Copy link
Copy Markdown
Member Author

Benchmarks Report for tracer 🐌

Benchmarks for #6399 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.119
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.97μs 42.9ns 235ns 0.00799 0 0 5.61 KB
master StartStopWithChild netcoreapp3.1 10.3μs 55.8ns 382ns 0.0203 0.0101 0 5.81 KB
master StartStopWithChild net472 16.3μs 47.8ns 179ns 1.04 0.298 0.0886 6.21 KB
#6399 StartStopWithChild net6.0 7.97μs 45.5ns 328ns 0.0161 0.00805 0 5.61 KB
#6399 StartStopWithChild netcoreapp3.1 10.3μs 55.2ns 322ns 0.0205 0.0102 0 5.8 KB
#6399 StartStopWithChild net472 16.3μs 44.6ns 167ns 1.03 0.303 0.0902 6.21 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 476μs 498ns 1.93μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 652μs 387ns 1.5μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 844μs 346ns 1.34μs 0.419 0 0 3.3 KB
#6399 WriteAndFlushEnrichedTraces net6.0 492μs 250ns 970ns 0 0 0 2.7 KB
#6399 WriteAndFlushEnrichedTraces netcoreapp3.1 656μs 410ns 1.59μs 0 0 0 2.7 KB
#6399 WriteAndFlushEnrichedTraces net472 840μs 325ns 1.17μs 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 151μs 882ns 8.18μs 0.163 0 0 14.47 KB
master SendRequest netcoreapp3.1 170μs 1.06μs 10.4μs 0.176 0 0 17.27 KB
master SendRequest net472 0.00274ns 0.000976ns 0.00378ns 0 0 0 0 b
#6399 SendRequest net6.0 142μs 772ns 4.16μs 0.146 0 0 14.47 KB
#6399 SendRequest netcoreapp3.1 161μs 937ns 7.9μs 0.159 0 0 17.27 KB
#6399 SendRequest net472 0.00323ns 0.000715ns 0.00268ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 582μs 2.98μs 14.3μs 0.558 0 0 41.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 681μs 3.98μs 35.6μs 0.327 0 0 41.81 KB
master WriteAndFlushEnrichedTraces net472 829μs 3.84μs 14.9μs 8.12 2.44 0.406 53.3 KB
#6399 WriteAndFlushEnrichedTraces net6.0 552μs 1.46μs 5.25μs 0.551 0 0 41.64 KB
#6399 WriteAndFlushEnrichedTraces netcoreapp3.1 639μs 1.09μs 4.08μs 0.314 0 0 41.75 KB
#6399 WriteAndFlushEnrichedTraces net472 853μs 2.62μs 10.1μs 8.39 2.52 0.419 53.3 KB
Benchmarks.Trace.DbCommandBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6399

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0 1.120 1,245.58 1,394.86

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.25μs 1ns 3.89ns 0.0143 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.73μs 1.41ns 5.29ns 0.013 0 0 1.02 KB
master ExecuteNonQuery net472 2.09μs 2.33ns 9.03ns 0.156 0.00104 0 987 B
#6399 ExecuteNonQuery net6.0 1.4μs 0.696ns 2.61ns 0.014 0 0 1.02 KB
#6399 ExecuteNonQuery netcoreapp3.1 1.76μs 2.65ns 10.3ns 0.0134 0 0 1.02 KB
#6399 ExecuteNonQuery net472 2.05μs 2.98ns 11.2ns 0.156 0.00103 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.23μs 0.529ns 1.98ns 0.0135 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.65μs 0.639ns 2.39ns 0.0132 0 0 976 B
master CallElasticsearch net472 2.58μs 2.46ns 9.52ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.21μs 0.969ns 3.75ns 0.0133 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.57μs 0.575ns 2.15ns 0.0133 0 0 1.02 KB
master CallElasticsearchAsync net472 2.66μs 1.72ns 6.67ns 0.166 0 0 1.05 KB
#6399 CallElasticsearch net6.0 1.28μs 1.82ns 6.81ns 0.0134 0 0 976 B
#6399 CallElasticsearch netcoreapp3.1 1.49μs 0.906ns 3.51ns 0.0134 0 0 976 B
#6399 CallElasticsearch net472 2.54μs 1.15ns 4.15ns 0.157 0 0 995 B
#6399 CallElasticsearchAsync net6.0 1.27μs 1.25ns 4.66ns 0.0134 0 0 952 B
#6399 CallElasticsearchAsync netcoreapp3.1 1.54μs 0.838ns 3.14ns 0.0139 0 0 1.02 KB
#6399 CallElasticsearchAsync net472 2.63μs 1.43ns 5.34ns 0.167 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.37μs 1.37ns 5.13ns 0.0131 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.72μs 1.13ns 4.21ns 0.0129 0 0 952 B
master ExecuteAsync net472 1.81μs 0.823ns 3.19ns 0.145 0 0 915 B
#6399 ExecuteAsync net6.0 1.32μs 1.31ns 4.71ns 0.0131 0 0 952 B
#6399 ExecuteAsync netcoreapp3.1 1.61μs 1.11ns 4.14ns 0.0129 0 0 952 B
#6399 ExecuteAsync net472 1.85μs 0.785ns 3.04ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.39μs 2.34ns 8.75ns 0.0328 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.31μs 1.6ns 6.18ns 0.037 0 0 2.85 KB
master SendAsync net472 7.48μs 1.86ns 7.21ns 0.493 0 0 3.12 KB
#6399 SendAsync net6.0 4.46μs 1.94ns 7.26ns 0.0311 0 0 2.31 KB
#6399 SendAsync netcoreapp3.1 5.33μs 2.51ns 9.72ns 0.0373 0 0 2.85 KB
#6399 SendAsync net472 7.4μs 1.43ns 5.53ns 0.493 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.43μs 0.522ns 1.88ns 0.0229 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.26μs 0.877ns 3.28ns 0.0216 0 0 1.64 KB
master EnrichedLog net472 2.73μs 2.1ns 8.12ns 0.249 0 0 1.57 KB
#6399 EnrichedLog net6.0 1.54μs 0.757ns 2.83ns 0.0231 0 0 1.64 KB
#6399 EnrichedLog netcoreapp3.1 2.2μs 0.885ns 3.31ns 0.0224 0 0 1.64 KB
#6399 EnrichedLog net472 2.69μs 1.4ns 5.05ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 119μs 123ns 477ns 0.0599 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 123μs 262ns 1.01μs 0.0615 0 0 4.28 KB
master EnrichedLog net472 151μs 180ns 699ns 0.683 0.228 0 4.46 KB
#6399 EnrichedLog net6.0 119μs 101ns 391ns 0 0 0 4.28 KB
#6399 EnrichedLog netcoreapp3.1 123μs 192ns 719ns 0 0 0 4.28 KB
#6399 EnrichedLog net472 151μs 109ns 420ns 0.681 0.227 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.02μs 1.13ns 4.23ns 0.0302 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.19μs 1.25ns 4.68ns 0.0294 0 0 2.2 KB
master EnrichedLog net472 4.86μs 1.26ns 4.87ns 0.319 0 0 2.02 KB
#6399 EnrichedLog net6.0 2.99μs 1.46ns 5.64ns 0.0314 0 0 2.2 KB
#6399 EnrichedLog netcoreapp3.1 4.14μs 1.59ns 5.95ns 0.0289 0 0 2.2 KB
#6399 EnrichedLog net472 4.82μs 1.44ns 5.38ns 0.318 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.32μs 0.675ns 2.53ns 0.016 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.72μs 0.739ns 2.86ns 0.0154 0 0 1.14 KB
master SendReceive net472 2.06μs 2.04ns 7.89ns 0.183 0 0 1.16 KB
#6399 SendReceive net6.0 1.34μs 3.36ns 12.6ns 0.016 0 0 1.14 KB
#6399 SendReceive netcoreapp3.1 1.8μs 0.691ns 2.68ns 0.0152 0 0 1.14 KB
#6399 SendReceive net472 2.11μs 1.56ns 5.82ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.85μs 1.37ns 5.14ns 0.0228 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.97μs 1.99ns 7.72ns 0.0219 0 0 1.65 KB
master EnrichedLog net472 4.37μs 2.06ns 7.13ns 0.323 0 0 2.04 KB
#6399 EnrichedLog net6.0 2.78μs 1.33ns 4.98ns 0.0218 0 0 1.6 KB
#6399 EnrichedLog netcoreapp3.1 3.87μs 1.86ns 6.71ns 0.0212 0 0 1.65 KB
#6399 EnrichedLog net472 4.36μs 3.12ns 12.1ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6399

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.118 483.30 540.41

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 456ns 0.117ns 0.439ns 0.00819 0 0 576 B
master StartFinishSpan netcoreapp3.1 612ns 0.431ns 1.67ns 0.00783 0 0 576 B
master StartFinishSpan net472 701ns 0.35ns 1.35ns 0.0916 0 0 578 B
master StartFinishScope net6.0 483ns 0.207ns 0.803ns 0.00974 0 0 696 B
master StartFinishScope netcoreapp3.1 795ns 0.468ns 1.75ns 0.00919 0 0 696 B
master StartFinishScope net472 869ns 0.484ns 1.88ns 0.104 0 0 658 B
#6399 StartFinishSpan net6.0 462ns 0.165ns 0.637ns 0.00808 0 0 576 B
#6399 StartFinishSpan netcoreapp3.1 643ns 0.383ns 1.48ns 0.00801 0 0 576 B
#6399 StartFinishSpan net472 690ns 0.816ns 3.16ns 0.0915 0 0 578 B
#6399 StartFinishScope net6.0 540ns 0.38ns 1.47ns 0.0097 0 0 696 B
#6399 StartFinishScope netcoreapp3.1 764ns 0.506ns 1.96ns 0.00952 0 0 696 B
#6399 StartFinishScope net472 893ns 0.17ns 0.637ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 685ns 0.359ns 1.39ns 0.00954 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 955ns 0.461ns 1.79ns 0.00954 0 0 696 B
master RunOnMethodBegin net472 1.11μs 0.797ns 2.87ns 0.105 0 0 658 B
#6399 RunOnMethodBegin net6.0 667ns 0.394ns 1.53ns 0.00973 0 0 696 B
#6399 RunOnMethodBegin netcoreapp3.1 942ns 0.584ns 2.26ns 0.00949 0 0 696 B
#6399 RunOnMethodBegin net472 1.07μs 0.471ns 1.83ns 0.104 0 0 658 B

@andrewlock

andrewlock commented Dec 5, 2024

Copy link
Copy Markdown
Member Author

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6399) (10.904M)   : 0, 10904226
    master (11.201M)   : 0, 11200575
    benchmarks/2.9.0 (11.033M)   : 0, 11032866

    section Automatic
    This PR (6399) (7.284M)   : 0, 7284316
    master (7.193M)   : 0, 7192592
    benchmarks/2.9.0 (7.786M)   : 0, 7785853

    section Trace stats
    master (7.637M)   : 0, 7636936

    section Manual
    master (11.152M)   : 0, 11151831

    section Manual + Automatic
    This PR (6399) (6.704M)   : 0, 6703813
    master (6.647M)   : 0, 6647097

    section DD_TRACE_ENABLED=0
    master (10.391M)   : 0, 10390637

Loading
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6399) (9.543M)   : 0, 9543095
    master (9.756M)   : 0, 9756093
    benchmarks/2.9.0 (9.495M)   : 0, 9494821

    section Automatic
    This PR (6399) (6.344M)   : 0, 6344407
    master (6.416M)   : 0, 6416022

    section Trace stats
    master (6.677M)   : 0, 6677436

    section Manual
    master (9.406M)   : 0, 9405878

    section Manual + Automatic
    This PR (6399) (5.971M)   : 0, 5971239
    master (5.969M)   : 0, 5969446

    section DD_TRACE_ENABLED=0
    master (8.683M)   : 0, 8683346

Loading
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6399) (9.701M)   : 0, 9701376
    master (9.691M)   : 0, 9690567
    benchmarks/2.9.0 (10.020M)   : 0, 10019592

    section Automatic
    This PR (6399) (6.361M)   : 0, 6361166
    master (6.374M)   : 0, 6373753
    benchmarks/2.9.0 (7.255M)   : 0, 7255257

    section Trace stats
    master (7.051M)   : 0, 7051311

    section Manual
    master (9.664M)   : 0, 9664005

    section Manual + Automatic
    This PR (6399) (6.091M)   : 0, 6090830
    master (6.028M)   : 0, 6028279

    section DD_TRACE_ENABLED=0
    master (9.015M)   : 0, 9015445

Loading

@andrewlock

andrewlock commented Dec 5, 2024

Copy link
Copy Markdown
Member Author

Benchmarks Report for appsec 🐌

Benchmarks for #6399 compared to master:

  • All benchmarks have the same speed
  • 1 benchmarks have fewer allocations
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 212μs 84.6ns 305ns 2.88 0 0 205.48 KB
master AllCycleSimpleBody netcoreapp3.1 309μs 153ns 591ns 2.8 0 0 212.92 KB
master AllCycleSimpleBody net472 278μs 95.1ns 356ns 38 2.9 0 240.1 KB
master AllCycleMoreComplexBody net6.0 223μs 104ns 388ns 2.89 0 0 208.98 KB
master AllCycleMoreComplexBody netcoreapp3.1 324μs 94.6ns 354ns 2.97 0 0 216.34 KB
master AllCycleMoreComplexBody net472 284μs 220ns 853ns 38.6 2.84 0 243.62 KB
master ObjectExtractorSimpleBody net6.0 137ns 0.117ns 0.439ns 0.00394 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 212ns 1.2ns 8.33ns 0.00376 0 0 272 B
master ObjectExtractorSimpleBody net472 212ns 0.136ns 0.508ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.96μs 1.6ns 6.21ns 0.0532 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.8μs 1.81ns 6.78ns 0.0495 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 4.21μs 3.12ns 10.8ns 0.602 0.00629 0 3.8 KB
#6399 AllCycleSimpleBody net6.0 217μs 83.8ns 313ns 2.85 0 0 205.66 KB
#6399 AllCycleSimpleBody netcoreapp3.1 330μs 870ns 3.37μs 2.86 0 0 213.15 KB
#6399 AllCycleSimpleBody net472 285μs 485ns 1.88μs 38.2 2.85 0 240.52 KB
#6399 AllCycleMoreComplexBody net6.0 223μs 123ns 462ns 2.99 0 0 209.16 KB
#6399 AllCycleMoreComplexBody netcoreapp3.1 339μs 172ns 642ns 2.86 0 0 216.56 KB
#6399 AllCycleMoreComplexBody net472 288μs 243ns 943ns 38.7 2.87 0 244.03 KB
#6399 ObjectExtractorSimpleBody net6.0 143ns 0.0734ns 0.284ns 0.00395 0 0 280 B
#6399 ObjectExtractorSimpleBody netcoreapp3.1 195ns 0.172ns 0.642ns 0.00365 0 0 272 B
#6399 ObjectExtractorSimpleBody net472 209ns 0.128ns 0.497ns 0.0446 0 0 281 B
#6399 ObjectExtractorMoreComplexBody net6.0 3.04μs 1.21ns 4.54ns 0.0534 0 0 3.78 KB
#6399 ObjectExtractorMoreComplexBody netcoreapp3.1 3.9μs 1.97ns 7.36ns 0.0506 0 0 3.69 KB
#6399 ObjectExtractorMoreComplexBody net472 4.22μs 6.76ns 26.2ns 0.603 0.00626 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 38.4μs 16.5ns 61.6ns 0.443 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54.4μs 32ns 124ns 0.433 0 0 32.4 KB
master EncodeArgs net472 68.3μs 56.9ns 213ns 5.16 0.0683 0 32.5 KB
master EncodeLegacyArgs net6.0 73.4μs 45.1ns 162ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 109μs 105ns 392ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 156μs 122ns 474ns 0.31 0 0 2.15 KB
#6399 EncodeArgs net6.0 38.8μs 20.5ns 76.8ns 0.448 0 0 32.4 KB
#6399 EncodeArgs netcoreapp3.1 54μs 10.6ns 39.7ns 0.432 0 0 32.4 KB
#6399 EncodeArgs net472 67.1μs 49.7ns 193ns 5.14 0.0667 0 32.5 KB
#6399 EncodeLegacyArgs net6.0 74.1μs 77.1ns 298ns 0.037 0 0 2.14 KB
#6399 EncodeLegacyArgs netcoreapp3.1 106μs 110ns 428ns 0 0 0 2.14 KB
#6399 EncodeLegacyArgs net472 156μs 83ns 321ns 0.311 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 184μs 164ns 634ns 0 0 0 2.44 KB
master RunWafRealisticBenchmark netcoreapp3.1 197μs 262ns 1.01μs 0 0 0 2.39 KB
master RunWafRealisticBenchmark net472 209μs 167ns 624ns 0.317 0 0 2.46 KB
master RunWafRealisticBenchmarkWithAttack net6.0 122μs 31.4ns 117ns 0 0 0 1.47 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 130μs 95.5ns 344ns 0 0 0 1.46 KB
master RunWafRealisticBenchmarkWithAttack net472 141μs 128ns 494ns 0.209 0 0 1.49 KB
#6399 RunWafRealisticBenchmark net6.0 184μs 106ns 410ns 0 0 0 2.44 KB
#6399 RunWafRealisticBenchmark netcoreapp3.1 193μs 118ns 440ns 0 0 0 2.39 KB
#6399 RunWafRealisticBenchmark net472 208μs 140ns 543ns 0.311 0 0 2.46 KB
#6399 RunWafRealisticBenchmarkWithAttack net6.0 121μs 42.7ns 165ns 0 0 0 1.47 KB
#6399 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 130μs 130ns 503ns 0 0 0 1.46 KB
#6399 RunWafRealisticBenchmarkWithAttack net472 140μs 115ns 430ns 0.209 0 0 1.49 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6399

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 255.31 KB 262.64 KB 7.33 KB 2.87%

Fewer allocations 🎉 in #6399

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 62.14 KB 61.66 KB -480 B -0.77%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 62.2μs 1.09μs 10.7μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 54μs 269ns 1.11μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.3μs 111ns 414ns 0 0 0 62.14 KB
master StringConcatAspectBenchmark net6.0 316μs 1.62μs 11μs 0 0 0 255.49 KB
master StringConcatAspectBenchmark netcoreapp3.1 349μs 1.99μs 14.9μs 0 0 0 255.31 KB
master StringConcatAspectBenchmark net472 281μs 4.24μs 40.4μs 0 0 0 278.53 KB
#6399 StringConcatBenchmark net6.0 62.4μs 1.14μs 11.2μs 0 0 0 43.44 KB
#6399 StringConcatBenchmark netcoreapp3.1 62.3μs 905ns 8.86μs 0 0 0 42.64 KB
#6399 StringConcatBenchmark net472 37.9μs 142ns 511ns 0 0 0 61.66 KB
#6399 StringConcatAspectBenchmark net6.0 351μs 1.88μs 10.3μs 0 0 0 255.24 KB
#6399 StringConcatAspectBenchmark netcoreapp3.1 314μs 5.49μs 54.6μs 0 0 0 262.64 KB
#6399 StringConcatAspectBenchmark net472 286μs 5.63μs 54.6μs 0 0 0 278.53 KB

andrewlock added a commit that referenced this pull request Dec 6, 2024
## Summary of changes

Delete the Public API Generator

## Reason for change

We added this generator to reduce the amount of boilerplate you need to
write to correctly record telemetry for public APIs. However, with the
move to Datadog.Trace.Manual, these APIs are completely unused, so we
may as well remove the complexity and duplication.

## Implementation details

- Delete the `[GeneratePublicApi]` generator
- Keeping the `[PublicApi]` attribute for now - that will be cleared up
in a separate PR
- Rename the previously-decorated properties that _were_ called
`SomePropInternal` to `SomeProp`
- We still have some "leftover" `Internal` suffixes, will tidy those up
separately
- Main changes are in `TracerSettings`, `ExporterSettings`,
`IntegrationSettings` + their immutable counterparts

## Test coverage

Covered by existing tests

## Other details

I noticed that there were some properties that we were getting public
API telemetry for which we kind of lost in the move to
Datadog.Trace.Manual: `SpanContext.Parent`, `SpanContext.ParentId`,
`SpanContext.ServiceName`. I don't _think_ that actually matters, as
these can't actually be accessed any more for technical reasons...

Part of stack 
- #6370
- #6376 👈 This PR
- #6385
- #6386
- #6397
- #6399
- #6405
@andrewlock
andrewlock force-pushed the andrew/ci-vis-use-configuration-source branch from bf1018d to 455d139 Compare December 6, 2024 14:10
andrewlock added a commit that referenced this pull request Dec 6, 2024
## Summary of changes

- Delete legacy `IConfigurationSource`
- Rename `ITelmeteredConfigurationSource` => `IConfigurationSource`
- Remove some "public" API workarounds
- Make some APIs `public` instead of `internal` (just to satisfy
compiler)
- Fix tests

## Reason for change

`ITelmeteredConfigurationSource` was introduced when
`IConfigurationSource` was public, and so could not be changed. We have
since removed that interface from the public API (it's not in
`Datadog.trace.Manual`) so now it simply adds confusion and complexity
(as it should _not_ be used internally in Datadog.Trace).

## Implementation details

This PR looks more complex than it is, because in order to replace all
the usages of the old `IConfigurationSource` with the new one, I had to
mark some other APIs public (and document them to satisfy the
analyzers).

It is mostly

- Delete legacy `IConfigurationSource`
- Rename `ITelmeteredConfigurationSource` => `IConfigurationSource`
- Fix any errors

I took the opportunity to also remove some of the `*Internal` methods
which were used to avoid calling the "public" APIs; seeing as these
aren't _actually_ public any more, that's just unnecessary duplication.

## Test coverage

The testing was a pain. The `ConfigurationBuilder` tests were all
designed to check that `ITelmeteredConfigurationSource` gave the same
results as `IConfigurationSource` (while never _explicitly_ specifying
what the "expected" behaviour was. I took some time to enumerate the
_actual_ expected values for the `NameValueCollection` source, but the
`Json` source has very different behaviour that is more of a pain to
test, so I chose to simplify a lot there. We could do a _lot_ to clean
up those tests, but I didn't want to add even more complexity in this
PR.

## Other details

To help "fix" the ASM tests I introduced a
`DictionaryObjectConfigurationSource`, in which you pass in a
`Dictionary<string, object>`. This is useful for testing (as it was
previously used) but is actually going to be useful for a future
clean-up PR too, so I kept it in Datadog.Trace instead of in the test
project.

Part of stack 
- #6370
- #6376
- #6385
- #6386 👈 This PR
- #6397
- #6399
- #6400
- #6405
- #6408
@andrewlock
andrewlock force-pushed the andrew/manual-instrumentation-config-source branch from dd8d03c to 219e031 Compare December 6, 2024 16:32
@andrewlock
andrewlock force-pushed the andrew/ci-vis-use-configuration-source branch from 455d139 to c6a2380 Compare December 6, 2024 16:32
andrewlock added a commit that referenced this pull request Dec 9, 2024
…nSource` (#6397)

Change how we "apply" settings from manual configuration to the
automatic side to use `IConfigurationSource`

## Reason for change

The previous design of how we "apply" settings made in code using manual
instrumentation required mutating a `TracerSettings` object after it was
already built. In fact, this is pretty much the _only_ place that we
mutate the settings.

By switching to using a "configuration source" approach, so that the
settings are built _once_ with the correct values opens up the option to
make these immutable (and therefore delete all of the `Immutable*`
settings we currently have. This reduces both code duplication and work
we do on startup, and opens the path to further refactoring
improvements.

Note that the public API does not change, so consumers of
Datadog.Trace.Manual are still working with a mutable object initially.

## Implementation details

Currently we pass a `Dictionary<string, object>` between the manual and
automatic side. Previously, we then iterated the keys, compared against
known values, and modified `TracerSettings` as required.

With the changes, we have a `ManualInstrumentationConfigurationSource`
which just "wraps" the `Dictionary<>`, and returns the correct value as
required for a given key. Overall, I think this is much cleaner.

Where things get messy is how we handle disabling specific integrations.
The existing dictionary is optimised for looping through the provided
values, fetching the setting that needs to be modified, and changing all
the required properties. Unfortunately, the `IConfigurationSource`
approach where we're looking up by a key like `DD_TRACE_NPGSQL_ENABLED`
works _horribly_ for this pattern 🙁. So I introduced an additional
approach which explicitly _additionally_ transfers the settings using
these values, making them just "standard" lookups.

> Note that due to backwards + forwards compatibility requirements
> - We _still_ need to send the "old" version of integration settings
from the manual side, in case it's used with an old version of the auto
instrumentation
> - We _still_ need to handle the "old" version of integration settings
in the auto side, in case it's used with an old version of the manual
instrumentation.
> - At least in this case we can use the more efficient
`IConfigurationSource` reader, so we don't pay the expense of retrieving
the settings. The only downside is a couple of extra allocations when
they _do_ disable integrations in code.


Minor other changes:
- Add helper ctor to `CompositeConfigurationSource` for creating the
internal list from a collection of `IConfigurationSource`
- Tweak `DictionaryObjectConfigurationSource` so we can derive from it
- Create a separate integration for <3.7.0 manual instrumentation that
uses the legacy settings, otherwise use the new settings objects

## Test coverage

Mostly covered by existing unit tests (and indirectly by integration
tests). Tweaked the test to test both the new and legacy configuration
source.

## Other details

Requires #6393 to fix how
we read integration settings


Part of stack 
- #6370
- #6376
- #6385
- #6386
- #6397 👈 This PR
- #6399
- #6400
- #6405
- #6408
Base automatically changed from andrew/manual-instrumentation-config-source to master December 9, 2024 09:07
@andrewlock
andrewlock force-pushed the andrew/ci-vis-use-configuration-source branch from c6a2380 to 244bf25 Compare December 9, 2024 09:08
@andrewlock
andrewlock merged commit 1795527 into master Dec 9, 2024
@andrewlock
andrewlock deleted the andrew/ci-vis-use-configuration-source branch December 9, 2024 17:52
@github-actions github-actions Bot added this to the vNext-v3 milestone Dec 9, 2024
andrewlock added a commit that referenced this pull request Dec 10, 2024
…utableDirectLogSubmissionSettings` (#6400)

## Summary of changes

Merge `DirectLogSubmissionSettings` with
`ImmutableDirectLogSubmissionSettings`

## Reason for change

There was never really a good reason for having these as separate types.
It was primarily to make testing a little easier and to mirror the
`TracerSettings`/`ImmutableTracerSettings` dichotomy, but as that's
going away, this is just unnecessary complexity

## Implementation details

- Moved the additional logic that was previously inside
`ImmutableDirectLogSubmissionSettings` into
`DirectLogSubmissionSettings`
- Renamed the `DirectLogSubmissionSettings` properties to match the
"Immutable" version (and remove the unnecessary prefix)
- Replace all usages of `Immutable*` with `DirectLogSubmissionSettings`
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization (using `IConfigurationSource`)

## Test coverage

Essentially the same. I removed/tweaked some tests that are no longer
relevant

## Other details

Part of stack

- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399
- #6400  👈 This PR
- #6405
- #6408
- #6415
andrewlock added a commit that referenced this pull request Dec 11, 2024
…tegrationSettings` (#6405)

## Summary of changes

Merge `IntegrationSettings` with `ImmutableIntegrationSettings`

## Reason for change

This stack of PRs is about doing one-shot configuration instead of
mutation. We never mutate these in the tracer after creation, so there's
no need for the separate types.

## Implementation details

- Moved additional logic (handling `DisabledNames` that was previously
inside `ImmutableIntegrationSettings` into `IntegrationSettings`
- Replace all usages of `Immutable*` with `IntegrationSettings`
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization
- Reorder initialization of `DisabledIntegrationNames` in
`TracerSettings` so that it can be used in the `IntegrationSettings`
constructor

## Test coverage

All covered by existing details

## Other details

Part of Stack
- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399
- #6400
- #6405 👈 This PR
- #6408
- #6415
andrewlock added a commit that referenced this pull request Dec 11, 2024
…terSettings` (#6408)

## Summary of changes

Merge `ExporterSettings` with `ImmutableExporterSettings`

## Reason for change

This stack of PRs is about doing one-shot configuration instead of
mutation. We never mutate these in the tracer after creation, so there's
no need for the separate types.

## Implementation details

- Made the properties in `ExporterSettings` get-only. This required
quite a lot of work because we were doing a lot of mutating of the
settings in the "helper" functions.
- I only _lightly_ refactored those methods (as much as possible) to
avoid setting the properties in the functions and instead returning the
details to set later.
- These are prime candidates for some _much_ heavier refactoring later,
but I didn't want to get bogged down with that in this PR
- Replace all usages of `Immutable*` with `ExporterSettings`
- Replace usages of `AgentUriInternal` with `AgentUri`
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization

## Test coverage

All covered by existing details

## Other details

Part of Stack
- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399
- #6400
- #6405
- #6408 👈 This PR
- #6415
andrewlock added a commit that referenced this pull request Dec 12, 2024
…ettings` (#6415)

## Summary of changes

Merge `TracerSettings` with `ImmutableTracerSettings`

## Reason for change

This stack of PRs is about doing one-shot configuration instead of
mutation. We never mutate these in the tracer after creation, so there's
no need for the separate types.

## Implementation details

- Make the properties in `TracerSettings` get-only. 
- Make the collections in `TracerSettings` readonly.
- Move logic that used to be in the constructor of
`ImmutableTracerSettings` into `TracerSettings`
- e.g. Service/Version/Env were being changed based on DD_TAGS values.
Moved that to TracerSettings and (importantly) added missing telemetry
recording of these values.
  - Added missing recording of _effective_ `DisabledInstegrations`
- Moving this logic caused some _tests_ to be broken (checking default
values). Updated the expected values of those tests in a single
- Replace all usages of `ImmutableTracerSettings` with `TracerSettings`
- Move `ITracer` to Datadog.Trace.Manual
- It's only used there, and references the manual-version of
`ImmutableTracerSettings` which we _want_ to keep.
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization in tests

## Test coverage

All covered by existing tests (I hope) 🤞 

## Other details

There's still a _lot_ of scope to improve this 

Part of Stack
- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399
- #6400
- #6405
- #6408
- #6415 👈 This PR
veerbia pushed a commit that referenced this pull request Dec 16, 2024
## Summary of changes

- Delete the tracer settings snapshot source generator
- Delete the snapshot objects

## Reason for change

We used these for tracking when customers use config in code so we know
which properties they're changing. However, in v3 we explicitly set the
values ourselves in the `ConfigureIntegration` so we may as well do all
the work there

## Implementation details

- Delete the source generator
- Delete usages of `[GenerateSnapshot]`, `[IgnoreForSnapshot]`, and
`[ConfigKey]`
- Delete `TracerSettingsSnapshot` and associated files and usages
- Record config changes in `ConfigureIntegration`

## Test coverage

Added a simple basic assertion to the manual instrumentation tests to
assert that we're still recording the `code` origin

## Other details

This is step 1 of a whole load of post-v3 code cleanup 😄 

- #6370 👈 This PR
- #6376
- #6385
- #6386
- #6397
- #6399
veerbia pushed a commit that referenced this pull request Dec 16, 2024
## Summary of changes

Delete the Public API Generator

## Reason for change

We added this generator to reduce the amount of boilerplate you need to
write to correctly record telemetry for public APIs. However, with the
move to Datadog.Trace.Manual, these APIs are completely unused, so we
may as well remove the complexity and duplication.

## Implementation details

- Delete the `[GeneratePublicApi]` generator
- Keeping the `[PublicApi]` attribute for now - that will be cleared up
in a separate PR
- Rename the previously-decorated properties that _were_ called
`SomePropInternal` to `SomeProp`
- We still have some "leftover" `Internal` suffixes, will tidy those up
separately
- Main changes are in `TracerSettings`, `ExporterSettings`,
`IntegrationSettings` + their immutable counterparts

## Test coverage

Covered by existing tests

## Other details

I noticed that there were some properties that we were getting public
API telemetry for which we kind of lost in the move to
Datadog.Trace.Manual: `SpanContext.Parent`, `SpanContext.ParentId`,
`SpanContext.ServiceName`. I don't _think_ that actually matters, as
these can't actually be accessed any more for technical reasons...

Part of stack 
- #6370
- #6376 👈 This PR
- #6385
- #6386
- #6397
- #6399
- #6405
veerbia pushed a commit that referenced this pull request Dec 16, 2024
…6385)

## Summary of changes

Remove `PublicApiTests.PublicApiHasNotChanged` for Datadog.Trace.dll

## Reason for change

The public API surface of Datadog.Trace is not _really_ public any more,
as it's not directly referenced, so these tests are largely
unneccessary.

As of right now, `public` vs `internal` is essentially irrelevant in
this project.

## Implementation details

Remove the `PublicApiHasNotChanged` test. The other public API tests
e.g. `AssemblyReferencesHaveNotChanged` _are_ still relevant, so jumped
through some hoops to keep them

## Test coverage

Slightly less now.

## Other details

Part of stack 

- #6370
- #6376
- #6385 👈 This PR
- #6386
- #6397
- #6399
- #6400
- #6405
veerbia pushed a commit that referenced this pull request Dec 16, 2024
## Summary of changes

- Delete legacy `IConfigurationSource`
- Rename `ITelmeteredConfigurationSource` => `IConfigurationSource`
- Remove some "public" API workarounds
- Make some APIs `public` instead of `internal` (just to satisfy
compiler)
- Fix tests

## Reason for change

`ITelmeteredConfigurationSource` was introduced when
`IConfigurationSource` was public, and so could not be changed. We have
since removed that interface from the public API (it's not in
`Datadog.trace.Manual`) so now it simply adds confusion and complexity
(as it should _not_ be used internally in Datadog.Trace).

## Implementation details

This PR looks more complex than it is, because in order to replace all
the usages of the old `IConfigurationSource` with the new one, I had to
mark some other APIs public (and document them to satisfy the
analyzers).

It is mostly

- Delete legacy `IConfigurationSource`
- Rename `ITelmeteredConfigurationSource` => `IConfigurationSource`
- Fix any errors

I took the opportunity to also remove some of the `*Internal` methods
which were used to avoid calling the "public" APIs; seeing as these
aren't _actually_ public any more, that's just unnecessary duplication.

## Test coverage

The testing was a pain. The `ConfigurationBuilder` tests were all
designed to check that `ITelmeteredConfigurationSource` gave the same
results as `IConfigurationSource` (while never _explicitly_ specifying
what the "expected" behaviour was. I took some time to enumerate the
_actual_ expected values for the `NameValueCollection` source, but the
`Json` source has very different behaviour that is more of a pain to
test, so I chose to simplify a lot there. We could do a _lot_ to clean
up those tests, but I didn't want to add even more complexity in this
PR.

## Other details

To help "fix" the ASM tests I introduced a
`DictionaryObjectConfigurationSource`, in which you pass in a
`Dictionary<string, object>`. This is useful for testing (as it was
previously used) but is actually going to be useful for a future
clean-up PR too, so I kept it in Datadog.Trace instead of in the test
project.

Part of stack 
- #6370
- #6376
- #6385
- #6386 👈 This PR
- #6397
- #6399
- #6400
- #6405
- #6408
veerbia pushed a commit that referenced this pull request Dec 16, 2024
…nSource` (#6397)

Change how we "apply" settings from manual configuration to the
automatic side to use `IConfigurationSource`

## Reason for change

The previous design of how we "apply" settings made in code using manual
instrumentation required mutating a `TracerSettings` object after it was
already built. In fact, this is pretty much the _only_ place that we
mutate the settings.

By switching to using a "configuration source" approach, so that the
settings are built _once_ with the correct values opens up the option to
make these immutable (and therefore delete all of the `Immutable*`
settings we currently have. This reduces both code duplication and work
we do on startup, and opens the path to further refactoring
improvements.

Note that the public API does not change, so consumers of
Datadog.Trace.Manual are still working with a mutable object initially.

## Implementation details

Currently we pass a `Dictionary<string, object>` between the manual and
automatic side. Previously, we then iterated the keys, compared against
known values, and modified `TracerSettings` as required.

With the changes, we have a `ManualInstrumentationConfigurationSource`
which just "wraps" the `Dictionary<>`, and returns the correct value as
required for a given key. Overall, I think this is much cleaner.

Where things get messy is how we handle disabling specific integrations.
The existing dictionary is optimised for looping through the provided
values, fetching the setting that needs to be modified, and changing all
the required properties. Unfortunately, the `IConfigurationSource`
approach where we're looking up by a key like `DD_TRACE_NPGSQL_ENABLED`
works _horribly_ for this pattern 🙁. So I introduced an additional
approach which explicitly _additionally_ transfers the settings using
these values, making them just "standard" lookups.

> Note that due to backwards + forwards compatibility requirements
> - We _still_ need to send the "old" version of integration settings
from the manual side, in case it's used with an old version of the auto
instrumentation
> - We _still_ need to handle the "old" version of integration settings
in the auto side, in case it's used with an old version of the manual
instrumentation.
> - At least in this case we can use the more efficient
`IConfigurationSource` reader, so we don't pay the expense of retrieving
the settings. The only downside is a couple of extra allocations when
they _do_ disable integrations in code.


Minor other changes:
- Add helper ctor to `CompositeConfigurationSource` for creating the
internal list from a collection of `IConfigurationSource`
- Tweak `DictionaryObjectConfigurationSource` so we can derive from it
- Create a separate integration for <3.7.0 manual instrumentation that
uses the legacy settings, otherwise use the new settings objects

## Test coverage

Mostly covered by existing unit tests (and indirectly by integration
tests). Tweaked the test to test both the new and legacy configuration
source.

## Other details

Requires #6393 to fix how
we read integration settings


Part of stack 
- #6370
- #6376
- #6385
- #6386
- #6397 👈 This PR
- #6399
- #6400
- #6405
- #6408
veerbia pushed a commit that referenced this pull request Dec 16, 2024
…tead of mutating (#6399)

## Summary of changes

Change how CI Visibility creates its `TracerSettings` object to avoid
mutating settings afterwards

## Reason for change

This stack of changes is about removing duplication (among other things)
in the `TracerSettings` etc related objects. This is _partially_
required in order to remove the "snapshot generator" complexity that was
removed in #6370. Given
`TracerSettings` are not exposed publicly in Datadog.Trace, we want to
move to doing a "one shot" configuring of them, ultimately so that we
can make the object immutable (and remove `ImmutableTracerSettings` and
friends).

## Implementation details

CI Visibility is one of the few places where we mutate settings _after_
creating the `TracerSettings`. This is mostly because there's additional
logic that CI Visibility wants to perform.

Ultimately, the "solution" to that issue in this PR is to move that
logic into the `TracerSettings` constructor. I'm not entirely happy
about it, but it's the only approach I could find that works.

- Add a "dummy" configuration value to a configuration source when
creating the `TracerSettings`. This is used purely as a "switch" to say
"we're in CI Visibility mode".
- We absolutely could just pass this in as a constructor parameter. I
went that route first and then backed away, but can totally be swayed.
- Add an additional configuration source to update settings that we want
to change in CI Vis (e.g. logs injection).
- In the `TracerSettings` ctor, add an additional ignored URL when in CI
Vis mode, modify the default service name (if required) and add an
additional GlobalTag.

## Test coverage

I'd love to have some, but the CI Visibility configuration is kinda all
over the place, so if you have any pointers @tonyredondo I'm all ears...

## Other details

Part of stack 
- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399 👈 This PR
- #6400  
- #6405
- #6408
veerbia pushed a commit that referenced this pull request Dec 16, 2024
…utableDirectLogSubmissionSettings` (#6400)

## Summary of changes

Merge `DirectLogSubmissionSettings` with
`ImmutableDirectLogSubmissionSettings`

## Reason for change

There was never really a good reason for having these as separate types.
It was primarily to make testing a little easier and to mirror the
`TracerSettings`/`ImmutableTracerSettings` dichotomy, but as that's
going away, this is just unnecessary complexity

## Implementation details

- Moved the additional logic that was previously inside
`ImmutableDirectLogSubmissionSettings` into
`DirectLogSubmissionSettings`
- Renamed the `DirectLogSubmissionSettings` properties to match the
"Immutable" version (and remove the unnecessary prefix)
- Replace all usages of `Immutable*` with `DirectLogSubmissionSettings`
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization (using `IConfigurationSource`)

## Test coverage

Essentially the same. I removed/tweaked some tests that are no longer
relevant

## Other details

Part of stack

- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399
- #6400  👈 This PR
- #6405
- #6408
- #6415
veerbia pushed a commit that referenced this pull request Dec 16, 2024
…tegrationSettings` (#6405)

## Summary of changes

Merge `IntegrationSettings` with `ImmutableIntegrationSettings`

## Reason for change

This stack of PRs is about doing one-shot configuration instead of
mutation. We never mutate these in the tracer after creation, so there's
no need for the separate types.

## Implementation details

- Moved additional logic (handling `DisabledNames` that was previously
inside `ImmutableIntegrationSettings` into `IntegrationSettings`
- Replace all usages of `Immutable*` with `IntegrationSettings`
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization
- Reorder initialization of `DisabledIntegrationNames` in
`TracerSettings` so that it can be used in the `IntegrationSettings`
constructor

## Test coverage

All covered by existing details

## Other details

Part of Stack
- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399
- #6400
- #6405 👈 This PR
- #6408
- #6415
veerbia pushed a commit that referenced this pull request Dec 16, 2024
…terSettings` (#6408)

## Summary of changes

Merge `ExporterSettings` with `ImmutableExporterSettings`

## Reason for change

This stack of PRs is about doing one-shot configuration instead of
mutation. We never mutate these in the tracer after creation, so there's
no need for the separate types.

## Implementation details

- Made the properties in `ExporterSettings` get-only. This required
quite a lot of work because we were doing a lot of mutating of the
settings in the "helper" functions.
- I only _lightly_ refactored those methods (as much as possible) to
avoid setting the properties in the functions and instead returning the
details to set later.
- These are prime candidates for some _much_ heavier refactoring later,
but I didn't want to get bogged down with that in this PR
- Replace all usages of `Immutable*` with `ExporterSettings`
- Replace usages of `AgentUriInternal` with `AgentUri`
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization

## Test coverage

All covered by existing details

## Other details

Part of Stack
- #6370
- #6376
- #6385
- #6386
- #6397 
- #6399
- #6400
- #6405
- #6408 👈 This PR
- #6415
veerbia pushed a commit that referenced this pull request Dec 16, 2024
…ettings` (#6415)

Merge `TracerSettings` with `ImmutableTracerSettings`

This stack of PRs is about doing one-shot configuration instead of
mutation. We never mutate these in the tracer after creation, so there's
no need for the separate types.

- Make the properties in `TracerSettings` get-only.
- Make the collections in `TracerSettings` readonly.
- Move logic that used to be in the constructor of
`ImmutableTracerSettings` into `TracerSettings`
- e.g. Service/Version/Env were being changed based on DD_TAGS values.
Moved that to TracerSettings and (importantly) added missing telemetry
recording of these values.
  - Added missing recording of _effective_ `DisabledInstegrations`
- Moving this logic caused some _tests_ to be broken (checking default
values). Updated the expected values of those tests in a single
- Replace all usages of `ImmutableTracerSettings` with `TracerSettings`
- Move `ITracer` to Datadog.Trace.Manual
- It's only used there, and references the manual-version of
`ImmutableTracerSettings` which we _want_ to keep.
- Move `Immutable*Tests` into appropriate file and tweak
- Replace mutations with initialization in tests

All covered by existing tests (I hope) 🤞

There's still a _lot_ of scope to improve this

Part of Stack
- #6370
- #6376
- #6385
- #6386
- #6397
- #6399
- #6400
- #6405
- #6408
- #6415 👈 This PR
andrewlock added a commit that referenced this pull request Jan 9, 2025
## Summary of changes

Adds some more tests for CI Visibility configuration

## Reason for change

In #6399 we changed how this all worked. We _thought_ we had a
regression, but it seems that we fixed a bug. This adds tests to confirm
the CI Vis behaviour is as expected

## Implementation details

Adds some unit tests

## Test coverage

More
chojomok pushed a commit that referenced this pull request Jul 15, 2025
## Summary of changes

Adds some more tests for CI Visibility configuration

## Reason for change

In #6399 we changed how this all worked. We _thought_ we had a
regression, but it seems that we fixed a bug. This adds tests to confirm
the CI Vis behaviour is as expected

## Implementation details

Adds some unit tests

## Test coverage

More
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci-visibility area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) type:refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants