More remote config performance improvements#8241
Conversation
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8241) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (78ms) : 74, 82
master - mean (75ms) : 73, 77
section Bailout
This PR (8241) - mean (83ms) : 79, 86
master - mean (79ms) : 77, 81
section CallTarget+Inlining+NGEN
This PR (8241) - mean (1,116ms) : 1069, 1164
master - mean (1,086ms) : 1041, 1132
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (120ms) : 116, 125
master - mean (116ms) : 113, 119
section Bailout
This PR (8241) - mean (122ms) : 118, 126
master - mean (117ms) : 114, 121
section CallTarget+Inlining+NGEN
This PR (8241) - mean (785ms) : 734, 836
master - mean (776ms) : 715, 837
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (107ms) : 103, 111
master - mean (104ms) : 100, 108
section Bailout
This PR (8241) - mean (108ms) : 106, 110
master - mean (106ms) : 103, 108
section CallTarget+Inlining+NGEN
This PR (8241) - mean (783ms) : 712, 854
master - mean (765ms) : 695, 835
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (106ms) : 102, 109
master - mean (103ms) : 99, 106
section Bailout
This PR (8241) - mean (108ms) : 105, 110
master - mean (104ms) : 102, 106
section CallTarget+Inlining+NGEN
This PR (8241) - mean (707ms) : 685, 728
master - mean (681ms) : 655, 707
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (198ms) : 192, 203
master - mean (193ms) : 189, 198
section Bailout
This PR (8241) - mean (203ms) : 197, 210
master - mean (197ms) : 195, 199
section CallTarget+Inlining+NGEN
This PR (8241) - mean (1,178ms) : 1103, 1253
master - mean (1,164ms) : 1092, 1237
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (283ms) : 273, 293
master - mean (278ms) : 273, 284
section Bailout
This PR (8241) - mean (287ms) : 277, 297
master - mean (280ms) : 274, 287
section CallTarget+Inlining+NGEN
This PR (8241) - mean (955ms) : 916, 994
master - mean (948ms) : 906, 989
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (277ms) : 269, 284
master - mean (273ms) : 267, 279
section Bailout
This PR (8241) - mean (274ms) : 268, 281
master - mean (275ms) : 266, 284
section CallTarget+Inlining+NGEN
This PR (8241) - mean (946ms) : 914, 977
master - mean (936ms) : 903, 970
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8241) - mean (274ms) : 265, 284
master - mean (271ms) : 264, 277
section Bailout
This PR (8241) - mean (275ms) : 268, 282
master - mean (272ms) : 268, 276
section CallTarget+Inlining+NGEN
This PR (8241) - mean (846ms) : 817, 875
master - mean (835ms) : 815, 855
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
5723a48 to
1f2fe4b
Compare
33707ed to
c42e395
Compare
BenchmarksBenchmark execution time: 2026-03-10 15:59:26 Comparing candidate commit 01edb31 in PR branch Found 13 performance improvements and 8 performance regressions! Performance is the same for 158 metrics, 13 unstable metrics. scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1
scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0
scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1
scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0
scenario:Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery netcoreapp3.1
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472
scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1
scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472
|
c42e395 to
bc2260d
Compare
bcb7f2b to
01edb31
Compare
bouwkast
left a comment
There was a problem hiding this comment.
😅
There is a lot here I think it looks good though from what I can see
I thought that we can't swap to ulong compared to BigInteger thought that was an issue a while back? Maybe misremembering
|
|
||
| var remoteConfigurationCache = new RemoteConfigurationCache(remoteConfigurationPath, signedTarget.Length, signedTarget.Hashes, signedTarget.Custom.V); | ||
| _appliedConfigurations[remoteConfigurationCache.Path.Path] = remoteConfigurationCache; | ||
| var targetFile = response.TargetFiles?.FirstOrDefault(file => file.Path == remoteConfigurationPath.Path); |
There was a problem hiding this comment.
How long would TargetFiles be usually? It used to be a TryGetValue so wondering if this is where the increased time came?
There was a problem hiding this comment.
I tested this locally, by enabling ASM and debugging using
dd-trace run --set-env DD_TRACE_DEBUG=1 --set-env DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS=Serilog --set-env DD_APPSEC_ENABLED=1 --set-env DD_DYNAMIC_INSTRUMENTATION_ENABLED=1 --tracer-home C:\repos\dd-trace-dotnet-2\shared\bin\monitoring-home -- "C:\repos\test\logging_sample\bin\Debug\net5.0\publish\logging_sample.exe"
and the maximum size of target_files was "3" 😅 I expect it would likely get bigger than that if there's more being pushed via remote config, but I would be surprised to see more than 8, so I doubt this will be the source...
There was a problem hiding this comment.
It doesn't explain the benchmark regression though, because that was only 2/3 target files 😅
|
Just for some extra context, I did manual testing to look into this, and also some crude benchmarks #nullable enable
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.IO;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using Datadog.Trace.RemoteConfigurationManagement;
using Datadog.Trace.RemoteConfigurationManagement.Protocol;
using Datadog.Trace.RemoteConfigurationManagement.Protocol.Tuf;
using Datadog.Trace.Util;
using Datadog.Trace.Util.Json;
using Datadog.Trace.Util.Streams;
using Datadog.Trace.Vendors.Newtonsoft.Json;
namespace Benchmarks.Trace
{
/// <summary>
/// Benchmarks for the Remote Configuration Management polling pipeline.
/// Covers: request building, response processing (steady-state), and response deserialization.
/// </summary>
[MemoryDiagnoser, GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory), CategoriesColumn]
public class RemoteConfigBenchmark
{
private static readonly Task<GetRcmResponse?> NullResponseTask = Task.FromResult<GetRcmResponse?>(null);
private RcmSubscriptionManager _manager = null!;
private RcmClientTracer _rcmTracer = null!;
private Task<GetRcmResponse?> _steadyStateResponseTask = null!;
private byte[] _serializedResponseBytes = null!;
private byte[] _hugeSerializedResponseBytes = null!;
private GetRcmResponse? _response;
private string _pathToTest;
private string _result;
[GlobalSetup]
public void GlobalSetup()
{
_rcmTracer = RcmClientTracer.Create(
runtimeId: "benchmark-runtime-id",
tracerVersion: "2.55.0",
service: "benchmark-service",
env: "benchmark",
appVersion: "1.0.0",
globalTags: new ReadOnlyDictionary<string, string>(new Dictionary<string, string>
{
{ "team", "apm" },
{ "component", "benchmark" }
}),
processTags: null);
_manager = new RcmSubscriptionManager();
// Subscribe with a realistic subscription covering multiple products
var subscription = new Subscription(
(configs, removed) =>
{
var results = new List<ApplyDetails>();
foreach (var product in configs)
{
foreach (var config in product.Value)
{
results.Add(ApplyDetails.FromOk(config.Path.Path));
}
}
return results.ToArray();
},
RcmProducts.AsmFeatures,
RcmProducts.AsmDd);
_manager.SubscribeToChanges(subscription);
// Set capabilities as real code would
_manager.SetCapability(RcmCapabilitiesIndices.AsmActivation, true);
_manager.SetCapability(RcmCapabilitiesIndices.AsmIpBlocking, true);
_manager.SetCapability(RcmCapabilitiesIndices.AsmDdRules, true);
_manager.SetCapability(RcmCapabilitiesIndices.AsmExclusion, true);
_manager.SetCapability(RcmCapabilitiesIndices.ApmTracingSampleRate, true);
// Create initial response and prime the manager (applies all configs)
var initialResponse = CreateResponse(configCount: 5);
_manager.SendRequest(_rcmTracer, _ => Task.FromResult<GetRcmResponse?>(initialResponse))
.GetAwaiter().GetResult();
// Steady-state response: identical configs/hashes so ProcessResponse detects no changes
_steadyStateResponseTask = Task.FromResult<GetRcmResponse?>(CreateResponse(configCount: 5));
// Serialize for deserialization benchmark (includes base64-encoded TufRoot)
_serializedResponseBytes = Encoding.UTF8.GetBytes(
"""
{
"targets": "eyJzaWduZWQiOnsiX3R5cGUiOiJ0YXJnZXRzIiwiY3VzdG9tIjp7Im9wYXF1ZV9iYWNrZW5kX3N0YXRlIjoiZXlKbWIyOGlPaUFpWW1GeUluMD0ifSwiZXhwaXJlcyI6IjMwMDAtMDEtMDFUMDA6MDA6MDBaIiwic3BlY192ZXJzaW9uIjoiMS4wIiwidGFyZ2V0cyI6eyJkYXRhZG9nLzIvQVNNX0ZFQVRVUkVTL0FTTV9GRUFUVVJFUy1iYXNlL2NvbmZpZyI6eyJjdXN0b20iOnsidiI6Mn0sImhhc2hlcyI6eyJzaGEyNTYiOiJhMzhlYmY5ZmEyNTYwNzFmOTgyM2E1ZjUxMmE4NGU4YTc4NmM4ZGEyYjA3MTk0NTJkZWViNWRjMjg3ZWM5OTBmIn0sImxlbmd0aCI6NDh9LCJkYXRhZG9nLzIvQVNNX0ZFQVRVUkVTL0FTTV9GRUFUVVJFUy1zZWNvbmQvY29uZmlnIjp7ImN1c3RvbSI6eyJ2IjoxfSwiaGFzaGVzIjp7InNoYTI1NiI6IjkyMjFkZmQ5ZjYwODQxNTEzMTNlM2U0OTIwMTIxYWU4NDM2MTRjMzI4ZTQ2MzBlYTM3MWJhNjZlMmYxNWEwYTYifSwibGVuZ3RoIjo0N30sImRhdGFkb2cvMi9BU01fRkVBVFVSRVMvQVNNX0ZFQVRVUkVTLXRoaXJkL3Rlc3RuYW1lIjp7ImN1c3RvbSI6eyJ2IjoxfSwiaGFzaGVzIjp7InNoYTI1NiI6IjFlNGE3NWVkZmQ1ZjY1ZTBhZGVjNzA0Y2M3YzMyZjc5OTg3MTE3ZDg0NzU1NTQ0YWU0OTA1MDQ1Y2RiMGE0NDMifSwibGVuZ3RoIjo0MX0sImRhdGFkb2cvMi9ST0dVRV9QUk9EVUNUL215Y29uZmlnL2NvbmZpZyI6eyJjdXN0b20iOnsidiI6MX0sImhhc2hlcyI6eyJzaGEyNTYiOiI0NTliZDZhNDAzZDhkNTA5YzQ0M2Q3YjQ2NTVkMDI3YWRhM2NhZGUzM2U4YTM5ZjBkNTQ3ZDAyZjRkYjJkM2Q1In0sImxlbmd0aCI6OH0sImRhdGFkb2cvMi9ST0dVRV9QUk9EVUNUL215Y29uZmlnMi9jb25maWciOnsiY3VzdG9tIjp7InYiOjF9LCJoYXNoZXMiOnsic2hhMjU2IjoiYWZlZjkzNDJlNTZmNDQ2NTJhM2E5YWNkZjRiZGZlZGZlOGRhYjJlMDQ5Nzg4NzM0MTRmNjQzOTU0NDRmZjU0MSJ9LCJsZW5ndGgiOjEyfX0sInZlcnNpb24iOjEyfSwic2lnbmF0dXJlcyI6W3sia2V5aWQiOiJlZDc2NzJjOWEyNGFiZGE3ODg3MmVlMzJlZTcxYzdjYjFkNTIzNWU4ZGI0ZWNiZjFjYTI4YjljNTBlYjc1ZDllIiwic2lnIjoiNWY0YzQxOGQ1NWRhMGQ0ODU5YWIwNzM2MzUzODQ0NDAwMDczMjI1MTI3YjQxNzZmM2FhOGI1ZjViMWFkMDQxNGU4MTZiZWMyYWFiMGRmYTcyMzg0N2FmZDkzZmMxMTEzMmZiYjliMDY0Mzk0YWMwODYwZGNjZDcxZTgxNDc4MGYifV19",
"target_files": [
{
"path": "datadog/2/ASM_FEATURES/ASM_FEATURES-third/testname",
"raw": "ewogICJhc20iOiB7CiAgICAgICJlbmFibGVkIjogdHJ1ZQogIH0KfQo="
}
],
"client_configs": [
"datadog/2/ASM_FEATURES/ASM_FEATURES-base/config",
"datadog/2/ASM_FEATURES/ASM_FEATURES-second/config",
"datadog/2/ASM_FEATURES/ASM_FEATURES-third/testname"
]
}
""");
_hugeSerializedResponseBytes = Encoding.UTF8.GetBytes(RemoteConfigData.HugeConfig());
_pathToTest = "datadog/2/ASM_FEATURES/ASM_FEATURES-third/testname";
}
/// <summary>
/// Full poll cycle when nothing has changed (the most common real-world scenario).
/// Exercises BuildRequest + ProcessResponse: hash comparison, path parsing,
/// ToDictionary creation, List.Contains lookups, LINQ in Update().
/// </summary>
[Benchmark]
public Task PollSteadyState()
{
return _manager.SendRequest(_rcmTracer, _ => _steadyStateResponseTask);
}
/// <summary>
/// Full poll cycle when nothing has changed (the most common real-world scenario).
/// Exercises BuildRequest + ProcessResponse: hash comparison, path parsing,
/// ToDictionary creation, List.Contains lookups, LINQ in Update().
/// </summary>
[Benchmark]
public Task PollChanging()
{
return _manager.SendRequest(_rcmTracer, _ => _steadyStateResponseTask);
}
/// <summary>
/// Request building only (callback returns null so ProcessResponse is skipped).
/// Exercises: LINQ Select/ToList for hashes, RcmCachedTargetFile/RcmConfigState
/// allocation, GetCapabilities byte[] creation.
/// </summary>
[Benchmark]
public Task BuildRequest()
{
return _manager.SendRequest(_rcmTracer, _ => NullResponseTask);
}
}
} |
Summary of changes
A variety of minor performance improvements to remote config
Reason for change
I did some initial benchmarking of remote config, as well as running a test app with ASM/Debugger enabled (which use RCM), and the results weren't great. Given we make RCM requests every 5s, smallish changes here should add up, though the real-world effect will be tricky to gauge.
Implementation details
Most of the individual changes are small. In summary:
ExtraServicesProviderwill rarely see new service names, so cache the array (as the collection is append-only)BigInteger. We probably could/should useulonginstead but that I'll look at that in a separate PRBase64DecodingStream#8226, Add anIArrayPool<char>implementation for vendored Newtonsoft.JSON #8228)Additionally, I did a little bit of cleanup:
IRcmSubscriptionManagerinterface#nullable enableto the RCM types (and fix nullability where required)Test coverage
Mostly covered by existing unit tests, also did some manual testing. Finally, ran a few benchmarks, but it's a bit tricky to check reliably. This benchmark is benchmarking
_manager.SendRequest(_rcmTracer, _ => _steadyStateResponseTask)and passing the same response every time. In practice, the response changes every time, so this isn't strictly representative, but with the changes to request caching, this should actually mean our improvements are better than the original would be. The regression in the .NET duration is curious, but I'm not massively concerned, and the allocations are obviously down a lot at leastOther details
https://datadoghq.atlassian.net/browse/LANGPLAT-940
All part of the Remote Config perf stack. I think this is probably about the end of it for now