Skip to content

[Tracer] Skip inserting the startup hook into methods in the type Costura.AssemblyLoader#5910

Merged
zacharycmontoya merged 1 commit into
masterfrom
zach.montoya/fix-for-costura
Aug 16, 2024
Merged

[Tracer] Skip inserting the startup hook into methods in the type Costura.AssemblyLoader#5910
zacharycmontoya merged 1 commit into
masterfrom
zach.montoya/fix-for-costura

Conversation

@zacharycmontoya

@zacharycmontoya zacharycmontoya commented Aug 16, 2024

Copy link
Copy Markdown
Contributor

Summary of changes

For automatic instrumentation, the tracer inserts a startup hook inside the first method that we deem as a "valid" startup hook callsite. This change adds logic so that methods inside the Costura.AssemblyLoader type are not "valid" startup hook callsites, so that the insertion of the startup hook can be delayed to a later method call.

Reason for change

We've seen a case where a WCF client application uses Costura.Fody, and when instrumentation is added to the application it begins running into assembly loading issues with the following exception:

System.Configuration.ConfigurationErrorsException: Configuration binding extension 'system.serviceModel/bindings/basicHttpsBinding' could not be found. Verify that this binding extension is properly registered in system.serviceModel/extensions/bindingExtensions and that it is spelled correctly. (C:\scratch\APMS-00000\WcfServiceLibrary1\GettingStartedClient\bin\Debug\GettingStartedClient.exe.Config line 8)

   at System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult) in System.Configuration\BaseConfigurationRecord.cs:line 1379
   at System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1314
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1157
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1157
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1157
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey, Boolean getLkg, Boolean checkPermission) in System.Configuration\BaseConfigurationRecord.cs:line 1000
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey) in System.Configuration\BaseConfigurationRecord.cs:line 579
   at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection(String sectionName) in System.Configuration\ClientConfigurationSystem.cs:line 179
   at System.Configuration.ConfigurationManager.GetSection(String sectionName) in System.Configuration\ConfigurationManager.cs:line 171
   at System.ServiceModel.Activation.AspNetEnvironment.UnsafeGetSectionFromConfigurationManager(String sectionPath) in System.ServiceModel.Activation\AspNetEnvironment.cs:line 210
   at System.ServiceModel.Activation.AspNetEnvironment.UnsafeGetConfigurationSection(String sectionPath) in System.ServiceModel.Activation\AspNetEnvironment.cs:line 198
   at System.ServiceModel.Configuration.ConfigurationHelpers.UnsafeGetAssociatedSection(ContextInformation evalContext, String sectionPath) in System.ServiceModel.Configuration\ConfigurationHelpers.cs:line 141
   at System.ServiceModel.Configuration.ConfigurationHelpers.UnsafeGetSection(String sectionPath) in System.ServiceModel.Configuration\ConfigurationHelpers.cs:line 194
   at System.ServiceModel.Configuration.ClientSection.UnsafeGetSection() in System.ServiceModel.Configuration\ClientSection.cs:line 49
   at System.ServiceModel.Description.ConfigLoader.LookupChannel(ContextInformation configurationContext, String configurationName, ContractDescription contract, EndpointAddress address, Boolean wildcard, Boolean useChannelElementKind, ServiceEndpoint& serviceEndpoint) in System.ServiceModel.Description\ConfigLoader.cs:line 1034
   at System.ServiceModel.Description.ConfigLoader.LookupEndpoint(String configurationName, EndpointAddress address, ContractDescription contract, ContextInformation configurationContext) in System.ServiceModel.Description\ConfigLoader.cs:line 495
   at System.ServiceModel.Description.ConfigLoader.LookupEndpoint(String configurationName, EndpointAddress address, ContractDescription contract) in System.ServiceModel.Description\ConfigLoader.cs:line 488
   at System.ServiceModel.ChannelFactory.InitializeEndpoint(String configurationName, EndpointAddress address) in System.ServiceModel\ChannelFactory.cs:line 359
   at System.ServiceModel.ChannelFactory`1..ctor(String endpointConfigurationName, EndpointAddress remoteAddress) in System.ServiceModel\ChannelFactory.cs:line 653
   at System.ServiceModel.ChannelFactory`1..ctor(String endpointConfigurationName) in System.ServiceModel\ChannelFactory.cs:line 632
   at System.ServiceModel.ConfigurationEndpointTrait`1.CreateSimplexFactory() in System.ServiceModel\ConfigurationEndpointTrait.cs:line 78
   at System.ServiceModel.ConfigurationEndpointTrait`1.CreateChannelFactory() in System.ServiceModel\ConfigurationEndpointTrait.cs:line 60
   at System.ServiceModel.ClientBase`1.CreateChannelFactoryRef(EndpointTrait`1 endpointTrait) in System.ServiceModel\ClientBase.cs:line 1755
   at System.ServiceModel.ClientBase`1.InitializeChannelFactoryRef() in System.ServiceModel\ClientBase.cs:line 1749
   at System.ServiceModel.ClientBase`1..ctor() in System.ServiceModel\ClientBase.cs:line 1088
   at GettingStartedClient.ServiceReference1.CalculatorClient..ctor() in C:\scratch\APMS-00000\WcfServiceLibrary1\GettingStartedClient\Connected Services\ServiceReference1\Reference.cs:line 51
   at GettingStartedClient.Program.Main(String[] args) in C:\scratch\APMS-00000\WcfServiceLibrary1\GettingStartedClient\Program.cs:line 21

The issue, it appears, is that the Datadog tracing startup hook is being inserted into either the Costura.AssemblyLoader..cctor() or the Costura.AssemblyLoader.Attach() methods, which Costura adds into the module initializer of the target assembly. These methods add an AssemblyResolve event handler so that the application can load dependent assemblies from the assembly resources, since they are no longer on disk. Since the tracing startup hook also adds an AssemblyResolve event handler, I experimented with delaying the startup hook insertion so that the Costura AssemblyResolve event handler would be added first and it....worked.

Implementation details

Adds more logic to the startup hook logic to check against the Costura.AssemblyLoader type when considering if a callsite is valid for startup hook insertion.

Test coverage

I was able to reproduce the issue by:

  • Setting up a WCF client application and WCF server application using the following Getting Started tutorial
  • Updating endpoint with binding=basicHttpsBinding (and using a port on my machine with proper HTTPS permissions)
  • Adding the Costura.Fody v4.0.0 NuGet package to the client application with an empty FodyWeavers.xml
<Weavers>
  <Costura/>
</Weavers>
  • Running the WCF client application
  • Add Datadog .NET automatic instrumentation

Before the changes, the application would crash with the above error. After the changes, the application runs without issue.

If needed, I can check in a smoke test but to avoid repository bloat I figured the step-by-step above would be sufficient.

Other details

Internal ticket: APMS-12728

@zacharycmontoya zacharycmontoya self-assigned this Aug 16, 2024
@zacharycmontoya
zacharycmontoya requested a review from a team as a code owner August 16, 2024 00:06
@datadog-ddstaging

datadog-ddstaging Bot commented Aug 16, 2024

Copy link
Copy Markdown

Datadog Report

Branch report: zach.montoya/fix-for-costura
Commit report: b79e27a
Test service: dd-trace-dotnet

✅ 0 Failed, 352321 Passed, 2277 Skipped, 23h 37m 44.15s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

  • NoExceptions - Datadog.Trace.ClrProfiler.IntegrationTests.SmokeTests.RuntimeMetricsShutdownSmokeTest - Last Failure

    Expand for error
     Expected no errors in smoke test: Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
        at System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan\`1<Byte>)
        at System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent(UInt32, UInt32, System.DateTime, System.Guid, System.Guid, System.ReadOnlySpan\`1<Byte>)
        at System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object)
        at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
        at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
        at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
        at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
     ...
    

@andrewlock

Copy link
Copy Markdown
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I agree it's probably unnecessary to have a smoke test here to avoid bloat, but I wonder if it's worth having a regression example somewhere for cases like this 🤔 I'm undecided 🤷‍♂️

@bouwkast bouwkast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow nice!

@zacharycmontoya
zacharycmontoya merged commit 338ab55 into master Aug 16, 2024
@zacharycmontoya
zacharycmontoya deleted the zach.montoya/fix-for-costura branch August 16, 2024 19:43
@github-actions github-actions Bot added this to the vNext-v3 milestone Aug 16, 2024
lucaspimentel pushed a commit that referenced this pull request Aug 23, 2024
…tura.AssemblyLoader (#5910)

For automatic instrumentation, the tracer inserts a startup hook inside the first method that we deem as a "valid" startup hook callsite. This change adds logic so that methods inside the `Costura.AssemblyLoader` type are not "valid" startup hook callsites, so that the insertion of the startup hook can be delayed to a later method call.
@lucaspimentel lucaspimentel changed the title [Tracer] Skip inserting the startup hook into methods in the type Costura.AssemblyLoader [Tracer] Skip inserting the startup hook into methods in the type Costura.AssemblyLoader Aug 23, 2024
@lucaspimentel lucaspimentel added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:native-library Automatic instrumentation native C++ code (Datadog.Trace.ClrProfiler.Native) labels Aug 23, 2024
andrewlock pushed a commit that referenced this pull request Aug 23, 2024
…stura.AssemblyLoader` (#5910 -> v2) (#5934)

## Summary of changes
For automatic instrumentation, the tracer inserts a startup hook inside
the first method that we deem as a "valid" startup hook callsite. This
change adds logic so that methods inside the `Costura.AssemblyLoader`
type are not "valid" startup hook callsites, so that the insertion of
the startup hook can be delayed to a later method call.

## Reason for change
We've seen a case where a WCF client application uses
[Costura.Fody](https://github.com/Fody/Costura), and when
instrumentation is added to the application it begins running into
assembly loading issues with the following exception:

```
System.Configuration.ConfigurationErrorsException: Configuration binding extension 'system.serviceModel/bindings/basicHttpsBinding' could not be found. Verify that this binding extension is properly registered in system.serviceModel/extensions/bindingExtensions and that it is spelled correctly. (C:\scratch\APMS-00000\WcfServiceLibrary1\GettingStartedClient\bin\Debug\GettingStartedClient.exe.Config line 8)

   at System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult) in System.Configuration\BaseConfigurationRecord.cs:line 1379
   at System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1314
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1157
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1157
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) in System.Configuration\BaseConfigurationRecord.cs:line 1157
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey, Boolean getLkg, Boolean checkPermission) in System.Configuration\BaseConfigurationRecord.cs:line 1000
   at System.Configuration.BaseConfigurationRecord.GetSection(String configKey) in System.Configuration\BaseConfigurationRecord.cs:line 579
   at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection(String sectionName) in System.Configuration\ClientConfigurationSystem.cs:line 179
   at System.Configuration.ConfigurationManager.GetSection(String sectionName) in System.Configuration\ConfigurationManager.cs:line 171
   at System.ServiceModel.Activation.AspNetEnvironment.UnsafeGetSectionFromConfigurationManager(String sectionPath) in System.ServiceModel.Activation\AspNetEnvironment.cs:line 210
   at System.ServiceModel.Activation.AspNetEnvironment.UnsafeGetConfigurationSection(String sectionPath) in System.ServiceModel.Activation\AspNetEnvironment.cs:line 198
   at System.ServiceModel.Configuration.ConfigurationHelpers.UnsafeGetAssociatedSection(ContextInformation evalContext, String sectionPath) in System.ServiceModel.Configuration\ConfigurationHelpers.cs:line 141
   at System.ServiceModel.Configuration.ConfigurationHelpers.UnsafeGetSection(String sectionPath) in System.ServiceModel.Configuration\ConfigurationHelpers.cs:line 194
   at System.ServiceModel.Configuration.ClientSection.UnsafeGetSection() in System.ServiceModel.Configuration\ClientSection.cs:line 49
   at System.ServiceModel.Description.ConfigLoader.LookupChannel(ContextInformation configurationContext, String configurationName, ContractDescription contract, EndpointAddress address, Boolean wildcard, Boolean useChannelElementKind, ServiceEndpoint& serviceEndpoint) in System.ServiceModel.Description\ConfigLoader.cs:line 1034
   at System.ServiceModel.Description.ConfigLoader.LookupEndpoint(String configurationName, EndpointAddress address, ContractDescription contract, ContextInformation configurationContext) in System.ServiceModel.Description\ConfigLoader.cs:line 495
   at System.ServiceModel.Description.ConfigLoader.LookupEndpoint(String configurationName, EndpointAddress address, ContractDescription contract) in System.ServiceModel.Description\ConfigLoader.cs:line 488
   at System.ServiceModel.ChannelFactory.InitializeEndpoint(String configurationName, EndpointAddress address) in System.ServiceModel\ChannelFactory.cs:line 359
   at System.ServiceModel.ChannelFactory`1..ctor(String endpointConfigurationName, EndpointAddress remoteAddress) in System.ServiceModel\ChannelFactory.cs:line 653
   at System.ServiceModel.ChannelFactory`1..ctor(String endpointConfigurationName) in System.ServiceModel\ChannelFactory.cs:line 632
   at System.ServiceModel.ConfigurationEndpointTrait`1.CreateSimplexFactory() in System.ServiceModel\ConfigurationEndpointTrait.cs:line 78
   at System.ServiceModel.ConfigurationEndpointTrait`1.CreateChannelFactory() in System.ServiceModel\ConfigurationEndpointTrait.cs:line 60
   at System.ServiceModel.ClientBase`1.CreateChannelFactoryRef(EndpointTrait`1 endpointTrait) in System.ServiceModel\ClientBase.cs:line 1755
   at System.ServiceModel.ClientBase`1.InitializeChannelFactoryRef() in System.ServiceModel\ClientBase.cs:line 1749
   at System.ServiceModel.ClientBase`1..ctor() in System.ServiceModel\ClientBase.cs:line 1088
   at GettingStartedClient.ServiceReference1.CalculatorClient..ctor() in C:\scratch\APMS-00000\WcfServiceLibrary1\GettingStartedClient\Connected Services\ServiceReference1\Reference.cs:line 51
   at GettingStartedClient.Program.Main(String[] args) in C:\scratch\APMS-00000\WcfServiceLibrary1\GettingStartedClient\Program.cs:line 21
```

The issue, it appears, is that the Datadog tracing startup hook is being
inserted into either the `Costura.AssemblyLoader..cctor()` or the
`Costura.AssemblyLoader.Attach()` methods, which Costura adds into the
module initializer of the target assembly. These methods add an
AssemblyResolve event handler so that the application can load dependent
assemblies from the assembly resources, since they are no longer on
disk. Since the tracing startup hook also adds an AssemblyResolve event
handler, I experimented with delaying the startup hook insertion so that
the Costura AssemblyResolve event handler would be added first and
it....worked.

## Implementation details
Adds more logic to the startup hook logic to check against the
`Costura.AssemblyLoader` type when considering if a callsite is valid
for startup hook insertion.

## Test coverage
I was able to reproduce the issue by:
- Setting up a WCF client application and WCF server application using
the following [Getting
Started](https://learn.microsoft.com/en-us/dotnet/framework/wcf/getting-started-tutorial)
tutorial
- Updating endpoint with `binding=basicHttpsBinding` (and using a port
on my machine with proper HTTPS permissions)
- Adding the `Costura.Fody` v4.0.0 NuGet package to the client
application with an empty FodyWeavers.xml
```
<Weavers>
  <Costura/>
</Weavers>
```
- Running the WCF client application
- Add Datadog .NET automatic instrumentation

Before the changes, the application would crash with the above error.
After the changes, the application runs without issue.

If needed, I can check in a smoke test but to avoid repository bloat I
figured the step-by-step above would be sufficient.

## Other details

Internal ticket: APMS-12728
Backporting #5910 from 3.x to 2.x

Co-authored-by: Zach Montoya <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:native-library Automatic instrumentation native C++ code (Datadog.Trace.ClrProfiler.Native) area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants