Description

IncrementingPollingCounter uses a callback to retrieve current values of some metric and reports it via EventSource events. In the past the first invocation of the callback may have occurred synchronously on whatever thread was enabling the EventSource and future invocations occurred on a dedicated timer thread. Starting in .NET 9 the first callback always occurs asynchronously on the timer thread. This may result in counter changes that occurred just after the counter was enabled going unobserved because the first callback happens later.

This change appears most likely to impact tests that use EventListener to validate an IncrementingPollingCounter. If tests enable the counter and then immediately modify the state that is being polled by the counter, that modification may now be occurring prior to the first time the callback is invoked and go unnoticed.

Version

.NET 9 RC 1

Previous behavior

When an IncrementingPollingCounter was enabled, the first invocation of the callback may have occurred synchronously on the thread that did the enable operation.

This sample app will call the delegate () => SomeInterestingValue on the Main thread within the call to EnableEvents(). That callback will observe log.SomeInterestingValue is 0. A later call from a dedicated timer thread will observe log.SomeInterestingValue changed to 1 and an event will be sent with Increment value = 1.

using System.Diagnostics.Tracing;

var log = MyEventSource.Log;
using var listener = new Listener();

log.SomeInterestingValue++;

Console.ReadKey();

class MyEventSource : EventSource
{
    public static MyEventSource Log { get; } = new();
    private IncrementingPollingCounter? _counter;
    public int SomeInterestingValue;

    private MyEventSource() : base(nameof(MyEventSource)) 
    {
        _counter = new IncrementingPollingCounter("counter", this, () => SomeInterestingValue);
    }
}

class Listener : EventListener
{
    protected override void OnEventSourceCreated(EventSource eventSource)
    {
        if (eventSource.Name == nameof(MyEventSource))
        {
            EnableEvents(eventSource, EventLevel.Informational, EventKeywords.None,
                new Dictionary<string, string?> { { "EventCounterIntervalSec", "1.0" } });
        }
    }

    protected override void OnEventWritten(EventWrittenEventArgs eventData)
    {
        if (eventData.EventSource.Name == "EventCounters")
        {
            var counters = (IDictionary<string, object>)eventData.Payload![0]!;
            Console.WriteLine($"Increment: {counters["Increment"]}");
        }
    }
}

New behavior

Using the same code snippet as above now the first invocation of the callback occurs asynchronously on the timer thread. It may or may not occur prior to the Main thread running log.SomeInterestingValue++ depending on how the OS schedules multiple threads.

Depending on that timing the app will either output "Increment=0" or "Increment=1".

Type of breaking change

• Binary incompatible: Existing binaries might encounter a breaking change in behavior, such as failure to load or execute, and if so, require recompilation.
• Source incompatible: When recompiled using the new SDK or component or to target the new runtime, existing source code might require source changes to compile successfully.
• Behavioral change: Existing binaries might behave differently at run time.

Reason for change

The change was made to resolve a potential deadlock that can occur running callback functions while the EventListener lock is held. dotnet/runtime#93175

Recommended action

• For scenarios that use IncrementingPollingCounters to visualize metrics in external monitoring tools no action is necessary - these should continue working normally.

• For scenarios that do in-process testing or other consumption of counter data via EventListener, check if your code expects to observe a specific modification to the counter value made on the same thread that called EnableEvents(). If it does we recommend waiting to observe at least one counter event from the EventListener first, then modify the counter value. For example in the code snippet above to ensure that it will print Increment=1 you could add a ManualResetEvent to the EventListener, signal it when the first counter event is received and wait for it prior to calling log.SomeInterestingValue++

Feature area

Core .NET libraries

Affected APIs

IncrementingPollingCounter

Associated WorkItem - 297409