Skip to content

NetworkChange.NetworkAddressChanged event leaks threads on Linux #63788

@vdjuric

Description

@vdjuric

Description

When you enable HTTP/3 support on Linux and repeatedly get HTTP/2 or HTTP3 web page, ".NET Network Address Change" threads will start to accumulate.

It seems network address change event is triggered from https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs with call _poolManager.StartMonitoringNetworkChanges();

NetworkAddressChange.Unix.cs will start creating threads with name ".NET Network Address Change":
https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs

Reproduction Steps


.NET 6 console project, source code example:

using System.Diagnostics;

static async Task ThreadDiagnostics()
{
    var pid = Process.GetCurrentProcess().Id;
    var psi = new ProcessStartInfo("ps", $"-T -p {pid}");
    psi.RedirectStandardOutput = true;

    using var process = Process.Start(psi);
    if (process == null)
    {
        throw new Exception("Could not create process");
    }

    int threadCountNetworkAddressChange = 0;
    var output = await process.StandardOutput.ReadToEndAsync();
    using (var sr = new StringReader(output))
    {
        while (true)
        {
            var line = await sr.ReadLineAsync();
            if (line == null)
            {
                break;
            }

            if (line.IndexOf(".NET Network Ad") > 0)
            {
                threadCountNetworkAddressChange++;
            }

            //NOTE:
            //we are searching for threads containing ".NET Network Ad"

            //new Thread(s => LoopReadSocket((int)s!))
            //{
            //    IsBackground = true,
            //    Name = ".NET Network Address Change"
            //}.UnsafeStart(newSocket);
            //
            //https://source.dot.net/#System.Net.NetworkInformation/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs
        }
    }

    //Console.WriteLine(output);
    Console.WriteLine($"{DateTime.Now} '.NET Network Address Change' threads: {threadCountNetworkAddressChange}");
}

for (var i = 0; i < 60; i++)
{
    try
    {
        using (var client = new HttpClient())
        {
            client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact;
            using var req = new HttpRequestMessage(HttpMethod.Get, "https://cloudflare-quic.com/"); //HTTP/2 or HTTP/3 enabled web server
            req.Version = System.Net.HttpVersion.Version20;

            using var res = await client.SendAsync(req);
            res.EnsureSuccessStatusCode();
        }

        await ThreadDiagnostics();
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Unhandled exception: {ex}");
    }

    await Task.Delay(1000);
}

Dockerfile with installed libmsquic and required tools (procps, curl)

FROM mcr.microsoft.com/dotnet/runtime:6.0-bullseye-slim AS base
WORKDIR /app

#install some tools
RUN apt-get update && apt-get install -y procps curl

#install libmsquic
#https://docs.microsoft.com/en-us/dotnet/core/extensions/httpclient-http3
RUN curl https://packages.microsoft.com/debian/11/prod/pool/main/libm/libmsquic/libmsquic_1.9.0_amd64.deb --output libmsquic.deb
RUN dpkg -i libmsquic.deb

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["HttpClientMemoryLeak/HttpClientMemoryLeak.csproj", "HttpClientMemoryLeak/"]
RUN dotnet restore "HttpClientMemoryLeak/HttpClientMemoryLeak.csproj"
COPY . .
WORKDIR "/src/HttpClientMemoryLeak"
RUN dotnet build "HttpClientMemoryLeak.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "HttpClientMemoryLeak.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "HttpClientMemoryLeak.dll"]

Expected behavior

".NET Network Address Change" should not accumulate, especially if HTTP connections are not pooled.

Check https://source.dot.net/#System.Net.Http/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs

bool avoidStoringConnections =
                settings._maxConnectionsPerServer == int.MaxValue &&
                (settings._pooledConnectionIdleTimeout == TimeSpan.Zero ||
                 settings._pooledConnectionLifetime == TimeSpan.Zero);

Actual behavior

".NET Network Address Change" threads will start to accumulate and consume resources / memory.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions