Kafka AppHost health checks leak producer threads and cause high CPU #20091

Description

Aspire.Hosting.Kafka creates a new KafkaHealthCheck on every health-check execution. Each check creates and retains a Confluent Kafka producer, but the check is returned directly from the registration factory rather than resolved from DI. Neither the health-check service nor DI disposes these instances.

Over time, producer polling threads accumulate in the AppHost. We observed sustained AppHost CPU usage of roughly 350–400% on macOS, with a later observation near 700%.

Profiling evidence

A managed thread-sampling trace of the AppHost contained:

• 1,934 distinct sampled threads.
• 1,901 threads with Confluent.Kafka.Impl.SafeKafkaHandle.Poll on their stacks.
• The caller was Confluent.Kafka.Producer<TKey, TValue>.<StartPollTask>b__0.
• Kafka health-check execution was also present in the trace.

These are thread-time samples, including time inside native calls, not a measurement that Kafka consumed that percentage of CPU. The thread accumulation and disposal leak are confirmed; the exact native CPU contribution was not captured.

Isolated verification

Tested with Aspire.Hosting.Kafka 13.5.2, independently of any application workload:

1. Build an AppHost with two AddKafka resources and distinct allocated loopback endpoints.
2. Publish ConnectionStringAvailableEvent for each resource.
3. Invoke the registered checks through HealthCheckService four times per resource.
4. Track distinct KafkaHealthCheck instances and inspect their producers after disposing the AppHost.

No containers or Kafka brokers were started. The checks exercised the unavailable-broker path with bounded timeouts. All leaked producers were explicitly disposed by the test afterward.

The keyed-singleton comparison also preserved distinct bootstrap-server settings for the two resources.

Cause

The registration in src/Aspire.Hosting.Kafka/KafkaBuilderExtensions.cs returns new KafkaHealthCheck(options) directly from HealthCheckRegistration.Factory.

KafkaHealthCheck lazily creates a producer and disposes it in Dispose(). The health-check runner creates and disposes a DI scope, but directly constructed factory results are not tracked by that scope for disposal.

The same registration pattern remains on main at 47a0448248.

Suggested fix

Register KafkaHealthCheck as a DI-owned keyed singleton, keyed by the resource's health-check name. Resolve that instance from HealthCheckRegistration.Factory:

builder.Services.AddKeyedSingleton<KafkaHealthCheck>(healthCheckKey, (sp, _) =>
{
    var options = new KafkaHealthCheckOptions();
    options.Configuration = new ProducerConfig();
    options.Configuration.BootstrapServers = connectionString
        ?? throw new InvalidOperationException("Connection string is unavailable");
    return new KafkaHealthCheck(options);
});

var healthCheckRegistration = new HealthCheckRegistration(
    healthCheckKey,
    sp => sp.GetRequiredKeyedService<KafkaHealthCheck>(healthCheckKey),
    failureStatus: default,
    tags: default);

Regression coverage should verify repeated checks reuse the instance, different Kafka resources retain separate configurations, and AppHost disposal disposes the producers.

Related reports

• Memory Leak When Enabling Health Checks in Aspire.Confluent.Kafka #6035 reported the same lifetime leak in the client integration.
• Resolve KafkaHealthCheck from container (#6035) #6051 fixed the producer/consumer client integrations using DI-owned health checks, but did not change the hosting integration.
• Allow registering multiple Kafka health checks. Xabaril/AspNetCore.Diagnostics.HealthChecks#2298 explains the need to keep health-check configuration separate for multiple Kafka resources.