Recently we faced a problem with our DataDog metrics. We have several metrics we are sending from our ASP.Net Core app, and most of them are simple counter metrics. We call the IDogStatsd.Increment method. 99.999% of all metrics are OK and sent to DataDog without issues. But sometimes we generate a metric that doesn't fit the buffer, and then we get an InvalidOperationException.
See code here
|
throw new InvalidOperationException($"The metric size exceeds the buffer capacity {Capacity}: {serializedMetric.ToString()}"); |
It's not a problem by itself, and we've already fixed the issue on our side. But other counters using
IDogStatsd.Increment go nuts when it happens (all counter metrics are constantly increasing).
Looks like DogStatsD keeps trying to flush the metrics but keeps crashing on the broken one. The only way to solve this problem is to kill the service. I guess the broken metric should be excluded rather than retried every time. Looks like the problem started in version 7.0.0. Also, as the flush process runs asynchronously, we cannot handle this exception, making it hard to investigate the issue. DogStatsD doesn't return or write the exception to logs.
Recently we faced a problem with our DataDog metrics. We have several metrics we are sending from our ASP.Net Core app, and most of them are simple counter metrics. We call the
IDogStatsd.Incrementmethod. 99.999% of all metrics are OK and sent to DataDog without issues. But sometimes we generate a metric that doesn't fit the buffer, and then we get anInvalidOperationException.See code here
dogstatsd-csharp-client/src/StatsdClient/Bufferize/BufferBuilder.cs
Line 62 in 02d6a27
It's not a problem by itself, and we've already fixed the issue on our side. But other counters using
IDogStatsd.Incrementgo nuts when it happens (all counter metrics are constantly increasing).Looks like DogStatsD keeps trying to flush the metrics but keeps crashing on the broken one. The only way to solve this problem is to kill the service. I guess the broken metric should be excluded rather than retried every time. Looks like the problem started in version 7.0.0. Also, as the flush process runs asynchronously, we cannot handle this exception, making it hard to investigate the issue. DogStatsD doesn't return or write the exception to logs.