Add synchronous mode for serverless environments#227
Conversation
In AWS Lambda and similar serverless environments, background threads get frozen when the handler returns, causing buffered metrics to be lost. This adds a SynchronousMode option that bypasses the async queue and sends metrics directly on the calling thread. Changes: - Extract IStatsBufferize interface from StatsBufferize - Add SynchronousSender that routes metrics synchronously - Add StatsdConfig.SynchronousMode property (default false) - Suppress telemetry background timer in sync mode - Add unit tests, builder tests, and integration tests
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a138a5f742
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
In the async path, AsynchronousWorker.Dequeue() catches exceptions from Route/Flush and forwards them to optionalExceptionHandler. The sync path was letting exceptions propagate directly into user code, which could crash Lambda handlers on transport failures or serialization errors. Wrap Send and Flush in try-catch to match async behavior.
StephenWakely
left a comment
There was a problem hiding this comment.
Nice. I think this can definitely be useful! I just have some minor nits on the naming.
| /// <summary> | ||
| /// IStatsBufferize defines the contract for sending stats to the pipeline. | ||
| /// </summary> | ||
| internal interface IStatsBufferize : IDisposable |
There was a problem hiding this comment.
The naming of this interface is confusing. It doesn't really capture what it is trying to do. I think IStatsSender would make more sense.
Then perhaps SynchronousSender and AsynchronousBufferizedSender implementing them.
There was a problem hiding this comment.
Renamed as suggested: IStatsBufferize → IStatsSender, StatsBufferize → AsynchronousBufferizedSender. Also renamed the factory types for consistency (IStatsBufferizeFactory → IStatsSenderFactory, StatsBufferizeFactory → StatsSenderFactory).
| } | ||
| catch (Exception e) | ||
| { | ||
| _optionalExceptionHandler?.Invoke(e); |
There was a problem hiding this comment.
I don't think it is a good idea to swallow the exceptions if a handler isn't specified.
I understand the purpose, given asynchronous does hide the exceptions, it does mean you can seamlessly swap between synchronous and asynchronous. But I imagine someone using a synchronous sender would expect the exceptions to be surfaced right away.
There was a problem hiding this comment.
Thinking about it, to keep consistent with the async version, we should have the optionalExceptionHandler here - but if it isn't defined rethrow the exception rather than swallow it.
There was a problem hiding this comment.
Updated: when optionalExceptionHandler is null, exceptions are now rethrown instead of swallowed. When a handler is provided, the existing behavior is preserved (caught and forwarded to the handler). Note that through the normal DogStatsdService.Configure path, the handler is always set — it defaults to Debug.WriteLine when the caller does not provide one — so in practice exceptions are always caught and logged. The rethrow is a safety net for direct construction without a handler. Added a test to verify the rethrow behavior.
Rename IStatsBufferize -> IStatsSender, StatsBufferize -> AsynchronousBufferizedSender, and factory types for consistency per reviewer suggestion. Update SynchronousSender to rethrow exceptions when no handler is configured instead of silently swallowing them.
Summary
StatsdConfig.SynchronousModeoption that bypasses the async background worker and sends metrics directly on the calling threadIStatsSenderinterface to enable swapping async/sync implementationsStatsBufferizetoAsynchronousBufferizedSenderfor clarityMotivation
In AWS Lambda and similar serverless environments, the runtime freezes the sandbox immediately after the handler returns. The DogStatsD client processes metrics on a background thread via
AsynchronousWorker, so buffered metrics can be lost when the freeze hits. The existingFlush()method has a hardcoded 3-second timeout and still relies on a background thread to drain the queue — making it fundamentally unreliable in serverless contexts.SynchronousModeeliminates this class of bugs entirely: no background threads, no async queue, no race with the sandbox freeze. Metrics are routed and serialized on the calling thread, batched into efficient UDP packets via the existingBufferBuilder, and the user callsFlush()at the end of their handler to send any remaining partial buffer.Usage
Test plan
SynchronousSender(pool behavior, send/flush, batching, auto-flush on full buffer, dispose triggers flush, thread safety, empty flush, exception forwarding to handler, exception rethrow when no handler)CreateAsynchronousBufferizedSender, passessynchronousModeto telemetry)