MessagePack has an internal pool of buffers, but they are thread-static. It's bad because:

• Given that we flush traces only every second, there's little chance that we end up using the same thread
• Every new buffer will stay alive in the thread TLS, probably long enough to end up in gen 2

We almost never send traces to the agent concurrently (I believe the call to IAgentWriter.Ping() at startup is the only exception so far). Therefore it makes sense to use a single cached buffer, and fallback to allocating a new one in the unlikely case we need it from two different threads at the same time.

I also updated the WriteAndFlushTraces benchmark to use a fake stream, because the MemoryStream was the source of a lot of extra allocations. In the real world we would be writing to a network stream and we have (AFAIK) no control over how much it'll allocate. It's better to focus on measuring what's actionable.

Benchmark: AgentWriterBenchmark.WriteAndFlushTraces

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.329 (2004/?/20H1)
Intel Core i7-9750H CPU 2.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.1.302
  [Host]        : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT
  .NET 4.7.2    : .NET Framework 4.8 (4.8.4180.0), X64 RyuJIT
  .NET Core 3.1 : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT

The main source of allocations remaining on this codepath is GetUniqueTraceIds. I tried replacing the hashset by a Linq Distinct() operation. It reduces heap allocations by about 40% (down to ~45KB) but also increase processing time by about 10% on .NET Framework (up to 440µs, no significant increase on .NET Core), so I don't know if it's worth it.