Is it possible that test failures here are real and that we should improve the rate limiting logic? Or are we ok with rate limits to be exceeded by 15%?

It's possible that 1) the rate limiter is not performing as we expect, 2) the timing calculations in this test are incorrect, or 3) our testing variance is too small. I'm going to look into 1 and 2 today

My current suspicion is that the RateLimiter is misbehaving. We build up a queue to hold onto timestamps from the last window, where the max size should be about the same as the configured interval limit. However, when one thread is busy dequeuing that as part of the refresh logic, a bunch of other threads may be trying to push Spans for the new window, and the queue toggles from full to not full. However, that's not the right indicator for the new window, so I'm trying to use the _windowAllowed value as the signal instead of queue size.

In retrospect, it seems like there is just a mismatch of the expected test output and the current rate limit implementation. In our tests we simply assume that we should see the number of allowed spans = intervalLimit * numberOfIntervals. But that doesn't take into account the behavior I just described. So one of the two should change, either the implementation or the test expectations

Talking offline, we'll allow for more tolerance on the lower limit side. This aligns with the reality that our current implementation of the rate limiter dequeues from the rolling window is slower than enqueueing to the rolling window, which may result in a fewer number of Allowed spans. This is fine and we'll account for that difference in our test expectations.