fix(utils): advance the token bucket timestamp even when the bucket is full - #2652

Fixes #680.

TokenBucket grants one extra full refill on the first burst after an idle period, so a rate limit built on it is exceeded at exactly the moment traffic resumes. At the default capacity=1 that is two consumes where one was configured.

The refill method in kombu/utils/limits.py (lines 81-87 on main) is def _get_tokens(self): if self._tokens < self.capacity: now = monotonic(); ...; self.timestamp = now. Both the clock read and the timestamp write sit inside the if, so while the bucket is at capacity that branch never runs and timestamp stays frozen at whatever it was, which for a fresh bucket is its construction time. The first can_consume() drains tokens without ever touching timestamp. The very next call then computes delta = self.fill_rate * (now - self.timestamp) against that stale value, so the entire idle period is counted as refill time and min() tops the bucket straight back up to capacity. Any gap long enough to notionally refill the bucket produces the same over-admission; a longer gap does not make it worse, because min() clamps at capacity.

from unittest.mock import patch
import kombu.utils.limits as L
now = [1000.0]
with patch.object(L, 'monotonic', lambda: now[0]):
    b = L.TokenBucket(fill_rate=0.1, capacity=1)
    now[0] = 1600.0
    print(b.can_consume())  # True
    now[0] = 1600.001
    print(b.can_consume())  # True on main, should be False

At one token per 10 seconds the bucket grants two consumes 1ms apart. Measured on unpatched main, the size of the hole is exactly one full refill: a capacity=10 bucket idle for 600s admits 11 one-token consumes instead of 10, and grants can_consume(10) twice in a row where it should grant it once.

Issue #680 proposed this change in 2017, character for character: hoist now = monotonic() above the if and move self.timestamp = now below it, matching the ActiveState recipe the class was taken from. auvipy replied "contributions are welcome" in January 2018. This PR is that diff plus tests and a changelog entry. The only other open PR touching this file, #1525 ("Annotate and refactor limits.py", conflicted and untouched since December 2022), only annotates the _get_tokens signature and leaves its body unchanged, so there is no overlap.

That is two moved lines. Nothing else in the class changes: can_consume, expected_time, add, pop, clear_pending and every other public attribute are untouched, and normal refill at fill_rate behaves exactly as before. expected_time() now also advances timestamp when the bucket is full; below capacity it already did, and at capacity there is nothing to lose. The stale #: comment on timestamp is updated to say what it now records. In-tree, ConsumerMixin.restart_limit in kombu/mixins.py is unaffected, because it is drained on the same call that creates it and never sits idle at capacity. Downstream, celery's per-task rate limits get the limit they were configured with instead of one extra on the first burst.

t/unit/utils/test_limits.py is new, since the module had no tests at all. test_can_consume_does_not_double_admit_after_idle and test_idle_bucket_admits_only_its_capacity both fail on unpatched main and pass with the fix; the second one covers the general capacity=10 case. test_refills_at_fill_rate passes both before and after, so ordinary refill is proven unchanged.

Verification, on Python 3.11 with requirements/default.txt and requirements/test.txt installed. pytest t/unit/utils/test_limits.py -q gives 2 failed, 1 passed in 0.11s with main's limits.py restored, failing on assert True is False and on At index 10 diff: True != False, and 3 passed in 0.06s with the fix. For the before/after comparison, the offline unit suite with the new test file removed gives 952 passed, 189 skipped both with main's limits.py and with the fix, so nothing else in the suite moves; with the new file it is 955 passed, 189 skipped in 17.05s. Those runs exclude t/unit/asynchronous/aws, t/unit/transport/SQS, test_azurestoragequeues.py, test_azureservicebus.py and test_gcpubsub.py, which need the boto3, azure and google cloud SDKs from requirements/test-ci.txt and are not installed locally. flake8 -j2 kombu t and pydocstyle kombu both exit 0, and mypy --config-file setup.cfg reports Success: no issues found in 22 source files.