Repository navigation
fix(sdk/python): keep rate limiter jitter off the global random state - #1078
Conversation
_calculate_backoff_delay called random.seed(self._container_seed + attempt) before drawing jitter, which reseeds the module-level generator shared by the whole host process. Every rate-limit retry therefore reset the application's random stream to a value derived from HOSTNAME and the pid, so all later random.random()/choice()/shuffle()/sample() calls in the process returned a repeatable sequence instead of continuing the stream the caller seeded. It also degraded the SDK's own jitter: rate_limiter.py is the only module that seeds the global generator, but client.py:1295 (_next_poll_interval) and harness/_runner.py:415 and :423 (transient-error backoff) draw from it, so one rate-limited LLM call made their poll intervals and backoff delays a deterministic function of the container seed -- identical across containers that share a hostname/pid shape, which is exactly the synchronisation jitter exists to prevent. Draw from a private random.Random instance instead. The returned jitter is bit-identical (verified across 240 seed/attempt/range combinations by comparing float.hex()), because CPython's module-level random.uniform delegates to a module-level Random seeded through the same path. Backoff timing, retry counts, and per-container jitter distribution are unchanged; the seed just stops leaking into global state, which also stops concurrent callers from drawing jitter off a seed another caller installed. This matches what the Go and TypeScript SDKs already do: sdk/go/ai/ rate_limiter.go:321 uses rand.New(rand.NewSource(containerSeed + attempt)) and sdk/typescript/src/ai/RateLimiter.ts:172 uses a local _createJitterRng closure, both with the same seeding scheme and the same (u*2-1)*r formula that random.uniform(-r, r) expands to. The existing test_calculate_backoff_applies_jitter_and_max_cap already computes its expectation with random.Random(container_seed + attempt) and compares only the returned number, which is why this survived: nothing asserted on the global state. Adds two tests pinning the contract at both the helper and the public execute_with_retry path. enable_rate_limit_retry defaults to True (types.py:497) and agent_ai.py:605-609 re-forces it, so the affected path is the default one.
Performance
✓ No regressions detected |
📊 Coverage gateThresholds from
✅ Gate passedNo surface regressed past the allowed threshold and the aggregate stayed above the floor. |
📐 Patch coverage gateThreshold: 80% on lines this PR touches vs
✅ Patch gate passedEvery surface whose lines were touched by this PR has patch coverage at or above the threshold. |
AbirAbbas
left a comment
There was a problem hiding this comment.
Reproduced this on main: one backoff calculation reseeds the global random stream to the container seed, and with threads the shared reseed even corrupts other callers' jitter values (about 5% wrong under a tight switch interval). With your change the global state is untouched and the jitter is bit-for-bit identical to before (compared float.hex across 44k seed/attempt/config cases on 3.10 and 3.12). Both new tests fail against main's rate_limiter.py and pass here, and ruff plus the full suite are green with current main merged in. Nice find, thanks for the thorough write-up.
Summary
StatelessRateLimiter._calculate_backoff_delaycallsrandom.seed(self._container_seed + attempt)before drawing jitter. That seeds the module-level generator, so every rate-limit retry resets
the random stream of the whole host process to a value derived from
HOSTNAMEand the pid. After anagent hits one 429, every later
random.random()/choice()/shuffle()/sample()in theapplication returns a repeatable sequence instead of continuing the stream the caller seeded —
silently, and identically in every container that shares the hostname/pid shape.
Drawing from a private
random.Randominstance removes the side effect. The jitter values areunchanged: CPython's module-level
random.uniformdelegates to a module-levelRandomseededthrough the same path, so
random.seed(x)+random.uniform(...)andrandom.Random(x).uniform(...)produce the same number — verified bit-for-bit across 240seed/attempt/range combinations by comparing
float.hex(). Backoff timing, retry counts, and theper-container jitter distribution all behave exactly as before; the seed just stops leaking into
global state, which also stops concurrent callers from drawing jitter off a seed another caller
installed. The Go and TypeScript limiters already isolate their jitter generator this way, so this
brings Python in line with them (details under Related issues / PRs).
It also degrades the SDK's own jitter.
rate_limiter.pyis the only module that seeds the globalgenerator, but two other retry paths draw from it:
client.py:1295(_next_poll_interval,random.uniform(0.8, 1.2)) andharness/_runner.py:415/:423(transient-error backoff,random.uniform(-delay * 0.25, delay * 0.25)). After one rate-limited LLM call, both produce adeterministic sequence derived from the container seed — so containers sharing a hostname/pid shape
poll and back off in lockstep, which is exactly what that jitter is there to prevent.
This is on by default:
AIConfig.enable_rate_limit_retryisField(default=True)(
agentfield/types.py:497) andagent_ai.py:605-609re-forces it when unset, so the path isagent.chat(...)→execute_with_retry→_calculate_backoff_delay→ global reseed.Reproducer against
main:Type of change
Test plan
Two tests added to
sdk/python/tests/test_rate_limiter_core.py. Both fail onmainand pass withthe fix; neither needs network, credentials, or a coding-agent binary.
test_calculate_backoff_leaves_global_random_state_alone— snapshots three global draws, calls_calculate_backoff_delay, and asserts the stream continued unchanged.test_execute_with_retry_leaves_global_random_state_alone— the same assertion through the publicexecute_with_retrypath against a real 429 (two retries,jitter_factor=0.25,asyncio.sleepmonkeypatched), so the contract is pinned at the API callers actually use.
Commands run locally (Windows, CPython 3.13.12 in an isolated venv,
pip install -e ".[dev]"plusruff==0.15.22at the versionsdk-python.ymlpins):cd sdk/python && python -m pytest tests/test_rate_limiter_core.py --no-cov→ 12 passed(10 pre-existing + 2 new; both new tests fail before the source change)
cd sdk/python && python -m pytest --ignore=tests/test_agent_mesh.py --no-cov→9 failed, 2392 passed, 6 skipped on Windows/CPython 3.13.12. The identical command on
unmodified
maingives 9 failed, 2390 passed, 6 skipped with a byte-identical set offailing test ids, so the +2 passed is exactly the two new tests and nothing regressed. Those 9
are pre-existing Windows-host failures (
'gbk' codec can't decodefrompathlib.read_text,and
b'inline\r' != b'inline\n'CRLF assertions) that pass on CI'subuntu-latest.tests/test_agent_mesh.pyis excluded because it hangs on this host at ~8% — verified to hangidentically on unmodified
main, so it is an environment issue, not a regression.cd sdk/python && python -m mypy --config-file mypy.ini agentfield/→ Success: no issues foundin 81 source files
cd sdk/python && python -m ruff check .→ All checks passedcd sdk/python && python -m ruff format --diff agentfield/rate_limiter.py tests/test_rate_limiter_core.py→ 2 files already formatted (no reformatting noise added)
Test coverage
coverage-baseline.json— not applicable, no source lines wereremoved, so no surface percentage can drop.
./scripts/coverage-summary.shlocally because it measures all five surfaces and this machinehas no Go, Bun, or Docker engine. Leaving this for CI rather than claiming it.
agentfield.rate_limiteris already in the pytest--covlist, so it is a gated surface. The onlychanged executable statements are
rng = random.Random(...)andjitter = rng.uniform(...)— theother four added lines are comments — and both execute on every jittered backoff, covered by the
pre-existing
test_calculate_backoff_applies_jitter_and_max_capand the two new tests.That pre-existing test is also why this survived: it computes its expectation with
random.Random(limiter._container_seed + attempt)and compares only the returned number, which isidentical either way. Nothing asserted on the global state.
Checklist
docs/CONTRIBUTING.mdanddocs/DEVELOPMENT.md.conventional-commits; the commit is not signed, as I have no signing key configured. Flagging
it rather than ticking the box.
existing issue for it.
Related issues / PRs
No open issue or PR covers this.
git log --all -- sdk/python/agentfield/rate_limiter.pyshows thefile has only ever been touched by #265 (default tuning) and the original Haxen→AgentField rename.
Checking every remote ref,
rate_limiter.pyis either the currentmainblob or the pre-#265 blob,and no ref anywhere contains
random.Randomin that file — so nothing in flight addresses it.The Go and TypeScript SDKs already do this correctly, with the same
containerSeed + attemptseeding, so this is a parity fix rather than a design change:
sdk/go/ai/rate_limiter.go:321—rng := rand.New(rand.NewSource(r.containerSeed + int64(attempt))),a private generator, with the
math/randpackage-level functions deliberately unused.sdk/typescript/src/ai/RateLimiter.ts:172—_createJitterRng(seed)returns a local LCG closure,called as
this._createJitterRng(this._containerSeed + attempt).The jitter formula matches too:
random.uniform(-r, r)is-r + 2r·u=r·(2u − 1), exactly the(rng() * 2 - 1) * jitterRangeboth other SDKs use. Python is the only one drawing it from globalstate.
I read the Go and TypeScript limiters but did not run them (no Go or Bun toolchain here), so this PR
changes and tests Python only.