Skip to content

fix(sdk/python): keep rate limiter jitter off the global random state - #1078

Merged
AbirAbbas merged 1 commit into
Agent-Field:mainfrom
remote-controlled-man:codex/agentfield-python-selfdisc-20260929
Oct 1, 2026
Merged

AbirAbbas merged 1 commit into
Agent-Field:mainfrom
remote-controlled-man:codex/agentfield-python-selfdisc-20260929

Conversation

@remote-controlled-man

Copy link
Copy Markdown
Contributor

Summary

StatelessRateLimiter._calculate_backoff_delay calls random.seed(self._container_seed + attempt)
before drawing jitter. That seeds the module-level generator, so every rate-limit retry resets
the random stream of the whole host process to a value derived from HOSTNAME and the pid. After an
agent hits one 429, every later random.random() / choice() / shuffle() / sample() in the
application returns a repeatable sequence instead of continuing the stream the caller seeded —
silently, and identically in every container that shares the hostname/pid shape.

Drawing from a private random.Random instance removes the side effect. The jitter values are
unchanged: CPython's module-level random.uniform delegates to a module-level Random seeded
through the same path, so random.seed(x) + random.uniform(...) and
random.Random(x).uniform(...) produce the same number — verified bit-for-bit across 240
seed/attempt/range combinations by comparing float.hex(). Backoff timing, retry counts, and the
per-container jitter distribution all behave exactly as before; the seed just stops leaking into
global state, which also stops concurrent callers from drawing jitter off a seed another caller
installed. The Go and TypeScript limiters already isolate their jitter generator this way, so this
brings Python in line with them (details under Related issues / PRs).

It also degrades the SDK's own jitter. rate_limiter.py is the only module that seeds the global
generator, but two other retry paths draw from it: client.py:1295 (_next_poll_interval,
random.uniform(0.8, 1.2)) and harness/_runner.py:415/:423 (transient-error backoff,
random.uniform(-delay * 0.25, delay * 0.25)). After one rate-limited LLM call, both produce a
deterministic sequence derived from the container seed — so containers sharing a hostname/pid shape
poll and back off in lockstep, which is exactly what that jitter is there to prevent.

This is on by default: AIConfig.enable_rate_limit_retry is Field(default=True)
(agentfield/types.py:497) and agent_ai.py:605-609 re-forces it when unset, so the path is
agent.chat(...) → execute_with_retry → _calculate_backoff_delay → global reseed.

Reproducer against main:

import random
from agentfield.rate_limiter import StatelessRateLimiter

random.seed(1234); before = random.random()
random.seed(1234)
StatelessRateLimiter(jitter_factor=0.25)._calculate_backoff_delay(0)
assert random.random() == before   # fails today

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Docs only
  • Tests only
  • CI / tooling
  • Breaking change

Test plan

Two tests added to sdk/python/tests/test_rate_limiter_core.py. Both fail on main and pass with
the fix; neither needs network, credentials, or a coding-agent binary.

  • test_calculate_backoff_leaves_global_random_state_alone — snapshots three global draws, calls
    _calculate_backoff_delay, and asserts the stream continued unchanged.
  • test_execute_with_retry_leaves_global_random_state_alone — the same assertion through the public
    execute_with_retry path against a real 429 (two retries, jitter_factor=0.25, asyncio.sleep
    monkeypatched), so the contract is pinned at the API callers actually use.

Commands run locally (Windows, CPython 3.13.12 in an isolated venv, pip install -e ".[dev]" plus
ruff==0.15.22 at the version sdk-python.yml pins):

  • cd sdk/python && python -m pytest tests/test_rate_limiter_core.py --no-cov → 12 passed
    (10 pre-existing + 2 new; both new tests fail before the source change)
  • cd sdk/python && python -m pytest --ignore=tests/test_agent_mesh.py --no-cov →
    9 failed, 2392 passed, 6 skipped on Windows/CPython 3.13.12. The identical command on
    unmodified main gives 9 failed, 2390 passed, 6 skipped with a byte-identical set of
    failing test ids, so the +2 passed is exactly the two new tests and nothing regressed. Those 9
    are pre-existing Windows-host failures ('gbk' codec can't decode from pathlib.read_text,
    and b'inline\r' != b'inline\n' CRLF assertions) that pass on CI's ubuntu-latest.
    tests/test_agent_mesh.py is excluded because it hangs on this host at ~8% — verified to hang
    identically on unmodified main, so it is an environment issue, not a regression.
  • cd sdk/python && python -m mypy --config-file mypy.ini agentfield/ → Success: no issues found
    in 81 source files
  • cd sdk/python && python -m ruff check . → All checks passed
  • cd sdk/python && python -m ruff format --diff agentfield/rate_limiter.py tests/test_rate_limiter_core.py
    → 2 files already formatted (no reformatting noise added)

Test coverage

  • I ran tests for the surface(s) I changed locally.
  • New code paths are covered by tests in this PR (no bare additions).
  • If I removed code, I updated coverage-baseline.json — not applicable, no source lines were
    removed, so no surface percentage can drop.
  • The coverage gate check is green in CI before requesting review — I cannot run
    ./scripts/coverage-summary.sh locally because it measures all five surfaces and this machine
    has no Go, Bun, or Docker engine. Leaving this for CI rather than claiming it.

agentfield.rate_limiter is already in the pytest --cov list, so it is a gated surface. The only
changed executable statements are rng = random.Random(...) and jitter = rng.uniform(...) — the
other four added lines are comments — and both execute on every jittered backoff, covered by the
pre-existing test_calculate_backoff_applies_jitter_and_max_cap and the two new tests.

That pre-existing test is also why this survived: it computes its expectation with
random.Random(limiter._container_seed + attempt) and compares only the returned number, which is
identical either way. Nothing asserted on the global state.

Checklist

  • I have read docs/CONTRIBUTING.md and docs/DEVELOPMENT.md.
  • Commits are signed and follow conventional-commits style. — the subject follows
    conventional-commits; the commit is not signed, as I have no signing key configured. Flagging
    it rather than ticking the box.
  • I have linked any related issues — none: this was found by reading the module, there is no
    existing issue for it.

Related issues / PRs

No open issue or PR covers this. git log --all -- sdk/python/agentfield/rate_limiter.py shows the
file has only ever been touched by #265 (default tuning) and the original Haxen→AgentField rename.
Checking every remote ref, rate_limiter.py is either the current main blob or the pre-#265 blob,
and no ref anywhere contains random.Random in that file — so nothing in flight addresses it.

The Go and TypeScript SDKs already do this correctly, with the same containerSeed + attempt
seeding, so this is a parity fix rather than a design change:

  • sdk/go/ai/rate_limiter.go:321 — rng := rand.New(rand.NewSource(r.containerSeed + int64(attempt))),
    a private generator, with the math/rand package-level functions deliberately unused.
  • sdk/typescript/src/ai/RateLimiter.ts:172 — _createJitterRng(seed) returns a local LCG closure,
    called as this._createJitterRng(this._containerSeed + attempt).

The jitter formula matches too: random.uniform(-r, r) is -r + 2r·u = r·(2u − 1), exactly the
(rng() * 2 - 1) * jitterRange both other SDKs use. Python is the only one drawing it from global
state.

I read the Go and TypeScript limiters but did not run them (no Go or Bun toolchain here), so this PR
changes and tests Python only.

_calculate_backoff_delay called random.seed(self._container_seed + attempt)
before drawing jitter, which reseeds the module-level generator shared by the
whole host process. Every rate-limit retry therefore reset the application's
random stream to a value derived from HOSTNAME and the pid, so all later
random.random()/choice()/shuffle()/sample() calls in the process returned a
repeatable sequence instead of continuing the stream the caller seeded.

It also degraded the SDK's own jitter: rate_limiter.py is the only module that
seeds the global generator, but client.py:1295 (_next_poll_interval) and
harness/_runner.py:415 and :423 (transient-error backoff) draw from it, so one
rate-limited LLM call made their poll intervals and backoff delays a
deterministic function of the container seed -- identical across containers
that share a hostname/pid shape, which is exactly the synchronisation jitter
exists to prevent.

Draw from a private random.Random instance instead. The returned jitter is
bit-identical (verified across 240 seed/attempt/range combinations by
comparing float.hex()), because CPython's module-level random.uniform
delegates to a module-level Random seeded through the same path. Backoff
timing, retry counts, and per-container jitter distribution are unchanged;
the seed just stops leaking into global state, which also stops concurrent
callers from drawing jitter off a seed another caller installed.

This matches what the Go and TypeScript SDKs already do: sdk/go/ai/
rate_limiter.go:321 uses rand.New(rand.NewSource(containerSeed + attempt)) and
sdk/typescript/src/ai/RateLimiter.ts:172 uses a local _createJitterRng closure,
both with the same seeding scheme and the same (u*2-1)*r formula that
random.uniform(-r, r) expands to.

The existing test_calculate_backoff_applies_jitter_and_max_cap already computes
its expectation with random.Random(container_seed + attempt) and compares only
the returned number, which is why this survived: nothing asserted on the global
state. Adds two tests pinning the contract at both the helper and the public
execute_with_retry path.

enable_rate_limit_retry defaults to True (types.py:497) and agent_ai.py:605-609
re-forces it, so the affected path is the default one.
Copilot AI balanced review requested due to automatic review settings September 30, 2026 02:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown
Contributor

Performance

SDK Memory Δ Latency Δ Tests Status
Python 9.0 KB - 0.30 µs -14% ✓ ✓

✓ No regressions detected

@github-actions

Copy link
Copy Markdown
Contributor

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface Current Baseline Δ
control-plane 87.90% 87.40% ↑ +0.50 pp 🟡
sdk-go 93.30% 92.00% ↑ +1.30 pp 🟢
sdk-python 94.72% 93.73% ↑ +0.99 pp 🟢
sdk-typescript 91.84% 90.42% ↑ +1.42 pp 🟢
web-ui 84.77% 84.79% ↓ -0.02 pp 🟡
aggregate 85.92% 85.75% ↑ +0.17 pp 🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

@github-actions

Copy link
Copy Markdown
Contributor

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface Touched lines Patch coverage Status
control-plane 0 — ➖ no changes
sdk-go 0 — ➖ no changes
sdk-python 2 100.00% ✅
sdk-typescript 0 — ➖ no changes
web-ui 0 — ➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.

@AbirAbbas AbirAbbas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced this on main: one backoff calculation reseeds the global random stream to the container seed, and with threads the shared reseed even corrupts other callers' jitter values (about 5% wrong under a tight switch interval). With your change the global state is untouched and the jitter is bit-for-bit identical to before (compared float.hex across 44k seed/attempt/config cases on 3.10 and 3.12). Both new tests fail against main's rate_limiter.py and pass here, and ruff plus the full suite are green with current main merged in. Nice find, thanks for the thorough write-up.

@AbirAbbas
AbirAbbas added this pull request to the merge queue Oct 1, 2026
Merged via the queue into Agent-Field:main with commit b675e2d Oct 1, 2026
28 checks passed
@remote-controlled-man
remote-controlled-man deleted the codex/agentfield-python-selfdisc-20260929 branch October 13, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants