Repository navigation
BackendSpanExporter drops the whole batch on HTTP 429 instead of retrying with Retry-After #5023
Description
Please read this first
- Have you read the docs? Yes: Tracing and the
BackendSpanExporterreference. - Have you searched for related issues? Yes.
BackendSpanExporter._shutdown_eventis never cleared, permanently disabling retries after one shutdown #4683 / Fix tracing exporter reuse after shutdown #4711 cover retries being disabled after shutdown, BackendSpanExporter: one metadata or custom span value that isn't JSON serializable drops the whole export batch, including unrelated traces #4983 covers a non-serializable value dropping a batch, Tracing exporter honours OPENAI_API_KEY/ORG/PROJECT from the environment but not the endpoint, so OPENAI_BASE_URL does not redirect traces #5008 covers the endpoint. None covers how the exporter classifies HTTP status codes.
Describe the bug
BackendSpanExporter._export_with_deadline() treats every 4xx response as final:
# If the response is a client error (4xx), we won't retry
if 400 <= response.status_code < 500:
logger.error("[non-fatal] Tracing client error %s: %s", ...)
breakThat includes 429 Too Many Requests. A single rate-limited response from /v1/traces/ingest discards the whole batch, up to max_batch_size (128) traces and spans, with no retry and without looking at Retry-After. The run itself succeeds, so the only symptom is a gap in the dashboard and one [non-fatal] log line. The same applies to 408 and 409, which the OpenAI Python client (_base_client._should_retry) retries together with 429 and 5xx; the exporter only mirrors the 5xx half of that policy.
The export path already has everything needed to retry: max_retries, base_delay, max_delay, the backoff loop, and agents.models._retry_runtime.parse_retry_after_value / parse_retry_after_ms for the header.
Debug information
- Agents SDK version:
mainatfbf59a40(also 0.22.0) - Python version: 3.10.12
- Operating system: Linux
- Model and model provider: none needed;
httpx2.Clientis patched - Does the issue reproduce with the latest Agents SDK release? Yes
- Does the issue occur consistently or intermittently? Consistently for any 429
ERROR:openai.agents:[non-fatal] Tracing client error 429. Response data is redacted.
post calls: 1
Repro steps
import logging
from unittest.mock import MagicMock, patch
from agents.tracing.processors import BackendSpanExporter
from agents.tracing.spans import SpanImpl
from agents.tracing.span_data import CustomSpanData
logging.basicConfig(level=logging.WARNING)
with patch("httpx2.Client") as client:
limited = MagicMock(status_code=429, headers={"retry-after": "1"})
limited.text = '{"error": {"message": "Rate limit reached"}}'
ok = MagicMock(status_code=200)
client.return_value.post.side_effect = [limited, ok]
exporter = BackendSpanExporter(api_key="sk-test", max_retries=3, base_delay=0.1)
span = SpanImpl(
trace_id="trace_1",
span_id="span_1",
parent_id=None,
processor=MagicMock(),
span_data=CustomSpanData(name="probe", data={}),
tracing_api_key=None,
)
exporter.export([span])
print("post calls:", client.return_value.post.call_count) # 1: the batch is goneExpected behavior
408, 409 and 429 are retried with the existing backoff up to max_retries, waiting at least the advertised Retry-After / retry-after-ms (bounded by max_delay), the way the OpenAI client and the SDK's own model retry helpers already treat these statuses. Other 4xx responses stay non-retried. In the script above the second call succeeds and post calls is 2.
A fix with regression tests is ready and I will attach it right away.
Activity
coderdailyone commentedon Sep 14, 2026
Fix with regression tests is ready on my fork: coderdailyone@2b265051 (branch
fix/tracing-retry-rate-limit, based onfbf59a40).408,409and429now go through the existing backoff loop up tomax_retries, mirroring the OpenAI client's_should_retry; the wait honoursRetry-After/retry-after-msvia the shared_retry_runtimeparsers (lazy import, no new import-time dependency for tracing), bounded bymax_delay. Other4xxresponses are unchanged.tests/test_trace_processor.py(three statuses, recovery on the second attempt, Retry-After in seconds / milliseconds / abovemax_delay/ unparsable / negative); all fail onmain, pass with the fix.tests/test_trace_processor.py+tests/tracing(130 passed), mypy, ruff clean.GitHub refuses
createPullRequestfor my account on this repository at the moment (same as #5006 reports), so I cannot open the PR myself; I will as soon as that clears, or a maintainer is welcome to pull the branch.L4XB commentedon Sep 15, 2026
Independent reproduction on
main@fbf59a40, plus one piece of in-repo evidence that I think strengthens the case for the classification proposed here.The repro is accurate. Running the script from the issue against
fbf59a40:The 429 response is consumed by the
400 <= response.status_code < 500branch in_export_with_deadline, whichbreaks before the backoff loop can run, so the batch is discarded on the first response.There is already a precedent for this classification inside the SDK, not just in the
openaiclient. The issue citesopenai._base_client._should_retry, which is correct — onopenai==3.0.0that method retries 408, 409, 429 and>= 500. But the SDK also carries its own copy of that policy:get_openai_retry_adviceinsrc/agents/models/_openai_retry.pyclassifiesas retry-suggested, and reaches for
Retry-Afterthrough the same_retry_runtimehelpers the issue points at. That makesBackendSpanExporterthe one place in the SDK that applies a different retry policy to the same set of statuses against the same backend, which seems more like drift than an intentional divergence for the tracing path.One point maintainers may want to rule on. Both
_should_retryandget_openai_retry_adviceconsult thex-should-retryheader before looking at the status code, and both treat an over-largeRetry-Afteras a reason not to retry (MAX_RETRY_AFTER_DELAY) rather than as a value to clamp. A tracing exporter that already has a deadline and a shutdown event arguably wants the clamping behaviour instead, and droppingx-should-retrykeeps the change small — but it is a deliberate difference from the two existing policies rather than an exact mirror, so it is worth an explicit decision either way.On the patch linked above. Applied to
fbf59a40it behaves as the issue's "Expected behavior" section describes — the same repro then reportspost calls: 2— and the checks come out clean:The nine added tests are discriminating: with the patch's test file in place but
src/agents/tracing/processors.pyreverted tofbf59a40, all nine fail (9 failed, 56 deselected) — three for the 408/409/429 parametrisation, one for recovery on the second attempt, five for theRetry-Afterseconds/ms/over-max_delay/unparsable/negative cases.I have not opened a competing pull request and do not intend to; this is only confirmation plus the
_openai_retry.pyreference, in case it helps triage.File and line references are against
main@fbf59a40e9da.coderdailyone commentedon Sep 15, 2026
Thanks for the independent run and the
get_openai_retry_advicepointer; that is the better in-repo precedent and I have cited it in the PR text.On the two divergences: I added
x-should-retry: falseas a veto incbb3a561(testtest_backend_span_exporter_rate_limit_obeys_x_should_retry_false), since honouring it costs one header read and keeps the classification identical to the model path. For an oversizedRetry-AfterI kept clamping tomax_delayrather than refusing the retry: the model path gives up because a request that waits minutes blocks the caller, whereas the exporter runs on the background thread with its own deadline and shutdown event, so waiting the boundedmax_delayand retrying keeps the batch alive at no cost to the run. Happy to switch to the refusal semantics if the maintainers prefer an exact mirror.fix(tracing): retry rate-limited trace exports instead of dropping th…
fix(tracing): retry rate-limited trace exports (Fixes openai#5023) (o…
Sync upstream main and update Garnet pins (#31)