Repository navigation
Python: [Bug]: Re-polling a background response after a failed poll re-runs its function calls, repeating their side effects #9031
Description
Description
This is the caller-side sibling of #5394.
With options={"background": True}, the caller holds a continuation token and
polls with it until it is None, as in the background-responses docs. When the
background response completes with function calls, the poll itself:
- runs the calls (the function loop);
- POSTs their outputs.
If step 2 fails, for example on a sustained 429 or 503 after the SDK's
retries, the poll raises ChatClientException. The token the caller still
holds names the response that requested the calls. Re-polling with it, the
obvious recovery after an exception (the docs present continuation tokens as
the handle for operations "interrupted by network issues or client timeouts"),
retrieves that response again and runs its calls a second time.
The fix for #5394 (#5462, and #8357 for the streaming resume) handled the
in-loop form of this: it pops continuation_token from
the options dict once the response completes (_chat_client.py, around lines
956–963 and 876–883 in agent-framework-openai 1.14.4; the comment there reads
"the tools run again each time"). The caller-held token is the remaining form.
After an exception the caller has no other token to use, and nothing tells it
that the calls behind that token have already run.
Code Sample
# --- repro.py (verbatim) ---
"""Agent Framework (Python): re-polling a background response with the held
continuation token re-runs the function calls of that response.
Self-contained: `pip install agent-framework-core==1.19.0 agent-framework-openai==1.14.4`
then `python repro.py`. No network access and no API key; nothing leaves 127.0.0.1.
- The agent is the documented background-responses flow: `agent.run(...,
options={"background": True})`, then poll with `continuation_token` until it
is None.
- A local fake OpenAI Responses endpoint queues the first request and, on
retrieve, returns it completed with one `pay_invoice` function call.
Submitting the function output fails with 429 on every attempt (a sustained
rate limit or outage after the tool ran).
- A local fake payment provider records each payment.
- When a poll raises, the caller polls once more with the token it holds,
which is the only token it has.
"""
import asyncio
import json
import threading
import time
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Annotated
MODEL, LEDGER = ("127.0.0.1", 8963), ("127.0.0.1", 8964)
PAYMENTS = [] # the provider's ledger: one entry per committed payment
STORED = {} # response id -> the completed response object
ARGS = {"invoice": "INV-1", "payee": "ACME", "amount": "100.00"}
class Ledger(BaseHTTPRequestHandler):
def do_POST(self): # noqa: N802
self.rfile.read(int(self.headers["Content-Length"]))
PAYMENTS.append(time.time())
data = json.dumps({"status": "PAID", "payment": len(PAYMENTS)}).encode()
self.send_response(200)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def log_message(self, *args):
pass
class FakeResponsesAPI(BaseHTTPRequestHandler):
def send(self, status, obj, headers=()):
data = json.dumps(obj).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
for k, v in headers:
self.send_header(k, v)
self.end_headers()
self.wfile.write(data)
def do_GET(self): # noqa: N802 retrieve: the stored response, completed
rid = self.path.split("?")[0].rstrip("/").rsplit("/", 1)[-1]
self.send(200, STORED[rid]) if rid in STORED else self.send(404, {"error": {"message": "no such response"}})
def do_POST(self): # noqa: N802
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
items = body.get("input") if isinstance(body.get("input"), list) else []
if any(isinstance(i, dict) and i.get("type") == "function_call_output" for i in items):
return self.send(429, {"error": {"message": "Rate limit reached", "type": "requests",
"code": "rate_limit_exceeded"}}, [("retry-after-ms", "50")])
rid = f"resp_{len(STORED) + 1}"
STORED[rid] = {
"id": rid, "object": "response", "created_at": int(time.time()), "status": "completed",
"model": "gpt-4o-mini", "background": bool(body.get("background")), "store": True,
"output": [{"type": "function_call", "id": "fc_1", "call_id": "call_1", "name": "pay_invoice",
"arguments": json.dumps(ARGS), "status": "completed"}],
"parallel_tool_calls": True, "tool_choice": "auto", "tools": [], "error": None,
"incomplete_details": None, "instructions": None, "metadata": {},
"previous_response_id": body.get("previous_response_id"), "text": {"format": {"type": "text"}},
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2,
"input_tokens_details": {"cached_tokens": 0}, "output_tokens_details": {"reasoning_tokens": 0}}}
self.send(200, dict(STORED[rid], status="queued", output=[], usage=None))
def log_message(self, *args):
pass
def serve(addr, handler):
s = ThreadingHTTPServer(addr, handler)
s.daemon_threads = True
threading.Thread(target=s.serve_forever, daemon=True).start()
serve(MODEL, FakeResponsesAPI)
serve(LEDGER, Ledger)
from importlib.metadata import version # noqa: E402
from agent_framework import Agent, tool # noqa: E402
from agent_framework.openai import OpenAIChatClient # noqa: E402
from pydantic import Field # noqa: E402
@tool(approval_mode="never_require")
def pay_invoice(invoice: Annotated[str, Field(description="Invoice number")],
payee: Annotated[str, Field(description="Payee account")],
amount: Annotated[str, Field(description="Amount to pay")]) -> str:
"""Pay a vendor invoice. This moves money and cannot be undone."""
req = urllib.request.Request(f"http://{LEDGER[0]}:{LEDGER[1]}/pay", method="POST",
data=json.dumps({"invoice": invoice, "payee": payee, "amount": amount}).encode())
with urllib.request.urlopen(req, timeout=10) as r:
return r.read().decode()
async def main():
client = OpenAIChatClient(model="gpt-4o-mini", api_key="placeholder",
base_url=f"http://{MODEL[0]}:{MODEL[1]}/v1")
agent = Agent(client=client, name="PayAgent", instructions="You pay vendor invoices.", tools=[pay_invoice])
session = agent.create_session()
response = await agent.run(messages="Pay invoice INV-1 to ACME for 100.00.", session=session,
options={"background": True})
token = response.continuation_token
print(f"started; holding continuation token {dict(token)}")
for attempt in (1, 2): # a poll, then one re-poll with the held token after it raised
await asyncio.sleep(0.2)
try:
response = await agent.run(session=session, options={"continuation_token": token})
token = response.continuation_token
print(f"poll {attempt}: ok")
if token is None:
break
except Exception as exc:
print(f"poll {attempt} raised {type(exc).__name__} (payments so far: {len(PAYMENTS)})")
print(f"agent-framework-core {version('agent-framework-core')}")
asyncio.run(main())
print(f"payments made for one requested payment: {len(PAYMENTS)}")
# --- end repro.py ---Error Messages / Stack Traces
### Reproduction
_Re-checked on 2026-10-03 against agent-framework-core 1.20.0 and agent-framework-openai 1.15.0, the current releases: same result._
Self-contained script below (`pip install agent-framework-core==1.19.0
agent-framework-openai==1.14.4`, then `python repro.py`). It needs no network
access and no API key. It uses:
- a local fake Responses endpoint that queues the first request and, on
retrieve, returns it completed with one `pay_invoice` call;
- a POST of the function output that returns 429 every time;
- a local fake payment provider.
Output:
agent-framework-core 1.19.0
started; holding continuation token {'response_id': 'resp_1'}
poll 1 raised ChatClientException (payments so far: 1)
poll 2 raised ChatClientException (payments so far: 2)
payments made for one requested payment: 2
The service never received either function output, so it has no record of
either payment.
### Expected
After a poll has run a response's function calls, re-polling with the same
token should not run them again. Failing that, the caller should get a way to
see that they ran.Package Versions
agent-framework-core 1.19.0, agent-framework-openai 1.14.4, openai 3.20.0
Python Version
Python 3.13.9
Additional Context
Possible changes
- Surface the executed calls when the output POST fails. For example,
return or raise with a token (or an error attribute) that carries the
function results already produced, so the next poll submits them instead of
re-running the tools. - Record executed
call_ids per response in the session. On a
re-retrieve of the same response, skip or refuse the calls that already ran. - At least, document it. Say that a failed poll may have run tools, that
re-polling with the same token runs them again, and that side-effecting
tools should key oncall_id.
Found while testing retry and resume paths across agent frameworks.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
jstar0 commentedon Oct 4, 2026
Would the Agent Framework core team welcome an external PR for this issue? If so, which behavior should it target: persist completed function outputs across a failed submission so a retry does not execute the calls again, or surface the completed calls/results so the caller can resubmit them? I will wait for the team’s direction before implementing.
Li-john1021 commentedon Oct 4, 2026
I'd like to take this issue. Please assign it to me if that works.
I saw the three options in the report, and King Star (@jstar0)'s question about which behavior to implement. I will not open a pull request until the core team picks one.
loyce-cheng commentedon Oct 4, 2026
I independently reproduced this on current
main(e9857bd75a21843803431a2e7dfa41ee9b017ac7) and on core 1.20.0 / OpenAI 1.15.0 with SDK 3.20.0, using an in-memory fake Responses transport and a side-effect counter.Both streaming and non-streaming polls execute the same tool twice after a tool-output POST fails with 429 or 503. This also happens when the second POST succeeds. Controls show zero executions when retrieval fails before execution, and one execution when retrieval and submission succeed. The SDK's retries were disabled to isolate the framework's behavior after a request failure.
My proposed scope is session-scoped recovery of completed local tool outputs: retain the completed call/result groups before the next provider request, then resubmit those results when the same background response is resumed. Identity would need to include the provider/response and call occurrence within the session, rather than globally deduplicating
call_id. Stateless recovery, stale-token handling, partial batches/cancellation, and concurrent polls need an explicit contract; this would not claim exactly-once side effects across crashes or ambiguous provider acceptance.The likely implementation spans core's function loop and the OpenAI continuation adapter. Validation would cover streaming/non-streaming retry behavior, result identity, reused call IDs, session isolation, and the affected continuation/history/approval matrix, plus the full core and OpenAI package tests, syntax, source typing, and test typing checks.
I see the existing requests to work on this. Could the core team confirm ownership, the intended recovery behavior, affected matrix rows, and validation scope before implementation, as required by the function-loop specification? If another contributor is taking the fix, I am happy to coordinate on regression coverage instead of opening a competing PR.
bayu911-source commentedon Oct 4, 2026
This failure mode maps directly to an execution-evidence invariant we are validating in NAEOS.
The key boundary is preserving the identity of an already-executed logical invocation across retry:
authorize → execute tool → observe result → persist result → resumeIf the provider submission fails after the side effect occurred, the next poll must not treat the same function call as a new executable invocation.
A useful regression contract could distinguish:
Only the first case is safe to replay as a recorded result without risking a duplicate side effect.
Authorization and execution outcome should remain separate, but both should bind to the same logical invocation ID. The verifier can then ask:
Recovery should follow the logical invocation, not worker/request-local state.
This is one of the recovery properties we are testing in NAEOS: recover → resume → preserve one logical execution/evidence chain without duplicate side effects or orphaned outcomes.
LortuArte commentedon Oct 5, 2026
I reproduced the boundary described here conceptually against the current AEGIS 3.4.0 execution model and I think there is one useful, narrower experiment that would complement the framework-level recovery work
AEGIS does not currently claim crash-safe or cross-process exactly-once execution.
What it can test independently is whether a stable logical invocation ID prevents a completed consequential tool call from receiving a second local execution authorization when the same invocation is re-presented.
I can provide a provider-free Python fixture with:
a fake side-effect counter;
one stable invocation ID;
successful execution followed by simulated submission failure;
repeated presentation of the same invocation;
assertion on the final downstream execution count.
Expected invariant: one logical invocation → at most one locally authorized downstream execution.
I would publish neither an Agent Framework integration claim nor an exactly-once claim from that result. PASS or FAIL is useful.
If the maintainers consider that bounded fixture relevant to the regression matrix, I can prepare it without opening a framework PR first.
e7raem commentedon Oct 6, 2026
I’d retain the ledger mutation count as a key regression assertion when testing the fix, rather than check only that polling succeeds.
I maintain Ota, an open-source execution contract shared by agents, developers and CI.
Would a small, separate non-blocking workflow around your existing fake endpoint and ledger be useful? I could propose a reviewable ota.yaml that retains the tested revision and outcomes, with scope agreed before opening a PR.
The framework would still own the retry fix. This lane would test the synthetic scenario, not prove exactly-once execution against real providers.
eavanvalkenburg commentedon Oct 27, 2026
/plan Investigate the caller-held background continuation retry boundary on current main, coordinating with owner sophia-ramsey and related #8574 without treating their different failure paths as duplicates. Current adapter token removal prevents in-loop re-retrieval but does not invalidate the token held by the caller when a subsequent tool-output POST fails; core executes local tools before that next provider await. Compare session-scoped completed-result resubmission with an explicit caller-visible recovery/error contract, preserving occurrence identity and approval/history semantics from docs/specs/004-python-function-calling-loop.md. Define provider/response/tool occurrence identity rather than globally deduplicating call_id; distinguish completed output, in-flight/unknown outcome and ambiguous provider acceptance. Cover partial batches, cancellation, concurrent polls, reused IDs, session isolation/serialization, stale tokens, middleware options replacement and stateless calls. Do not promise crash-safe or cross-process exactly-once side effects, rerun unknown consequential calls silently, or add an external execution-contract dependency. Validate both streaming and non-streaming with fake retrieval plus output POST failures429/503 and later success, asserting actual side-effect counts and retained result identity; include pre-execution retrieval failure controls and affected continuation/approval/history matrix rows. Report smallest safe production scope, compatibility/migration decisions and remaining evidence gaps before implementation. Several contributors have volunteered (jstar0, Li-john1021, loyce-cheng); coordinate selection with Sophia after planning rather than assigning competing implementations. This is a non-minor bug investigation, not an implementation-ready decision.
vijaym2k6 commentedon Nov 4, 2026
Thanks for the clear repro. The double payment is the dangerous part. Until the SDK tracks executed call_ids per response, the safest guard is on the tool side: give each side-effecting tool an idempotency key derived from the call_id and have it return the stored result if it has already run. That makes a re-poll harmless no matter how the SDK recovers.
A policy layer in front of tools can also help. SteerPlane (open source, http://github.com/vijaym2k6/SteerPlane) can block tool calls your policy does not allow and detect repeated actions within a run, which would flag a second identical payment call.
eavanvalkenburg commentedon Nov 12, 2026
/plan Investigate the caller-held background continuation retry boundary on current main, coordinating with owner sophia-ramsey and related #8574 without treating their different failure paths as duplicates. Current adapter token removal prevents in-loop re-retrieval but does not invalidate the token held by the caller when a subsequent tool-output POST fails; core executes local tools before that next provider await. Compare session-scoped completed-result resubmission with an explicit caller-visible recovery/error contract, preserving occurrence identity and approval/history semantics from docs/specs/004-python-function-calling-loop.md. Define provider/response/tool occurrence identity rather than globally deduplicating call_id; distinguish completed output, in-flight/unknown outcome and ambiguous provider acceptance. Cover partial batches, cancellation, concurrent polls, reused IDs, session isolation/serialization, stale tokens, middleware options replacement and stateless calls. Do not promise crash-safe or cross-process exactly-once side effects, rerun unknown consequential calls silently, or add an external execution-contract dependency. Validate both streaming and non-streaming with fake retrieval plus output POST failures429/503 and later success, asserting actual side-effect counts and retained result identity; include pre-execution retrieval failure controls and affected continuation/approval/history matrix rows. Report smallest safe production scope, compatibility/migration decisions and remaining evidence gaps before implementation. Several contributors have volunteered (jstar0, Li-john1021, loyce-cheng); coordinate selection with Sophia after planning rather than assigning competing implementations. This is a non-minor bug investigation, not an implementation-ready decision.
github-actions commentedon Nov 12, 2026
Summary
With
background=True, a poll retrieves the completed response, runs its local tools, then POSTs the outputs. If that POST fails (429/503), the caller still holds the token for the response that requested the calls. Re-polling retrieves it again and re-runs the tools. #9031 is about this caller-held token. The adapter fix for #5394 only drops the token inside the loop (_chat_client.py~L1052-1059 non-streaming, ~L4972 streaming), so it cannot help here. I read code and docs only. I did not run anything, so reproduction rests on the issue's script and the independent report in the comments.#8574 is a different path (a provider invalidates delivery of a completed mixed approval/Host batch, where results are held in a durable outbox). It shares a need to retain exact provider-bound results, so the two designs should stay compatible. They are not duplicates.
Recommendation
Code change is warranted, but not implementation-ready. Money-moving tools can run twice on a documented recovery path, and docs alone (the issue's option 3) leave that unsafe. Because this touches the function loop, spec 004 and core-team sign-off are required first (see
python/AGENTS.md). I suggest the smallest safe scope below, plus a documentation note.Options compared
Recommended scope: A for the narrow case, with B as the fallback for everything A cannot prove. Retain only completed local outputs, keyed by provider response ID plus call occurrence (not global
call_id). Resubmit them on re-poll of that same response in the same session. If there is no session, a stale token, or an unknown outcome, raise an explicit error that says tools may have run. Never silently re-run.Identity and states
call_ids across responses must stay distinct.Proposed plan (proposal only; nothing changed)
Core-team decisions first: A+B scope above, error type and token semantics, whether retained state is serialized with the session, and any change to the spec's matrix rows (continuation, approval, history).
Core function loop: record completed outputs per occurrence before the next provider await; clear only after the provider accepts and continuation state advances.
OpenAI adapter: on re-poll of a response with retained outputs, submit them instead of re-retrieving and re-executing. Cover streaming and non-streaming. Handle stale tokens and middleware replacing the options dict (the token must not be the only guard).
Tests, using fake retrieval and a fake output POST that fails with 429 and 503, then succeeds, for both modes:
Update spec 004 only in the affected sections and add a short docs note that re-polling after a failed poll can run tools unless retained, and that side-effecting tools should still be idempotent.
Done when the counted-side-effect tests pass in both modes and the spec matrix rows are mapped to tests.
Remaining evidence gaps
Coordination
sophia-ramsey owns this issue. I suggest she picks one implementer after the core team confirms scope, rather than parallel PRs. jstar0, Li-john1021 and loyce-cheng have volunteered. Others could help with the regression fixtures. The idempotency-key advice in the comments is a useful application-side workaround but not a framework fix.