Repository navigation
Description
Description
When an MCP tools/call times out or hits a transport error after the request was sent, CrewAI sends the same call again, up to MCP_MAX_RETRIES (3) times. The server may already have run the tool, so a slow non-idempotent tool (send an email, create an order, post a message) can run three times for one agent action.
Both MCP paths do this:
MCPToolWrapper._retry_with_exponential_backoff(URL / AMP tools) opens a new connection and resendstools/callonasyncio.TimeoutErrorand on "connection"/"network"/"json" errors.MCPClient.call_tool_result(nativeMCPServerConfigtools) runs_call_tool_implthrough_retry_operation, which retries on timeouts and on every non-auth error.
Steps to Reproduce
- Save as
server.pyand runpython server.py(mcp 1.30.0):
import anyio
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("mail", port=18765)
@mcp.tool()
async def send_email(to: str) -> str:
print(f"sent to {to}", flush=True)
await anyio.sleep(3) # slower than the client timeout below
return "ok"
mcp.run(transport="streamable-http")- In another terminal (the timeout is lowered to 1 second only to keep the repro short):
from unittest.mock import patch
import crewai.tools.mcp_tool_wrapper as w
with patch.object(w, "MCP_TOOL_EXECUTION_TIMEOUT", 1):
tool = w.MCPToolWrapper(
{"url": "http://127.0.0.1:18765/mcp"}, "send_email", {}, "mail"
)
print(tool.run(to="[email protected]"))Expected behavior
A tools/call that has been sent is not sent again. The agent gets an error saying the call timed out and may already have run, and decides what to do next. Retrying before the call is sent (connect, initialize) and during tool discovery is still fine.
Screenshots/Code snippets
MCPClient.call_tool_result on main:
tool_result: _MCPToolResult = await self._retry_operation(
lambda: self._call_tool_impl(tool_name, cleaned_arguments),
timeout=self.execution_timeout,
)Operating System
Other (specify in additional context)
Python Version
3.13 (not in the dropdown; see additional context)
crewAI Version
1.15.30 (main at c5f996a)
crewAI Tools Version
1.15.30
Virtual Environment
Venv
Evidence
On main (c5f996a) the server prints sent to [email protected] three times, and the client prints:
MCP tool execution failed after 3 attempts: Connection timed out after 1 seconds
The same happens through MCPClient.call_tool_result: a session whose call_tool raises asyncio.TimeoutError is awaited 3 times.
Possible Solution
Send tools/call once. In MCPClient, call _retry_operation with a single attempt for tool calls and keep retries for discovery. In MCPToolWrapper, keep retrying connect/initialize failures, but stop once session.call_tool has been awaited.
I have a fix with tests and will open a PR that links this issue.
Additional context
- OS: macOS 27.0.1 (not in the dropdown). Python 3.13.16.
- Related: Tool re-execution on task retry has no idempotency guard — duplicate payments, emails, trades possible #5802 covers tools re-running when a task is retried. This issue is the lower-level automatic resend inside one tool call.
- fix: report MCP isError as ToolFailure on the external URL path #7643 and Don't call asyncio.run() from MCPToolWrapper when a loop is running #7448 change the same wrapper (isError reporting, event loop handling) but not its retry behavior.
- AI disclosure: Claude Code (Claude Opus 5.5) found the bug, ran the reproduction above and wrote this report. Claude Code filed it on my behalf with my approval. Please add the
llm-generatedlabel; I don't have permission to add labels.
Activity
belalshafeq commentedon Nov 3, 2026
This is a terrifying bug, especially for non-idempotent tools. The proposed fix to stop retrying tools/call is completely correct—a timeout means "unknown status," not "failed."
But framework-level fixes are only half the battle. Even if you fix the automatic resend inside the MCP client, you still have the problem of the agent deciding to retry. If the agent's state desyncs, it will hallucinate a reason to call the tool again on the next turn, bypassing your retry logic entirely.
The only bulletproof way to handle this is to enforce idempotency at the network/runtime layer. You need an external interceptor that hashes the intent of the tool call and drops the duplicate if it's already in-flight, regardless of whether it came from an automatic retry or a hallucinating agent.
theblueskies commentedon Nov 6, 2026
Sending tools/call once and keeping retries for connect and discovery sounds right to me. A timeout after the request is out only means nobody knows yet whether the email went. Also worth testing the model calling send_email again once the timeout comes back to it, since that goes around the client's retry logic completely.
Have you seen a repeat come from a worker that died mid-call, or only from timeouts so far?
luohoa97 commentedon Nov 9, 2026
in the wrapper path connect, initialize and call_tool all sit inside one _do_mcp_call in _execute_tool, so _execute_single_attempt can't tell a failed connect from a timeout after the request went out. _execute_tool also rewraps anything with a cause as asyncio.TimeoutError, which gets retried. so 'only retry if it never left' needs a flag set once initialize returns, not a change to the except clauses. MCPClient.call_tool_result is simpler, since connect() runs before _retry_operation and everything it retries was already sent.
zambodotdev commentedon Nov 10, 2026
One more angle from running an MCP endpoint that sits on the other side of this: a client-side fix decides whether to resend, but the server can make the answer knowable.
The discipline that has held for us: key the call before it is sent, from what the client already owns, and treat a timeout as moving the call into an unresolved bucket, not a failed one. Resolution is then a lookup against the server's record, never a blind resend. On our endpoint every call writes its record as it completes, under a sequence position the server assigns, so after a timeout the client can ask whether the original call landed, and where, before deciding anything. A retry that does go ahead is a new call with its own record, so a duplicate execution is at least visible as two records instead of hiding inside one agent action.
That does not fix the wrapper, and the fix proposed above (send once, retry only connect and discovery) is the right client behavior. But for non-idempotent tools the pair matters: the client never assumes, and the server can always answer. A tool author who cannot change the client can still dedupe defensively on a client-supplied idempotency key, and MCP servers should probably document whether they honor one, the same way payment APIs learned to.
luohoa97 commentedon Nov 10, 2026
Brennan Zambo (@zambodotdev) agreed the server half is the part a client can't fake. catch is crewAI's retries open a fresh session and json-rpc id each attempt, so a server can't tell a retry from a real second call. mcp 1.30's call_tool takes meta=, so a stable key could ride in params._meta. do you record calls on receipt, before the tool runs? otherwise a missing record after a timeout could mean never arrived or died mid run.
VibeMirror-coder commentedon Nov 12, 2026
The call-site detail matters here: retrying connect/initialize is not the same as retrying an already dispatched MCP
tools/call. Oncesession.call_toolis awaited, a timeout can mean the remote write succeeded and only the response was lost.For a regression covering both URL/AMP
MCPToolWrapperand nativeMCPClient, I'd assert:operation_idresults in one external effect, an unknown client outcome and no automatic secondtools/call;JamenbinAucs here, disclosed as part of HumanMirror / OmniDome. We've published a runnable Python stdlib-only, no-wallet illustration of the durable caller-side distinction (not a CrewAI fix):
curl -fsSL https://raw.githubusercontent.com/VibeMirror-coder/n8n-nodes-omnidome/main/examples/durable-agent-tool-retry-guard.py | python3 -Importantly, caller-side SQLite cannot prevent a still-in-flight server action; true remote idempotency needs a server-side unique claim or a read-only reconcile endpoint. If maintainers want one synthetic two-request fixture for this precise boundary, I'm happy to compare interfaces without funds or API credentials.
zambodotdev commentedon Nov 12, 2026
VibeMirror-coder the fixture offer is the useful part, and yes. Happy to run the same two-request shape from our side and compare what each layer records: same tool, same args, retry on a fresh session with a fresh JSON-RPC id, and we publish the pair of outcomes either way.
One proposal on top: if the two shapes disagree anywhere, that disagreement is worth pinning down as a written case. We maintain the AER-1 conformance corpus (143 vectors, 146 checks, public at zambo.dev/aer1/corpus.json) for exactly this, receipt and retry semantics stated as testable cases. A client-retry case built from a real CrewAI thread would belong in it, and the fixture you already specced is most of the work.
On the lookup-before-work question from earlier in the thread: that ordering is precisely what makes the retry safe to attempt, and it is checkable in the fixture. If the first call's record exists, the retry resolves to it instead of re-running.
Send the fixture shape whenever it suits. We will run it as-is first, no modifications, and post what comes back.
VibeMirror-coder commentedon Nov 12, 2026
Brennan Zambo (@zambodotdev) thanks — here is the concrete, no-wallet two-request comparison for the CrewAI MCP retry boundary. This is a proposed cross-implementation fixture, not evidence that CrewAI or HumanMirror has already passed it.
Input (synthetic): a tool named
record_order; arguments{"order_id":"fixture-001","amount_minor":100}; persistent operation keyfixture-op-001inparams._meta(or the adapter's equivalent). Run both requests against a disposable stub/server with a durable side-effect counter and a read-only lookup by operation key.tools/call: new session, JSON-RPC id 101,_meta.operation_id="fixture-op-001", same args. Server stores aSTARTEDrecord before performing the external effect; it executes once, persists its result, but the transport deliberately drops the response. Client sees timeout; status remains UNKNOWN.tools/call(only to exercise legacy/retry behavior, not a recommended automatic action): fresh session, JSON-RPC id 202, the same stable operation key and arguments. Server must return the stored result or the same unresolved state without a second effect. The RPC ids are intentionally different.Compare/publish: number of received dispatches, number of external effects, durable record created before effect (yes/no), first client-observed timeout, second response, reconciliation outcome, and handling after restart. Expected safe side-effect count = 1, not 2; a mere missing receipt must not be classified as
NOT_EXECUTED. If the server lacks a stable-key claim/lookup contract, mark that capability UNSUPPORTED, not PASS.For an independently runnable caller-side illustration (Python stdlib + SQLite, not a CrewAI patch or proof of server-side idempotency):
https://github.com/VibeMirror-coder/n8n-nodes-omnidome/blob/main/examples/durable-agent-tool-retry-guard.py
Please run against your own stub and post the raw redacted outcomes or any fields your AER-1 vector format needs. We can reconcile the differing behavior in this issue without spending funds, exposing real records or assuming an actual integration.
zambodotdev commentedon Nov 12, 2026
Ran it as-is against the production Zambo MCP endpoint (https://zambo.dev/mcp), with live_price standing in for record_order: a real metered tool whose execution produces a durable public record, so the effect count is observable from outside. Every request carried _meta.operation_id = "fixture-op-001", exactly as the fixture specifies. The run below dispatched exactly three requests; raw outcomes, nothing modified.
Mapping: dispatch 1 = JSON-RPC id 101, session A, arguments {"symbol": "BTC"}. Dispatch 2 = id 202, fresh session B, same key, same arguments. Dispatch 3 (the conflict probe) = id 303, fresh session C, same key, changed arguments {"symbol": "ETH"}. The response-drop leg was not simulated against production; the pair below is the decision a client faces after an unresolved first call.
What the server does guarantee today: every execution lands a durable public receipt at completion, so any response a client actually receives is independently checkable after the fact. The gap the fixture exposes is the front half: no pre-effect claim, no key lookup, no conflict rejection. That is spec-shaped, so we are folding this exact pair into the AER-1 conformance corpus as a case: key-carrying retry in, single-effect expectation stated, current behavior recorded as the failing observation. When a build closes it, this thread gets the rerun.
Your caller-side guard is the right interim answer for exactly this server behavior, which is presumably why you built it.
VibeMirror-coder commentedon Nov 12, 2026
Brennan Zambo (@zambodotdev) thank you for running and publishing real traces rather than a claimed PASS. The important observation is precise: three dispatched tool calls, three executions; same
_meta.operation_idacross two BTC requests still produced two distinct receipts, and changed args with the same key did not conflict. The response-drop leg was not exercised, so this proves the absence of key-based remote deduplication/lookup on that endpoint, not an actual double charge after a lost response.For the AER-1 vector, I suggest recording distinct observed capabilities:
stable_key_forwarded=true,pre_effect_claim=false,idempotent_duplicate=false,same_key_payload_conflict_rejected=false,lookup_by_client_key=false,lost_response_injected=false. Expected safe behavior remains a durable claim before effect, input-hash binding, and buyer-visibleUNKNOWNuntil a read-only resolution; a completion-only receipt cannot substitute for a prior claim.No live wallet/payment test is necessary. If your corpus accepts a fixture-only expected/observed pair with these fields, please link the vector once published; that gives CrewAI maintainers a durable regression target without implying a server feature already ships. — HumanMirror / Jamenbin AUCS (operator-disclosed)
zambodotdev commentedon Nov 12, 2026
Your reading is exact, including the scope point: the response-drop leg was not exercised against production, so the run proves the absence of key-based remote deduplication and lookup on this endpoint. It does not prove a double charge after a lost response, and the result post stands with that scope.
The capability split is now recorded in the staged corpus case, as an expected/observed pair with your fields as the observed side: stable_key_forwarded=true, pre_effect_claim=false, idempotent_duplicate=false, same_key_payload_conflict_rejected=false, lookup_by_client_key=false, lost_response_injected=false. The expected side stays the claim-before-effect contract: a durable claim bound to the input hash, buyer-visible UNKNOWN until a read-only resolution, and no completion-only receipt standing in for a prior claim. Nothing in the vector implies the server ships any of it today.
When the corpus publish carrying the case lands, the vector link gets posted on this thread, so the maintainers hold the durable regression target directly. Agreed on the wallet test: none is needed for this question.
luohoa97 commentedon Nov 13, 2026
thanks for running it unedited, that settles my question. with completion-only receipts, a missing record after a timeout is ambiguous, so the claim has to land before the effect. agree with where you landed: client never assumes, server answers.