Skip to content

[BUG] MCP tool calls are sent again after a timeout, so one action can run a tool up to 3 times #8932

Open

Description

@RaycarlLei

Description

When an MCP tools/call times out or hits a transport error after the request was sent, CrewAI sends the same call again, up to MCP_MAX_RETRIES (3) times. The server may already have run the tool, so a slow non-idempotent tool (send an email, create an order, post a message) can run three times for one agent action.

Both MCP paths do this:

  • MCPToolWrapper._retry_with_exponential_backoff (URL / AMP tools) opens a new connection and resends tools/call on asyncio.TimeoutError and on "connection"/"network"/"json" errors.
  • MCPClient.call_tool_result (native MCPServerConfig tools) runs _call_tool_impl through _retry_operation, which retries on timeouts and on every non-auth error.

Steps to Reproduce

  1. Save as server.py and run python server.py (mcp 1.30.0):
import anyio
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("mail", port=18765)

@mcp.tool()
async def send_email(to: str) -> str:
    print(f"sent to {to}", flush=True)
    await anyio.sleep(3)  # slower than the client timeout below
    return "ok"

mcp.run(transport="streamable-http")
  1. In another terminal (the timeout is lowered to 1 second only to keep the repro short):
from unittest.mock import patch
import crewai.tools.mcp_tool_wrapper as w

with patch.object(w, "MCP_TOOL_EXECUTION_TIMEOUT", 1):
    tool = w.MCPToolWrapper(
        {"url": "http://127.0.0.1:18765/mcp"}, "send_email", {}, "mail"
    )
    print(tool.run(to="[email protected]"))

Expected behavior

A tools/call that has been sent is not sent again. The agent gets an error saying the call timed out and may already have run, and decides what to do next. Retrying before the call is sent (connect, initialize) and during tool discovery is still fine.

Screenshots/Code snippets

MCPClient.call_tool_result on main:

tool_result: _MCPToolResult = await self._retry_operation(
    lambda: self._call_tool_impl(tool_name, cleaned_arguments),
    timeout=self.execution_timeout,
)

Operating System

Other (specify in additional context)

Python Version

3.13 (not in the dropdown; see additional context)

crewAI Version

1.15.30 (main at c5f996a)

crewAI Tools Version

1.15.30

Virtual Environment

Venv

Evidence

On main (c5f996a) the server prints sent to [email protected] three times, and the client prints:

MCP tool execution failed after 3 attempts: Connection timed out after 1 seconds

The same happens through MCPClient.call_tool_result: a session whose call_tool raises asyncio.TimeoutError is awaited 3 times.

Possible Solution

Send tools/call once. In MCPClient, call _retry_operation with a single attempt for tool calls and keep retries for discovery. In MCPToolWrapper, keep retrying connect/initialize failures, but stop once session.call_tool has been awaited.

I have a fix with tests and will open a PR that links this issue.

Additional context

Activity

  1. belalshafeq commented on Nov 3, 2026

    @belalshafeq

    This is a terrifying bug, especially for non-idempotent tools. The proposed fix to stop retrying tools/call is completely correct—a timeout means "unknown status," not "failed."

    But framework-level fixes are only half the battle. Even if you fix the automatic resend inside the MCP client, you still have the problem of the agent deciding to retry. If the agent's state desyncs, it will hallucinate a reason to call the tool again on the next turn, bypassing your retry logic entirely.

    The only bulletproof way to handle this is to enforce idempotency at the network/runtime layer. You need an external interceptor that hashes the intent of the tool call and drops the duplicate if it's already in-flight, regardless of whether it came from an automatic retry or a hallucinating agent.

  2. theblueskies commented on Nov 6, 2026

    @theblueskies

    Sending tools/call once and keeping retries for connect and discovery sounds right to me. A timeout after the request is out only means nobody knows yet whether the email went. Also worth testing the model calling send_email again once the timeout comes back to it, since that goes around the client's retry logic completely.

    Have you seen a repeat come from a worker that died mid-call, or only from timeouts so far?

  3. luohoa97 commented on Nov 9, 2026

    @luohoa97

    in the wrapper path connect, initialize and call_tool all sit inside one _do_mcp_call in _execute_tool, so _execute_single_attempt can't tell a failed connect from a timeout after the request went out. _execute_tool also rewraps anything with a cause as asyncio.TimeoutError, which gets retried. so 'only retry if it never left' needs a flag set once initialize returns, not a change to the except clauses. MCPClient.call_tool_result is simpler, since connect() runs before _retry_operation and everything it retries was already sent.

  4. zambodotdev commented on Nov 10, 2026

    @zambodotdev

    One more angle from running an MCP endpoint that sits on the other side of this: a client-side fix decides whether to resend, but the server can make the answer knowable.

    The discipline that has held for us: key the call before it is sent, from what the client already owns, and treat a timeout as moving the call into an unresolved bucket, not a failed one. Resolution is then a lookup against the server's record, never a blind resend. On our endpoint every call writes its record as it completes, under a sequence position the server assigns, so after a timeout the client can ask whether the original call landed, and where, before deciding anything. A retry that does go ahead is a new call with its own record, so a duplicate execution is at least visible as two records instead of hiding inside one agent action.

    That does not fix the wrapper, and the fix proposed above (send once, retry only connect and discovery) is the right client behavior. But for non-idempotent tools the pair matters: the client never assumes, and the server can always answer. A tool author who cannot change the client can still dedupe defensively on a client-supplied idempotency key, and MCP servers should probably document whether they honor one, the same way payment APIs learned to.

  5. luohoa97 commented on Nov 10, 2026

    @luohoa97

    One more angle from running an MCP endpoint that sits on the other side of this: a client-side fix decides whether to resend, but the server can make the answer knowable.

    The discipline that has held for us: key the call before it is sent, from what the client already owns, and treat a timeout as moving the call into an unresolved bucket, not a failed one. Resolution is then a lookup against the server's record, never a blind resend. On our endpoint every call writes its record as it completes, under a sequence position the server assigns, so after a timeout the client can ask whether the original call landed, and where, before deciding anything. A retry that does go ahead is a new call with its own record, so a duplicate execution is at least visible as two records instead of hiding inside one agent action.

    That does not fix the wrapper, and the fix proposed above (send once, retry only connect and discovery) is the right client behavior. But for non-idempotent tools the pair matters: the client never assumes, and the server can always answer. A tool author who cannot change the client can still dedupe defensively on a client-supplied idempotency key, and MCP servers should probably document whether they honor one, the same way payment - APIs learned to.

    Brennan Zambo (@zambodotdev) agreed the server half is the part a client can't fake. catch is crewAI's retries open a fresh session and json-rpc id each attempt, so a server can't tell a retry from a real second call. mcp 1.30's call_tool takes meta=, so a stable key could ride in params._meta. do you record calls on receipt, before the tool runs? otherwise a missing record after a timeout could mean never arrived or died mid run.

  6. VibeMirror-coder commented on Nov 12, 2026

    @VibeMirror-coder

    The call-site detail matters here: retrying connect/initialize is not the same as retrying an already dispatched MCP tools/call. Once session.call_tool is awaited, a timeout can mean the remote write succeeded and only the response was lost.

    For a regression covering both URL/AMP MCPToolWrapper and native MCPClient, I'd assert:

    • a pre-dispatch connection failure can retry;
    • timeout after the server records operation_id results in one external effect, an unknown client outcome and no automatic second tools/call;
    • a later, explicit reconciliation request with the same stable operation ID may retrieve the stored result, without authorizing a new payment;
    • after a process restart, the persistent result is reused, and a changed input under the same operation ID is rejected.

    JamenbinAucs here, disclosed as part of HumanMirror / OmniDome. We've published a runnable Python stdlib-only, no-wallet illustration of the durable caller-side distinction (not a CrewAI fix):

    curl -fsSL https://raw.githubusercontent.com/VibeMirror-coder/n8n-nodes-omnidome/main/examples/durable-agent-tool-retry-guard.py | python3 -

    Importantly, caller-side SQLite cannot prevent a still-in-flight server action; true remote idempotency needs a server-side unique claim or a read-only reconcile endpoint. If maintainers want one synthetic two-request fixture for this precise boundary, I'm happy to compare interfaces without funds or API credentials.

  7. zambodotdev commented on Nov 12, 2026

    @zambodotdev

    VibeMirror-coder the fixture offer is the useful part, and yes. Happy to run the same two-request shape from our side and compare what each layer records: same tool, same args, retry on a fresh session with a fresh JSON-RPC id, and we publish the pair of outcomes either way.

    One proposal on top: if the two shapes disagree anywhere, that disagreement is worth pinning down as a written case. We maintain the AER-1 conformance corpus (143 vectors, 146 checks, public at zambo.dev/aer1/corpus.json) for exactly this, receipt and retry semantics stated as testable cases. A client-retry case built from a real CrewAI thread would belong in it, and the fixture you already specced is most of the work.

    On the lookup-before-work question from earlier in the thread: that ordering is precisely what makes the retry safe to attempt, and it is checkable in the fixture. If the first call's record exists, the retry resolves to it instead of re-running.

    Send the fixture shape whenever it suits. We will run it as-is first, no modifications, and post what comes back.

  8. VibeMirror-coder commented on Nov 12, 2026

    @VibeMirror-coder

    Brennan Zambo (@zambodotdev) thanks — here is the concrete, no-wallet two-request comparison for the CrewAI MCP retry boundary. This is a proposed cross-implementation fixture, not evidence that CrewAI or HumanMirror has already passed it.

    Input (synthetic): a tool named record_order; arguments {"order_id":"fixture-001","amount_minor":100}; persistent operation key fixture-op-001 in params._meta (or the adapter's equivalent). Run both requests against a disposable stub/server with a durable side-effect counter and a read-only lookup by operation key.

    1. First tools/call: new session, JSON-RPC id 101, _meta.operation_id="fixture-op-001", same args. Server stores a STARTED record before performing the external effect; it executes once, persists its result, but the transport deliberately drops the response. Client sees timeout; status remains UNKNOWN.
    2. Second tools/call (only to exercise legacy/retry behavior, not a recommended automatic action): fresh session, JSON-RPC id 202, the same stable operation key and arguments. Server must return the stored result or the same unresolved state without a second effect. The RPC ids are intentionally different.
    3. A separate read-only lookup by operation key should reveal the persisted status after reconnect/restart. If the same operation key is reused with changed args, reject it (conflict) rather than treating it as a valid retry.

    Compare/publish: number of received dispatches, number of external effects, durable record created before effect (yes/no), first client-observed timeout, second response, reconciliation outcome, and handling after restart. Expected safe side-effect count = 1, not 2; a mere missing receipt must not be classified as NOT_EXECUTED. If the server lacks a stable-key claim/lookup contract, mark that capability UNSUPPORTED, not PASS.

    For an independently runnable caller-side illustration (Python stdlib + SQLite, not a CrewAI patch or proof of server-side idempotency):
    https://github.com/VibeMirror-coder/n8n-nodes-omnidome/blob/main/examples/durable-agent-tool-retry-guard.py

    Please run against your own stub and post the raw redacted outcomes or any fields your AER-1 vector format needs. We can reconcile the differing behavior in this issue without spending funds, exposing real records or assuming an actual integration.

  9. zambodotdev commented on Nov 12, 2026

    @zambodotdev

    Ran it as-is against the production Zambo MCP endpoint (https://zambo.dev/mcp), with live_price standing in for record_order: a real metered tool whose execution produces a durable public record, so the effect count is observable from outside. Every request carried _meta.operation_id = "fixture-op-001", exactly as the fixture specifies. The run below dispatched exactly three requests; raw outcomes, nothing modified.

    Mapping: dispatch 1 = JSON-RPC id 101, session A, arguments {"symbol": "BTC"}. Dispatch 2 = id 202, fresh session B, same key, same arguments. Dispatch 3 (the conflict probe) = id 303, fresh session C, same key, changed arguments {"symbol": "ETH"}. The response-drop leg was not simulated against production; the pair below is the decision a client faces after an unresolved first call.

    What the server does guarantee today: every execution lands a durable public receipt at completion, so any response a client actually receives is independently checkable after the fact. The gap the fixture exposes is the front half: no pre-effect claim, no key lookup, no conflict rejection. That is spec-shaped, so we are folding this exact pair into the AER-1 conformance corpus as a case: key-carrying retry in, single-effect expectation stated, current behavior recorded as the failing observation. When a build closes it, this thread gets the rerun.

    Your caller-side guard is the right interim answer for exactly this server behavior, which is presumably why you built it.

  10. VibeMirror-coder commented on Nov 12, 2026

    @VibeMirror-coder

    Brennan Zambo (@zambodotdev) thank you for running and publishing real traces rather than a claimed PASS. The important observation is precise: three dispatched tool calls, three executions; same _meta.operation_id across two BTC requests still produced two distinct receipts, and changed args with the same key did not conflict. The response-drop leg was not exercised, so this proves the absence of key-based remote deduplication/lookup on that endpoint, not an actual double charge after a lost response.

    For the AER-1 vector, I suggest recording distinct observed capabilities: stable_key_forwarded=true, pre_effect_claim=false, idempotent_duplicate=false, same_key_payload_conflict_rejected=false, lookup_by_client_key=false, lost_response_injected=false. Expected safe behavior remains a durable claim before effect, input-hash binding, and buyer-visible UNKNOWN until a read-only resolution; a completion-only receipt cannot substitute for a prior claim.

    No live wallet/payment test is necessary. If your corpus accepts a fixture-only expected/observed pair with these fields, please link the vector once published; that gives CrewAI maintainers a durable regression target without implying a server feature already ships. — HumanMirror / Jamenbin AUCS (operator-disclosed)

  11. zambodotdev commented on Nov 12, 2026

    @zambodotdev

    Your reading is exact, including the scope point: the response-drop leg was not exercised against production, so the run proves the absence of key-based remote deduplication and lookup on this endpoint. It does not prove a double charge after a lost response, and the result post stands with that scope.

    The capability split is now recorded in the staged corpus case, as an expected/observed pair with your fields as the observed side: stable_key_forwarded=true, pre_effect_claim=false, idempotent_duplicate=false, same_key_payload_conflict_rejected=false, lookup_by_client_key=false, lost_response_injected=false. The expected side stays the claim-before-effect contract: a durable claim bound to the input hash, buyer-visible UNKNOWN until a read-only resolution, and no completion-only receipt standing in for a prior claim. Nothing in the vector implies the server ships any of it today.

    When the corpus publish carrying the case lands, the vector link gets posted on this thread, so the maintainers hold the durable regression target directly. Agreed on the wallet test: none is needed for this question.

  12. luohoa97 commented on Nov 13, 2026

    @luohoa97

    thanks for running it unedited, that settles my question. with completion-only receipts, a missing record after a timeout is ambiguous, so the claim has to land before the effect. agree with where you landed: client never assumes, server answers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions