Skip to content

fix: serialize tool result delivery to preserve message ordering#21231

Merged
joshavant merged 2 commits intoopenclaw:mainfrom
ahdernasr:fix/serialize-tool-result-delivery
Feb 20, 2026
Merged

fix: serialize tool result delivery to preserve message ordering#21231
joshavant merged 2 commits intoopenclaw:mainfrom
ahdernasr:fix/serialize-tool-result-delivery

Conversation

@ahdernasr
Copy link
Copy Markdown
Contributor

@ahdernasr ahdernasr commented Feb 19, 2026

Summary

Fixes #11044 — outbound Telegram messages arrive out of order when multiple tool results fire concurrently.

Problem

Tool result callbacks in agent-runner-execution.ts are dispatched as fire-and-forget async tasks via pendingToolTasks. Each callback independently runs through typing signals and message delivery with no ordering guarantee. When multiple tool calls complete near-simultaneously, network latency jitter causes messages to arrive at the user in the wrong order.

Tool A completes → fires async send (slow network) ──────────────→ arrives 2nd
Tool B completes → fires async send (fast network) ───→ arrives 1st

This is consistently visible on Telegram and has been confirmed by multiple users.

Root Cause

The onToolResult handler creates independent promises added to a Set<Promise>:

const task = (async () => { /* send */ })();
params.pendingToolTasks.add(task);  // No ordering — runs concurrently

The set is only Promise.allSettled() after the run completes, by which point messages are already out of order.

Fix

Replace the fire-and-forget pattern with a serialized promise chain. Each tool result awaits completion of the previous one before starting its own typing signal and delivery:

let toolResultChain: Promise<void> = Promise.resolve();
return (payload: ReplyPayload) => {
  const task = (toolResultChain = toolResultChain.then(async () => {
    // normalize, signal typing, deliver — in order
  }))
    .catch(/* ... */)
    .finally(() => { params.pendingToolTasks.delete(task); });
  params.pendingToolTasks.add(task);
};

The IIFE creates the chain once per agent run. Each invocation appends to the chain, guaranteeing FIFO delivery. The existing pendingToolTasks tracking is preserved so the post-run drain (Promise.allSettled) still works correctly.

What this does NOT change

  • Block reply delivery — already serialized via BlockReplyPipeline and sendChain
  • Reply dispatcher — already serialized via its own sendChain (enqueue())
  • Tool result content/filteringnormalizeStreamingText, heartbeat stripping, silent reply detection are all untouched
  • Error handling — individual failures are still caught and logged without breaking the chain

Test

Added a test that fires two tool results concurrently where the first has higher simulated latency than the second. Verifies delivery order matches dispatch order (FIFO), not completion order.

25/25 tests passing

Greptile Summary

This PR introduces serialized tool result delivery to fix out-of-order message delivery on Telegram when multiple tool calls complete concurrently. The fix replaces fire-and-forget async task dispatching with a promise chain that ensures FIFO delivery order.

Key changes:

  • Wraps onToolResult handler in an IIFE that creates a closure with a toolResultChain variable
  • Each tool result callback appends to the chain via .then(), guaranteeing sequential execution
  • Preserves existing pendingToolTasks tracking for post-run drain
  • Adds test verifying delivery order matches dispatch order despite variable latency

Critical issue found:

  • The .catch() error handler is applied AFTER the chain assignment, meaning toolResultChain contains an unhandled promise. If any tool result throws an error, subsequent tool results will chain onto a rejected promise and fail immediately, breaking the serialization guarantee. The .catch() must be moved inside the assignment parentheses to ensure the chain continues after errors (see inline comment).

Confidence Score: 2/5

  • This PR is NOT safe to merge due to a critical error handling bug
  • The implementation has the right idea (promise chain serialization) but contains a critical bug where errors break the chain. The .catch() handler is applied after the chain assignment, so toolResultChain stores an unhandled promise. When any tool result throws, subsequent results will chain onto a rejected promise and fail immediately instead of continuing. This defeats the serialization guarantee and could cause message loss or ordering issues in error scenarios.
  • src/auto-reply/reply/agent-runner-execution.ts requires immediate attention to fix the error handling bug before merge

Last reviewed commit: ac53614

(2/5) Greptile learns from your feedback when you react with thumbs up/down!

Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

Comment on lines +377 to +395
const task = (toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
}))
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
})
.finally(() => {
params.pendingToolTasks.delete(task);
});
params.pendingToolTasks.add(task);
};
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error in the chain breaks serialization — .catch() must be part of the chain assignment

the assignment toolResultChain = toolResultChain.then(...) happens before .catch() is applied, so if the async function throws, toolResultChain contains a rejected promise and subsequent tool results will fail to chain

fix by moving the assignment after error handling:

Suggested change
const task = (toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
}))
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
})
.finally(() => {
params.pendingToolTasks.delete(task);
});
params.pendingToolTasks.add(task);
};
const task = toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
})
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
});
params.pendingToolTasks.add(task);
task.finally(() => {
params.pendingToolTasks.delete(task);
});
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/auto-reply/reply/agent-runner-execution.ts
Line: 377-395

Comment:
error in the chain breaks serialization — `.catch()` must be part of the chain assignment

the assignment `toolResultChain = toolResultChain.then(...)` happens before `.catch()` is applied, so if the async function throws, `toolResultChain` contains a rejected promise and subsequent tool results will fail to chain

fix by moving the assignment after error handling:

```suggestion
                    const task = toolResultChain = toolResultChain.then(async () => {
                      const { text, skip } = normalizeStreamingText(payload);
                      if (skip) {
                        return;
                      }
                      await params.typingSignals.signalTextDelta(text);
                      await onToolResult({
                        text,
                        mediaUrls: payload.mediaUrls,
                      });
                    })
                      .catch((err) => {
                        logVerbose(`tool result delivery failed: ${String(err)}`);
                      });
                    params.pendingToolTasks.add(task);
                    task.finally(() => {
                      params.pendingToolTasks.delete(task);
                    });
```

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines +377 to +395
const task = (toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
}))
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
})
.finally(() => {
params.pendingToolTasks.delete(task);
});
params.pendingToolTasks.add(task);
};
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error in chain breaks serialization — .catch() must be inside the assignment parentheses

toolResultChain is assigned the promise before .catch() is applied, so if the async function throws, subsequent tool results chain onto a rejected promise and fail immediately

move .catch() inside the assignment:

Suggested change
const task = (toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
}))
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
})
.finally(() => {
params.pendingToolTasks.delete(task);
});
params.pendingToolTasks.add(task);
};
const task = (toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
})
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
}))
.finally(() => {
params.pendingToolTasks.delete(task);
});
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/auto-reply/reply/agent-runner-execution.ts
Line: 377-395

Comment:
error in chain breaks serialization — `.catch()` must be inside the assignment parentheses

`toolResultChain` is assigned the promise before `.catch()` is applied, so if the async function throws, subsequent tool results chain onto a rejected promise and fail immediately

move `.catch()` inside the assignment:

```suggestion
                    const task = (toolResultChain = toolResultChain.then(async () => {
                      const { text, skip } = normalizeStreamingText(payload);
                      if (skip) {
                        return;
                      }
                      await params.typingSignals.signalTextDelta(text);
                      await onToolResult({
                        text,
                        mediaUrls: payload.mediaUrls,
                      });
                    })
                      .catch((err) => {
                        logVerbose(`tool result delivery failed: ${String(err)}`);
                      }))
                      .finally(() => {
                        params.pendingToolTasks.delete(task);
                      });
```

How can I resolve this? If you propose a fix, please make it concise.

@joshavant joshavant assigned joshavant and unassigned joshavant Feb 20, 2026
Berg and others added 2 commits February 19, 2026 17:13
Tool result callbacks were dispatched as fire-and-forget tasks via
pendingToolTasks, with no ordering guarantee. When multiple tool calls
completed near-simultaneously, their typing signals and message sends
raced through independent async paths, causing out-of-order delivery
to the user — particularly visible on Telegram.

Replace the fire-and-forget Set<Promise> pattern with a serialized
promise chain (toolResultChain). Each tool result now awaits the
previous one before starting its typing signal and delivery, ensuring
messages arrive in the order they were produced by the agent.

The existing pendingToolTasks tracking is preserved so the post-run
Promise.allSettled() drain still works correctly.

Fixes openclaw#11044
@joshavant joshavant force-pushed the fix/serialize-tool-result-delivery branch from ac53614 to 68adbf5 Compare February 20, 2026 01:17
@joshavant joshavant merged commit e321f21 into openclaw:main Feb 20, 2026
23 checks passed
@joshavant
Copy link
Copy Markdown
Contributor

Merged via squash.

Thanks @ahdernasr!

vignesh07 pushed a commit to pahdo/openclaw that referenced this pull request Feb 20, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
anisoptera pushed a commit to anisoptera/openclaw that referenced this pull request Feb 20, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
rodrigogs pushed a commit to rodrigogs/openclaw that referenced this pull request Feb 20, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
Hansen1018 added a commit to Hansen1018/openclaw that referenced this pull request Feb 21, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
vincentkoc pushed a commit that referenced this pull request Feb 21, 2026
)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
vincentkoc pushed a commit that referenced this pull request Feb 21, 2026
)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
mmyyfirstb pushed a commit to mmyyfirstb/openclaw that referenced this pull request Feb 21, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
@ahdernasr
Copy link
Copy Markdown
Contributor Author

ahdernasr commented Feb 22, 2026

Hey @joshavant , found another issue in the same dispatch code. Tool error payloads (isError: true) leak to the user channel, exposing file paths and internal errors. Put up a fix: #23147. Single guard in resolveToolDeliveryPayload(), 3 tests, covers #17828 #12928 #20279.

obviyus pushed a commit to guirguispierre/openclaw that referenced this pull request Feb 22, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
hughdidit pushed a commit to hughdidit/DAISy-Agency that referenced this pull request Mar 1, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant

(cherry picked from commit e321f21)

# Conflicts:
#	src/auto-reply/reply/agent-runner.runreplyagent.test.ts
hughdidit pushed a commit to hughdidit/DAISy-Agency that referenced this pull request Mar 3, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant

(cherry picked from commit e321f21)

# Conflicts:
#	src/auto-reply/reply/agent-runner-execution.ts
#	src/auto-reply/reply/agent-runner.runreplyagent.test.ts
zooqueen pushed a commit to hanzoai/bot that referenced this pull request Mar 6, 2026
…nclaw#21231)

Merged via /review-pr -> /prepare-pr -> /merge-pr.

Prepared head SHA: 68adbf5
Co-authored-by: ahdernasr <[email protected]>
Co-authored-by: joshavant <[email protected]>
Reviewed-by: @joshavant
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Outbound Telegram messages arrive out of order when multiple tool results fire concurrently

2 participants