fix: serialize tool result delivery to preserve message ordering#21231
fix: serialize tool result delivery to preserve message ordering#21231joshavant merged 2 commits intoopenclaw:mainfrom
Conversation
| const task = (toolResultChain = toolResultChain.then(async () => { | ||
| const { text, skip } = normalizeStreamingText(payload); | ||
| if (skip) { | ||
| return; | ||
| } | ||
| await params.typingSignals.signalTextDelta(text); | ||
| await onToolResult({ | ||
| text, | ||
| mediaUrls: payload.mediaUrls, | ||
| }); | ||
| })) | ||
| .catch((err) => { | ||
| logVerbose(`tool result delivery failed: ${String(err)}`); | ||
| }) | ||
| .finally(() => { | ||
| params.pendingToolTasks.delete(task); | ||
| }); | ||
| params.pendingToolTasks.add(task); | ||
| }; |
There was a problem hiding this comment.
error in the chain breaks serialization — .catch() must be part of the chain assignment
the assignment toolResultChain = toolResultChain.then(...) happens before .catch() is applied, so if the async function throws, toolResultChain contains a rejected promise and subsequent tool results will fail to chain
fix by moving the assignment after error handling:
| const task = (toolResultChain = toolResultChain.then(async () => { | |
| const { text, skip } = normalizeStreamingText(payload); | |
| if (skip) { | |
| return; | |
| } | |
| await params.typingSignals.signalTextDelta(text); | |
| await onToolResult({ | |
| text, | |
| mediaUrls: payload.mediaUrls, | |
| }); | |
| })) | |
| .catch((err) => { | |
| logVerbose(`tool result delivery failed: ${String(err)}`); | |
| }) | |
| .finally(() => { | |
| params.pendingToolTasks.delete(task); | |
| }); | |
| params.pendingToolTasks.add(task); | |
| }; | |
| const task = toolResultChain = toolResultChain.then(async () => { | |
| const { text, skip } = normalizeStreamingText(payload); | |
| if (skip) { | |
| return; | |
| } | |
| await params.typingSignals.signalTextDelta(text); | |
| await onToolResult({ | |
| text, | |
| mediaUrls: payload.mediaUrls, | |
| }); | |
| }) | |
| .catch((err) => { | |
| logVerbose(`tool result delivery failed: ${String(err)}`); | |
| }); | |
| params.pendingToolTasks.add(task); | |
| task.finally(() => { | |
| params.pendingToolTasks.delete(task); | |
| }); |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/auto-reply/reply/agent-runner-execution.ts
Line: 377-395
Comment:
error in the chain breaks serialization — `.catch()` must be part of the chain assignment
the assignment `toolResultChain = toolResultChain.then(...)` happens before `.catch()` is applied, so if the async function throws, `toolResultChain` contains a rejected promise and subsequent tool results will fail to chain
fix by moving the assignment after error handling:
```suggestion
const task = toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
})
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
});
params.pendingToolTasks.add(task);
task.finally(() => {
params.pendingToolTasks.delete(task);
});
```
How can I resolve this? If you propose a fix, please make it concise.| const task = (toolResultChain = toolResultChain.then(async () => { | ||
| const { text, skip } = normalizeStreamingText(payload); | ||
| if (skip) { | ||
| return; | ||
| } | ||
| await params.typingSignals.signalTextDelta(text); | ||
| await onToolResult({ | ||
| text, | ||
| mediaUrls: payload.mediaUrls, | ||
| }); | ||
| })) | ||
| .catch((err) => { | ||
| logVerbose(`tool result delivery failed: ${String(err)}`); | ||
| }) | ||
| .finally(() => { | ||
| params.pendingToolTasks.delete(task); | ||
| }); | ||
| params.pendingToolTasks.add(task); | ||
| }; |
There was a problem hiding this comment.
error in chain breaks serialization — .catch() must be inside the assignment parentheses
toolResultChain is assigned the promise before .catch() is applied, so if the async function throws, subsequent tool results chain onto a rejected promise and fail immediately
move .catch() inside the assignment:
| const task = (toolResultChain = toolResultChain.then(async () => { | |
| const { text, skip } = normalizeStreamingText(payload); | |
| if (skip) { | |
| return; | |
| } | |
| await params.typingSignals.signalTextDelta(text); | |
| await onToolResult({ | |
| text, | |
| mediaUrls: payload.mediaUrls, | |
| }); | |
| })) | |
| .catch((err) => { | |
| logVerbose(`tool result delivery failed: ${String(err)}`); | |
| }) | |
| .finally(() => { | |
| params.pendingToolTasks.delete(task); | |
| }); | |
| params.pendingToolTasks.add(task); | |
| }; | |
| const task = (toolResultChain = toolResultChain.then(async () => { | |
| const { text, skip } = normalizeStreamingText(payload); | |
| if (skip) { | |
| return; | |
| } | |
| await params.typingSignals.signalTextDelta(text); | |
| await onToolResult({ | |
| text, | |
| mediaUrls: payload.mediaUrls, | |
| }); | |
| }) | |
| .catch((err) => { | |
| logVerbose(`tool result delivery failed: ${String(err)}`); | |
| })) | |
| .finally(() => { | |
| params.pendingToolTasks.delete(task); | |
| }); |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/auto-reply/reply/agent-runner-execution.ts
Line: 377-395
Comment:
error in chain breaks serialization — `.catch()` must be inside the assignment parentheses
`toolResultChain` is assigned the promise before `.catch()` is applied, so if the async function throws, subsequent tool results chain onto a rejected promise and fail immediately
move `.catch()` inside the assignment:
```suggestion
const task = (toolResultChain = toolResultChain.then(async () => {
const { text, skip } = normalizeStreamingText(payload);
if (skip) {
return;
}
await params.typingSignals.signalTextDelta(text);
await onToolResult({
text,
mediaUrls: payload.mediaUrls,
});
})
.catch((err) => {
logVerbose(`tool result delivery failed: ${String(err)}`);
}))
.finally(() => {
params.pendingToolTasks.delete(task);
});
```
How can I resolve this? If you propose a fix, please make it concise.Tool result callbacks were dispatched as fire-and-forget tasks via pendingToolTasks, with no ordering guarantee. When multiple tool calls completed near-simultaneously, their typing signals and message sends raced through independent async paths, causing out-of-order delivery to the user — particularly visible on Telegram. Replace the fire-and-forget Set<Promise> pattern with a serialized promise chain (toolResultChain). Each tool result now awaits the previous one before starting its typing signal and delivery, ensuring messages arrive in the order they were produced by the agent. The existing pendingToolTasks tracking is preserved so the post-run Promise.allSettled() drain still works correctly. Fixes openclaw#11044
ac53614 to
68adbf5
Compare
|
Merged via squash. Thanks @ahdernasr! |
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
|
Hey @joshavant , found another issue in the same dispatch code. Tool error payloads (isError: true) leak to the user channel, exposing file paths and internal errors. Put up a fix: #23147. Single guard in resolveToolDeliveryPayload(), 3 tests, covers #17828 #12928 #20279. |
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant (cherry picked from commit e321f21) # Conflicts: # src/auto-reply/reply/agent-runner.runreplyagent.test.ts
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant (cherry picked from commit e321f21) # Conflicts: # src/auto-reply/reply/agent-runner-execution.ts # src/auto-reply/reply/agent-runner.runreplyagent.test.ts
…nclaw#21231) Merged via /review-pr -> /prepare-pr -> /merge-pr. Prepared head SHA: 68adbf5 Co-authored-by: ahdernasr <[email protected]> Co-authored-by: joshavant <[email protected]> Reviewed-by: @joshavant
Summary
Fixes #11044 — outbound Telegram messages arrive out of order when multiple tool results fire concurrently.
Problem
Tool result callbacks in
agent-runner-execution.tsare dispatched as fire-and-forget async tasks viapendingToolTasks. Each callback independently runs through typing signals and message delivery with no ordering guarantee. When multiple tool calls complete near-simultaneously, network latency jitter causes messages to arrive at the user in the wrong order.This is consistently visible on Telegram and has been confirmed by multiple users.
Root Cause
The
onToolResulthandler creates independent promises added to aSet<Promise>:The set is only
Promise.allSettled()after the run completes, by which point messages are already out of order.Fix
Replace the fire-and-forget pattern with a serialized promise chain. Each tool result awaits completion of the previous one before starting its own typing signal and delivery:
The IIFE creates the chain once per agent run. Each invocation appends to the chain, guaranteeing FIFO delivery. The existing
pendingToolTaskstracking is preserved so the post-run drain (Promise.allSettled) still works correctly.What this does NOT change
BlockReplyPipelineandsendChainsendChain(enqueue())normalizeStreamingText, heartbeat stripping, silent reply detection are all untouchedTest
Added a test that fires two tool results concurrently where the first has higher simulated latency than the second. Verifies delivery order matches dispatch order (FIFO), not completion order.
Greptile Summary
This PR introduces serialized tool result delivery to fix out-of-order message delivery on Telegram when multiple tool calls complete concurrently. The fix replaces fire-and-forget async task dispatching with a promise chain that ensures FIFO delivery order.
Key changes:
onToolResulthandler in an IIFE that creates a closure with atoolResultChainvariable.then(), guaranteeing sequential executionpendingToolTaskstracking for post-run drainCritical issue found:
.catch()error handler is applied AFTER the chain assignment, meaningtoolResultChaincontains an unhandled promise. If any tool result throws an error, subsequent tool results will chain onto a rejected promise and fail immediately, breaking the serialization guarantee. The.catch()must be moved inside the assignment parentheses to ensure the chain continues after errors (see inline comment).Confidence Score: 2/5
.catch()handler is applied after the chain assignment, sotoolResultChainstores an unhandled promise. When any tool result throws, subsequent results will chain onto a rejected promise and fail immediately instead of continuing. This defeats the serialization guarantee and could cause message loss or ordering issues in error scenarios.Last reviewed commit: ac53614
(2/5) Greptile learns from your feedback when you react with thumbs up/down!