Skip to content

memory_search tool-level timeout orphans background embedding work #91718

Description

@NOVA-Openclaw

Problem

runMemorySearchToolWithDeadline() in extensions/memory-core/src/tools.ts uses Promise.race with a 15s timeout to fail the memory_search tool call open when the embedding backend stalls. When the timeout wins the race, the tool returns a clean "timed out" error to the agent.

However, the underlying embedding operation (and its retry loop) continues running in the background as an orphaned promise. No AbortSignal is propagated from the tool-level timeout to cancel the in-flight HTTP fetch or abort the embedQueryWithRetry loop.

Impact

When the embedding backend hangs (e.g., Ollama model scheduler wedge), the tool correctly returns after 15s, but:

  1. The orphaned embedQueryWithRetry continues retrying (up to 3 attempts × 60s timeout each = ~3-4 min of background work)
  2. Each retry holds an HTTP connection to the embedding backend
  3. On resource-constrained hosts sharing a GPU (e.g., Ollama + Stable Diffusion), this ties up the embedding backend for minutes after the tool has already given up
  4. If the backend recovers mid-retry, the orphaned work completes silently with no consumer for the result

Expected behavior

The tool-level timeout should propagate an AbortSignal to the search operation so that when the 15s deadline fires, the underlying embedQueryWithRetry loop and its HTTP fetches are cancelled — not just orphaned.

The per-call runEmbeddingOperationWithTimeout() already accepts and uses an AbortSignal correctly. The gap is that no signal flows from the tool-level deadline down to the search/embedding call chain.

Relevant code

  • extensions/memory-core/src/tools.tsrunMemorySearchToolWithDeadline() (Promise.race, no signal propagation)
  • extensions/memory-core/src/memory/manager-embedding-ops.tsembedQueryWithRetry() and runEmbeddingOperationWithTimeout() (already accept AbortSignal)
  • extensions/memory-core/src/memory/manager.ts:613embedQueryWithRetry call site during search

Suggested fix

Pass an AbortController into runMemorySearchToolWithDeadline, abort it on timeout, and thread the signal through manager.search()embedQueryWithRetry()runEmbeddingOperationWithTimeout(). The plumbing for signal acceptance already exists at the lower layers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions