Skip to content

fix: add hard timeout to memory_search to prevent session wedging#993

Open
BingqingLyu wants to merge 2 commits into
mainfrom
fork-pr-49794-fix-memory-search-timeout
Open

fix: add hard timeout to memory_search to prevent session wedging#993
BingqingLyu wants to merge 2 commits into
mainfrom
fork-pr-49794-fix-memory-search-timeout

Conversation

@BingqingLyu

@BingqingLyu BingqingLyu commented Apr 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • Problem: A stalled memory_search tool call (e.g. iCloud/network mount in extraPaths, unresponsive embedding provider, or slow manager init) can wedge the entire agent turn indefinitely — the user gets no reply and the session appears dead.
  • Why it matters: This is a liveness bug. One memory tool call can paralyze a live session even though the gateway and channel connection are healthy. Users see a "bot is dead" experience with no error, no recovery, and no visibility.
  • What changed: Wrapped the full memory_search execution (manager init + search) with a 30-second hard timeout using Promise.race. On timeout, the existing catch block returns the standard { disabled: true, unavailable: true } result, allowing the assistant to continue the turn with a degraded reply like "memory retrieval is unavailable right now."
  • What did NOT change: No changes to the memory manager, sync pipeline, embedding timeouts, or watcher infrastructure. The timeout is applied at the tool boundary only — internal memory subsystem behavior is untouched.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Memory / storage
  • Skills / tool execution

Linked Issue/PR

User-visible / Behavior Changes

  • memory_search now fails open after 30 seconds instead of hanging forever
  • The assistant receives disabled: true and can tell the user memory is unavailable rather than going silent
  • No config changes required — the timeout is a safety net, not a tunable

Security Impact (required)

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command/tool execution surface changed? No
  • Data access scope changed? No

Repro + Verification

Environment

  • OS: macOS (reported with iCloud/Obsidian vault in extraPaths)
  • Runtime: Node 22+
  • Model/provider: Any (embedding provider: OpenAI text-embedding-3-small in report)
  • Integration/channel: Telegram (reported), affects all channels
  • Relevant config: agents.defaults.memorySearch.extraPaths pointing at iCloud/network mount

Steps

  1. Configure memorySearch.extraPaths with a slow/flaky mount (iCloud, network drive)
  2. Trigger a message that invokes memory_search
  3. Memory manager init or search hangs on file I/O

Expected

  • Tool returns { disabled: true } after timeout, assistant continues with degraded reply

Actual (before fix)

  • Tool call hangs forever, turn never completes, user gets no reply

Evidence

  • Failing test/log before + passing after
pnpm test -- src/agents/tools/memory-tool.test.ts   # 3 passed (including new timeout test)

The new test uses vi.useFakeTimers() to simulate a never-resolving search, advances past 30s, and verifies the tool returns { disabled: true, error: "...timed out" }.

Human Verification (required)

  • Verified scenarios: Timeout fires correctly on stalled search (fake timer test), existing quota/error unavailable payloads still work
  • Edge cases checked: Timer cleanup on success (finally block with clearTimeout), never-resolving promise doesn't leak
  • What I did not verify: Live end-to-end test with actual iCloud mount stall (requires specific filesystem conditions)

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? Yes
  • Config/env changes? No
  • Migration needed? No

Failure Recovery (if this breaks)

  • How to disable/revert this change quickly: Revert this commit
  • Files/config to restore: None
  • Known bad symptoms: If 30s is too aggressive for slow embedding providers, searches may time out prematurely. The existing per-embedding timeouts (60s remote / 5min local) are still in place inside the manager — this outer timeout catches the case where the manager never reaches the embedding call at all.

Risks and Mitigations

  • Risk: 30s timeout may be too short for very slow QMD/embedding setups
    • Mitigation: The timeout only covers total wall-clock time including manager init. Normal searches complete in <5s. The 30s value provides generous headroom while still preventing indefinite hangs. Per-embedding timeouts inside the manager are unchanged.

AI-assisted: Yes (lightly tested via unit tests; I understand what the code does)
Testing degree: Fully tested with 1 new unit test using fake timers

A stalled memory manager init or search (e.g. iCloud/network mount in
extraPaths) can wedge the entire agent turn indefinitely — the user
gets no reply and the session appears dead.

Wrap the full memory_search execution (manager init + search) with a
30s hard timeout using Promise.race. On timeout, the existing catch
block returns the standard unavailable result with disabled=true,
allowing the assistant to continue the turn with a degraded reply.

Closes openclaw#49524
Address review feedback: distinguish timeout from generic provider errors
in the unavailable result warning/action fields, and strengthen test
assertions to use expectUnavailableMemorySearchDetails.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

memory_search can stall a live session when the memory manager/tool-result path does not fail open

2 participants