fix(memory): fall back to keyword search without sqlite by mushuiyu886 · Pull Request #93260 · openclaw/openclaw

mushuiyu886 · 2026-06-15T10:36:01Z

Summary

Problem: On Node runtimes where the builtin node:sqlite module is unavailable, memory_search previously lost the builtin indexed manager even though durable markdown memory files were still readable.
Solution: Keep SQLite/vector search as the canonical path when available, and only for the explicit missing-node:sqlite failure seam return a degraded keyword fallback manager over MEMORY.md, memory/*.md, and configured memory extraPaths.
What changed: The fallback returns bounded markdown keyword results with file paths, line ranges, truncated snippets, and debug/status metadata marking keyword-fallback / node-sqlite-unavailable.
What did not change: SQLite/vector indexing, QMD behavior, embeddings, provider selection, sync, runtime service APIs, tool schemas, SDK exports, config shape, migrations, and file formats are unchanged.
Why it matters / User impact: Users on Node builds without node:sqlite get a basic local memory recall path instead of an unavailable manager response for a runtime capability gap outside their control.
Fixes [Feature]: Add keyword fallback for memory_search when node:sqlite is unavailable #93150

Real behavior proof

Behavior addressed: memory_search returns degraded markdown keyword results when the current OpenClaw CLI process cannot load node:sqlite, instead of returning unavailable manager metadata. This exercises the production CLI path through node scripts/run-node.mjs memory ..., loadMemoryCommandConfig, withMemoryManagerForAgent, getMemorySearchManager, and the PR's KeywordFallbackMemoryManager.
Real environment tested: Local fix(memory): fall back to keyword search without sqlite #93260 worktree at commit 96d9ef309aead772a74e68e9aaece8021f94f7e2; git status clean; isolated OPENCLAW_HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, and workspace under /media/vdb/code/ai/aispace/openclaw-pr-93260-evidence; local markdown corpus containing MEMORY.md and memory/preferences.md; scoped preload that makes node:sqlite unavailable only for the memory index storage loader.
Exact steps or command run after this patch:

cd "/media/vdb/code/ai/aispace/openclaw-worktrees/pr-93260" && \
  env NODE_OPTIONS="--require /media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/block-node-sqlite.cjs" \
    OPENCLAW_HOME="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-home" \
    OPENCLAW_STATE_DIR="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-state" \
    OPENCLAW_CONFIG_PATH="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-openclaw.json" \
    OPENCLAW_TEST_FAST=1 \
    NO_COLOR=1 \
    node scripts/run-node.mjs memory status --agent main --json \
      > "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-status.json" \
      2> "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-status.stderr" && \
  env NODE_OPTIONS="--require /media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/block-node-sqlite.cjs" \
    OPENCLAW_HOME="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-home" \
    OPENCLAW_STATE_DIR="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-state" \
    OPENCLAW_CONFIG_PATH="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-openclaw.json" \
    OPENCLAW_TEST_FAST=1 \
    NO_COLOR=1 \
    node scripts/run-node.mjs memory search "orbit ramen" --agent main --max-results 1 --json \
      > "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search.json" \
      2> "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search.stderr" && \
  env NODE_OPTIONS="--require /media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/block-node-sqlite.cjs" \
    OPENCLAW_HOME="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-home" \
    OPENCLAW_STATE_DIR="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-state" \
    OPENCLAW_CONFIG_PATH="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-openclaw.json" \
    OPENCLAW_TEST_FAST=1 \
    NO_COLOR=1 \
    node scripts/run-node.mjs memory search "notpresent" --agent main --json \
      > "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search-empty.json" \
      2> "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search-empty.stderr"

The verification was recorded with daily-fix against the same head and evidence files:

DAILY_FIX="/home/0668001029/.claude/skills/daily-fix/scripts/daily_fix.sh" \
DAILY_FIX_CONTEXT="/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/context.env" \
"$DAILY_FIX" record-verification \
  "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-status.json" \
  "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search.json" \
  "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search-empty.json"

Evidence after fix:

memory status --agent main --json returned fallback status for the actual CLI manager:

[
  {
    "agentId": "main",
    "status": {
      "backend": "builtin",
      "provider": "keyword-fallback",
      "requestedProvider": "keyword-fallback",
      "workspaceDir": "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-workspace",
      "sources": ["memory"],
      "fallback": {
        "from": "builtin-sqlite",
        "reason": "SQLite support is unavailable in this Node runtime (missing node:sqlite). No such built-in module: node:sqlite | No such built-in module: node:sqlite | ERR_UNKNOWN_BUILTIN_MODULE"
      },
      "custom": {
        "fallback": {
          "mode": "keyword",
          "reason": "SQLite support is unavailable in this Node runtime (missing node:sqlite). No such built-in module: node:sqlite | No such built-in module: node:sqlite | ERR_UNKNOWN_BUILTIN_MODULE"
        }
      }
    },
    "scan": {
      "sources": [
        {
          "source": "memory",
          "totalFiles": 2,
          "issues": []
        }
      ],
      "totalFiles": 2,
      "issues": []
    }
  }
]

memory search "orbit ramen" --agent main --max-results 1 --json returned a markdown memory result:

{
  "results": [
    {
      "path": "memory/preferences.md",
      "startLine": 1,
      "endLine": 4,
      "score": 1,
      "textScore": 1,
      "snippet": "# Preferences\nFavorite ramen is shoyu.\nThe orbit codename is ORBIT-22.\n",
      "source": "memory"
    }
  ]
}

A no-match probe stayed empty instead of producing unrelated fallback hits:

{
  "results": []
}

The daily-fix verification marker binds those files to the current PR head:

{
  "head": "96d9ef309aead772a74e68e9aaece8021f94f7e2",
  "pending_diff_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "evidence_files": [
    "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-status.json",
    "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search.json",
    "/media/vdb/code/ai/aispace/openclaw-pr-93260-evidence/verify-search-empty.json"
  ]
}

Observed result after fix: The OpenClaw CLI process reached the memory-core manager resolution path, classified the simulated node:sqlite load failure as node-sqlite-unavailable, returned a builtin keyword-fallback manager, scanned the isolated markdown memory corpus, returned memory/preferences.md with the ORBIT-22 snippet for orbit ramen, and returned no rows for notpresent.
What was not tested: Full gateway/UI behavior, semantic/vector ranking parity, indexed SQLite performance, session transcript search, QMD, embeddings, provider calls, live service APIs, and a separate physical Node build without node:sqlite were not tested. The proof instead uses a scoped preload to make node:sqlite unavailable at the memory index loader seam while preserving the rest of OpenClaw state startup, so the real CLI command can exercise this PR's production fallback path.
Fix classification: Root cause fix

Review findings addressed

RF-001: Fixed for contributor-readiness purposes. ClawSweeper did not complete because of a Codex infrastructure/structured-output failure, so it produced no source-level author finding to patch. The PR now includes current-head real CLI behavior proof showing the no-node:sqlite path returns a degraded keyword fallback manager and markdown search results. A fresh ClawSweeper review or maintainer policy decision is still appropriate before merge because the prior bot review did not complete.

Regression Test Plan

Target test file: extensions/memory-core/src/memory/search-manager.test.ts
Scenario locked in: The regression forces MemoryIndexManager.get() to throw the wrapped missing-node:sqlite error, then verifies getMemorySearchManager() returns a keyword fallback manager, searches a real MEMORY.md / memory/*.md fixture, emits fallback debug metadata, preserves path/line/snippet metadata, and avoids a direct fs.readFile fallback corpus path.
Why this is the smallest reliable guardrail: The bug is at the memory-core manager-resolution seam, not in provider calls or gateway transport. The test isolates the exact SQLite initialization failure classification and the safe memory-file read boundary without requiring a live embedding provider or vector index.
Current-head CI: GitHub checks are aligned to 96d9ef309aead772a74e68e9aaece8021f94f7e2; CI, CodeQL Critical Quality, Dependency Guard, Real behavior proof, Workflow Sanity, OpenGrep PR Diff, iOS Periphery Dead Code, Labeler, Auto response, and ClawSweeper Dispatch are reported successful in the collected current-head checkout context.
New proof verification: The no-node:sqlite CLI proof above was recorded against HEAD 96d9ef309aead772a74e68e9aaece8021f94f7e2 with clean git status and no pending source diff.

Patch quality notes

Patch quality notes: The fallback wording in the diff is intentional and narrow: it names a degraded manager that only activates for errors classified as missing node:sqlite; other builtin manager setup failures still surface unavailable metadata instead of being hidden.
Default returns: Empty-result returns are limited to normal no-match/no-memory-source cases inside the degraded fallback search. They do not swallow SQLite initialization failures outside the explicit missing-node:sqlite classifier.
Filesystem boundary: Fallback corpus reads go through readMemoryFile, preserving the existing memory host boundary for workspace paths, configured extraPaths, regular-file checks, line limits, and character limits.
Scope boundary: This PR only changes extensions/memory-core/src/memory/search-manager.ts and its focused regression test. It does not alter provider routing, embedding calls, QMD, gateway transport, public tool schemas, public config schema, storage format, migrations, or runtime service APIs.
Architecture / source-of-truth check: SQLite/vector indexing remains the source-of-truth search path when available. The keyword manager is a compatibility fallback for a known runtime capability gap and delegates allowed corpus access to existing memory host helpers.
Out-of-scope boundary: Semantic/vector ranking parity, session transcript search, new public config, new schema fields, SDK export changes, migrations, provider/embedding behavior, QMD behavior, gateway/UI behavior, and live service APIs are intentionally out of scope.

Merge risk

Risk labels considered: merge-risk: compatibility, merge-risk: availability, and the local memory-file/source boundary.
Risk explanation: The compatibility risk is the intentional response-shape change for the explicit missing-node:sqlite path: callers now see provider: "keyword-fallback" and markdown result rows instead of unavailable-manager metadata. The availability risk is that fallback searches scan configured markdown memory files on each search. The boundary risk is local memory-file access and memory result sourcing.
Why acceptable: The changed behavior is limited to the explicit missing-node:sqlite runtime capability gap; runtimes with SQLite keep the existing indexed path. Fallback reads use readMemoryFile, results are source-scoped to memory files, bounded by maxResults, snippet-truncated, and marked with fallback debug/status metadata so callers can distinguish degraded keyword results from vector-backed search.

Root Cause

Root cause: The builtin memory search manager treated the missing node:sqlite runtime capability as a generic terminal setup failure, even though the invariant for durable memory is that readable markdown memory should remain available through the memory host file boundary when the indexed backend is unavailable for a known runtime capability gap.
Why this is root-cause fix: The manager-resolution seam now classifies the explicit missing-node:sqlite failure at its source and returns a degraded keyword fallback manager for that runtime capability gap, while leaving unrelated manager setup failures as unavailable errors. The fallback corpus is read through existing memory-file helpers and reports degraded mode in runtime debug/status metadata, so the fix changes the source manager selection behavior rather than masking one downstream call site.
Maintainer-ready confidence: Medium-high after the current-head no-node:sqlite CLI proof. The remaining decision is policy acceptance of degraded keyword fallback semantics and its compatibility/availability tradeoff.
Source-of-truth boundary: SQLite/vector search remains canonical whenever node:sqlite is available. The fallback is not a new canonical ranking engine; it is a bounded compatibility path for markdown memory recall when the builtin SQLite dependency cannot load.
Compatibility / default behavior: Runtimes where node:sqlite is available keep the existing SQLite/vector search behavior. Only the explicit missing-node:sqlite path changes response shape from unavailable metadata to keyword-fallback status and markdown result rows.
Availability tradeoff: The fallback favors bounded local availability over semantic/vector quality and indexed performance when SQLite is absent; maintainers should explicitly accept that degraded behavior before merge.
Fix classification: Root cause fix
Out-of-scope boundary: Semantic/vector ranking parity, new public config, new schema fields, runtime service API changes, SDK export changes, migrations, provider/embedding behavior, QMD behavior, and session transcript search.

clawsweeper · 2026-06-15T10:38:52Z

Thanks for the contribution. ClawSweeper proposes closing this for now: the implementation may be reasonable, but passing review and proof does not establish that OpenClaw should add this product surface.

Close: the patch is technically coherent and has useful CLI proof, but it adds new degraded keyword-fallback semantics for the missing-node:sqlite path without maintainer-confirmed product direction. The canonical issue should remain the place for maintainers to choose degraded recall, scan bounds, or remediation-only behavior.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #93150
Summary: This PR is the candidate implementation for the open feature request to add keyword fallback when node:sqlite is unavailable; guidance-only and FTS5 items overlap but do not replace it.

Members:

canonical: [Feature]: Add keyword fallback for memory_search when node:sqlite is unavailable #93150 - The PR body uses closing linkage for the same degraded keyword fallback behavior requested there.
partial_overlap: fix(memory): improve node:sqlite unavailable guidance #95916 - That merged PR improves missing-node:sqlite unavailable guidance but intentionally keeps unavailable-result semantics.
superseded: fix(memory): improve error message when node:sqlite is unavailable #69199 - That unmerged source PR was replaced by the merged guidance-only PR and does not implement fallback search rows.
superseded: [Feature]: Add file-based fallback search when node:sqlite is unavailable #59451 - The canonical issue identifies it as the earlier stale-closed tracker for the same file-based fallback request.
adjacent_distinct: node:sqlite missing FTS5 module — memory search keyword fallback broken #62328 - That issue concerns missing FTS5 inside an available node:sqlite runtime, not the missing builtin module path.
1 more in the report.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

This is a proposal only until the separate default-off apply policy is enabled and all live maintainer-signal checks pass. A maintainer can sponsor the direction, request a narrower version, or apply clawsweeper:human-review to keep it open.

Review details

Best possible solution:

Keep the canonical issue open for maintainer product direction; sponsor a new or reopened PR only after maintainers choose degraded fallback semantics and any required scan bounds.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main routes builtin manager construction failure to unavailable metadata, and the PR adds a classifier plus fallback manager for the same missing-node:sqlite error path. The PR body also supplies after-fix CLI output with node:sqlite blocked and markdown search results returned.

Is this the best way to solve the issue?

No as a maintainer-ready product solution without sponsorship. The implementation targets the right manager-resolution seam, but the best behavior depends on a maintainer choice between degraded recall, explicit scan bounds, or unavailable-remediation only.

Security review:

Security review cleared: No security or supply-chain concern identified; the diff adds memory-core runtime and test code only and does not change dependencies, workflows, scripts, permissions, lockfiles, or secret handling.

AGENTS.md: found and applied where relevant.

What I checked:

Repository policy applied: Root AGENTS.md treats fallback behavior as compatibility-sensitive and states fallback is a product decision rather than an implementation convenience. (AGENTS.md:31, 816038e97a5d)
Scoped extension policy applied: The scoped extensions guide keeps bundled plugin production code on plugin-SDK boundaries; the PR stays inside memory-core and imports memory host SDK subpaths rather than core internals. (extensions/AGENTS.md:27, 816038e97a5d)
Current main behavior: Current main's builtin manager path loads MemoryIndexManager and converts setup failure into manager:null plus error; it does not return a keyword fallback manager. (extensions/memory-core/src/memory/search-manager.ts:444, 816038e97a5d)
Current unavailable response path: When the manager context has an error, memory_search records cooldown and returns buildMemorySearchUnavailableResult, so current main returns unavailable metadata rather than fallback search rows. (extensions/memory-core/src/tools.ts:468, 816038e97a5d)
Current missing-node:sqlite guidance: Current main already has targeted missing-node:sqlite unavailable warning/action text, confirming the PR is adding a new fallback behavior rather than only fixing the message. (extensions/memory-core/src/tools.shared.ts:118, 816038e97a5d)
SQLite dependency seam: The memory host SDK throws a specific SQLite support error when node:sqlite cannot be required, which is the failure text the PR classifies. (packages/memory-host-sdk/src/host/sqlite.ts:17, 816038e97a5d)

Likely related people:

vincentkoc: Authored the merged replacement PR that improved node:sqlite unavailable guidance, and local source history points to Vincent Koc on the grafted memory manager snapshot. (role: recent memory guidance contributor; confidence: medium; commits: 19627c7dd9b6, 51e0997c2bbd; files: extensions/memory-core/src/tools.shared.ts, extensions/memory-core/src/tools.test.ts, extensions/memory-core/src/memory/search-manager.ts)
mushuiyu886: Authored merged PR fix #92218: memory_search tool disabled with QMD backend #92618 in the same search-manager surface, so they are relevant beyond opening this PR. (role: recent adjacent contributor; confidence: medium; commits: 3a1a5c0dac68; files: extensions/memory-core/src/memory/search-manager.ts, extensions/memory-core/src/memory/search-manager.test.ts)
rrrrrredy: Opened the canonical fallback issue and authored the earlier unavailable-message source PR that was replaced by the merged guidance PR. (role: fallback issue author and guidance contributor; confidence: medium; commits: c4d709234fb2, 10e3b0db1d43; files: extensions/memory-core/src/tools.shared.ts, extensions/memory-core/src/tools.test.ts)

Codex review notes: model internal, reasoning high; reviewed against 816038e97a5d.

mushuiyu886 · 2026-06-16T09:05:42Z

@clawsweeper re-review

clawsweeper · 2026-06-16T09:05:45Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

mushuiyu886 · 2026-06-16T09:34:06Z

@clawsweeper re-review

mushuiyu886 · 2026-06-16T14:41:52Z

@clawsweeper re-review

mushuiyu886 · 2026-06-18T11:51:52Z

@clawsweeper re-review

rrrrrredy · 2026-06-22T02:10:33Z

From the adjacent #93150 / #69199 triage, the remaining question here looks like a maintainer product decision rather than a concrete contributor-side code defect.

The implementation changes the explicit missing-node:sqlite path from "memory search unavailable with remediation metadata" to "degraded keyword fallback over configured markdown memory files". That seems like the intended fix for #93150, while #69199 remains the narrower remediation-message fallback if maintainers prefer not to return degraded search rows.

The decision I think maintainers need to make is:

accept degraded keyword fallback semantics for missing node:sqlite and merge this PR as the closing implementation for [Feature]: Add keyword fallback for memory_search when node:sqlite is unavailable #93150;
ask for explicit scan bounds first if per-search markdown scanning is an availability concern for large configured memory paths; or
reject degraded recall and keep the behavior to unavailable metadata/remediation only.

I do not see a separate automated repair to queue from the current ClawSweeper review; it is waiting on that behavior/availability tradeoff.

fix(memory): fall back to keyword search without sqlite

3eb2088

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: M proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 15, 2026

clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 15, 2026

fix(memory): reuse safe reads for keyword fallback

f6ef73b

fix(memory): address fallback lint diagnostics

96d9ef3

clawsweeper Bot removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 16, 2026

clawsweeper Bot mentioned this pull request Jun 17, 2026

fix(memory): improve error message when node:sqlite is unavailable #69199

Closed

This was referenced Jun 19, 2026

doctor(memory): surface FTS5 unavailable state in doctor checks #62338

Closed

[Feature]: Add keyword fallback for memory_search when node:sqlite is unavailable #93150

Open

clawsweeper Bot mentioned this pull request Jun 25, 2026

node:sqlite missing FTS5 module — memory search keyword fallback broken #62328

Open

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 6, 2026

mushuiyu886 closed this Jul 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory): fall back to keyword search without sqlite#93260

fix(memory): fall back to keyword search without sqlite#93260
mushuiyu886 wants to merge 3 commits into
openclaw:mainfrom
mushuiyu886:feat/issue-93150

mushuiyu886 commented Jun 15, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

mushuiyu886 commented Jun 16, 2026

Uh oh!

clawsweeper Bot commented Jun 16, 2026 •

edited

Loading

Uh oh!

mushuiyu886 commented Jun 16, 2026

Uh oh!

mushuiyu886 commented Jun 16, 2026

Uh oh!

mushuiyu886 commented Jun 18, 2026

Uh oh!

rrrrrredy commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mushuiyu886 commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Real behavior proof

Review findings addressed

Regression Test Plan

Patch quality notes

Merge risk

Root Cause

Uh oh!

clawsweeper Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mushuiyu886 commented Jun 16, 2026

Uh oh!

clawsweeper Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mushuiyu886 commented Jun 16, 2026

Uh oh!

mushuiyu886 commented Jun 16, 2026

Uh oh!

mushuiyu886 commented Jun 18, 2026

Uh oh!

rrrrrredy commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mushuiyu886 commented Jun 15, 2026 •

edited

Loading

clawsweeper Bot commented Jun 15, 2026 •

edited

Loading

clawsweeper Bot commented Jun 16, 2026 •

edited

Loading