fix(memory-core): score CJK keyword search instead of textScore=0 with trigram FTS5 by draix · Pull Request #92164 · openclaw/openclaw

draix · 2026-06-11T12:21:44Z

Summary

Hybrid memory search scored every Chinese/Japanese/Korean (CJK) query at textScore: 0 when the FTS5 store uses the trigram tokenizer. Vector search still returned correct matches, but the BM25/full-text component contributed nothing, dragging down the final hybrid score for all CJK queries. English queries were unaffected.

This fixes the two root causes pinpointed in #92061, both in extensions/memory-core (planKeywordSearch / searchKeyword).

Fixes #92061

Root cause

planKeywordSearch tokenizes the query with /[\p{L}\p{N}_]+/gu, which collapses a run of CJK characters into a single token (e.g. 飞书插件配置 → one 6-char token). The previous code only diverted CJK to the LIKE path when the token was shorter than 3 characters:

if (SHORT_CJK_TRIGRAM_RE.test(token) && Array.from(token).length < 3) {
  substringTerms.push(token);   // LIKE path
  continue;
}
matchTerms.push(token);          // MATCH path

So any CJK token of 3+ characters became a quoted trigram phrase, e.g. MATCH "飞书插件配置". A quoted phrase only matches the exact contiguous substring, which rarely exists verbatim in documents — so MATCH returned 0 rows.

The LIKE fallback that could have rescued this only fired inside the catch block, i.e. when MATCH threw. An empty-but-successful MATCH skipped it entirely, leaving the keyword result set empty → textScore = 0 for the whole hybrid result.

I verified the mechanism directly against the real node:sqlite FTS5 trigram engine (no mocks):

phrase 飞书插件配置 (not contiguous in doc): OK rows=0   ← silent empty, no throw
phrase feishu (>=3 latin):                  OK rows=1

Fix

Bug 1 — split CJK tokens for substring matching. Any token containing CJK is decomposed into per-character LIKE terms (splitCjkToken). Contiguous non-CJK runs are kept whole, so API配置 → ["API", "配", "置"] rather than letter-by-letter, preserving precision for embedded ASCII.
Bug 2 — fall back to LIKE on an empty MATCH. When MATCH succeeds but returns 0 rows and the plan has substring terms, retry with the per-term LIKE search instead of returning empty. The exception-path fallback is refactored into a shared runLikeFallback helper so both paths use the same correctly-decomposed term set (this also drops the old over-strict whole-CJK-token LIKE term from the exception path).

CJK keyword hits now surface through the LIKE path with textScore = 1 (no BM25 rank available), exactly like the pre-existing short-CJK and MATCH-exception fallbacks.

Before / After

Query (`trigram` tokenizer)	Doc	Before	After
`飞书插件配置`	`如何配置飞书插件`	∅ → `textScore 0`	hit → `textScore 1`
`東京駅周辺`	`東京の駅の周辺を歩く`	∅ → `textScore 0`	hit → `textScore 1`
`id 配置` (sub-trigram latin + CJK)	`id配置文档说明`	∅ (empty MATCH, no fallback)	hit → `textScore 1`
`API飞书`	`查看API文档和飞书设置`	∅ → `textScore 0`	hit → `textScore 1`
`feishu 配置` (latin ≥3 + CJK)	`feishu 配置文档说明`	BM25 hit	BM25 hit (unchanged)

Minimal reproduction

agents.defaults.memorySearch.store.fts.tokenizer: "trigram"

Index a memory file containing 如何配置飞书插件.
memory_search with query 飞书插件配置.
Before: all results textScore: 0. After: keyword component contributes textScore: 1.

Tests

Added a CJK FTS textScore regression coverage (issue #92061) block to manager-search.test.ts (runs under the existing itWithTrigramFts guard, skipped where trigram FTS5 is unavailable):

multi-character CJK query scores textScore = 1 instead of 0 (core repro)
every CJK character of the query must be present (per-character AND semantics)
Japanese kanji query that is not a contiguous substring
empty-MATCH → LIKE fallback (sub-trigram latin id + CJK), isolating Bug 2
non-CJK runs kept whole when splitting a mixed token (API飞书)
mixed latin+CJK still BM25-scored through MATCH (no regression)

The first five fail against main (empty result set / textScore 0) and pass with the fix; I confirmed both directions by replaying the exact regex/SQL logic against node:sqlite FTS5 trigram.

Architecture note

memory-core runs lexical (FTS5) and vector (sqlite-vec) search independently and blends them in the hybrid scorer. The trigram tokenizer is the recommended option for CJK precisely because unicode61 word-segmentation does not apply to scriptio-continua languages — but trigram needs ≥3 characters to form a token and a quoted phrase demands contiguity. The robust lexical primitive for CJK is therefore per-character substring (LIKE) matching, which this change makes the consistent path for all CJK tokens while keeping BM25-ranked MATCH for trigram-eligible Latin/numeric terms. The empty-MATCH fallback closes the remaining gap for mixed queries so the keyword channel never silently collapses to zero.

Real behavior proof

This change is in a path that requires a live OpenClaw instance with an embedding provider (the issue used Ollama qwen3-embedding:4b) plus a populated CJK memory index to exercise end-to-end. That hardware/provider setup is not available in my environment, so I proved the actual failing mechanism at the layer where the bug lives — the real SQLite FTS5 trigram engine that memory-core drives:

$ node (real node:sqlite, FTS5 trigram)
doc indexed: "如何配置飞书插件"
MATCH "飞书插件配置"  → rows=0   (silent empty: reproduces textScore=0)
per-char LIKE %飞%..%置% → rows=1 (the fix's path)

I also replayed planKeywordSearch + searchKeyword's exact SQL/regex against node:sqlite for all six test scenarios: the five regression cases return empty under main and the correct hit under the patch.

Since a full hybrid memory_search live run with an embedding provider is not reproducible here, I'm requesting maintainers apply proof: override for the live-instance portion of the proof gate. Happy to run any additional maintainer-specified validation. The branch is pushed to a maintainer-pushable branch with edits enabled.

🤖 Generated with Claude Code

…igram FTS5) Multi-character CJK query tokens were emitted as a single quoted trigram MATCH phrase that only matches an exact contiguous substring, so MATCH returned zero rows and (because the LIKE fallback only fired on a thrown MATCH) the keyword channel collapsed to textScore=0 for every CJK query. - planKeywordSearch: decompose any CJK-bearing token into per-character LIKE substring terms (keeping contiguous non-CJK runs whole), not just tokens shorter than 3 characters. - searchKeyword: fall back to per-term LIKE when MATCH succeeds but returns zero rows, via a shared runLikeFallback helper reused by the exception path so both use the correctly decomposed term set. Adds CJK FTS regression tests (per-character AND semantics, empty-MATCH fallback, mixed ASCII+CJK token splitting); the failing cases reproduce textScore=0 against main. Fixes openclaw#92061 Co-Authored-By: Claude Opus 4.8 <[email protected]>

clawsweeper · 2026-06-11T12:23:55Z

Codex review: needs real behavior proof before merge. Reviewed June 15, 2026, 12:30 PM ET / 16:30 UTC.

Summary
The PR changes memory-core keyword planning/search so CJK-bearing trigram queries use decomposed LIKE fallback terms, and adds CJK textScore regression tests.

PR surface: Source +50, Tests +97. Total +147 across 2 files.

Reproducibility: yes. at source level: current main sends long CJK trigram tokens through MATCH and only falls back to LIKE on MATCH exceptions. I did not run a live OpenClaw memory_search instance with an embedding provider in this read-only review.

Review metrics: 1 noteworthy metric.

CJK LIKE fallback scope: 1 tokenizer branch broadened. The PR moves all CJK-bearing trigram tokens, not only short CJK tokens, onto the LIKE-based path maintainers need to evaluate for latency and relevance.

Stored data model
Persistent data-model change detected: unknown-data-model-change: extensions/memory-core/src/memory/manager-search.test.ts, unknown-data-model-change: extensions/memory-core/src/memory/manager-search.ts, vector/embedding metadata: extensions/memory-core/src/memory/manager-search.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted after-fix OpenClaw memory_search output or get an explicit maintainer proof override.
Provide latency evidence or a bounded fallback for a non-trivial CJK trigram memory index.
Rebase the dirty branch and refresh review against the current main behavior.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The PR has useful node:sqlite terminal proof, but it explicitly lacks after-fix OpenClaw memory_search proof and no maintainer proof override is present. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] The PR explicitly lacks an after-fix OpenClaw memory_search run with a populated CJK trigram index and no maintainer proof override is present.
[P2] The wider LIKE fallback uses ESCAPE, so SQLite's trigram LIKE index optimization may not apply and large memory indexes need latency proof or a bounded fallback.
[P2] A newer related issue challenges the exact per-character CJK fallback semantics, so maintainers need to choose the accepted split/ranking strategy before merge.
[P1] Live PR metadata reports the branch dirty against the base, so the actual merge result needs a rebase or refreshed review before landing.

Maintainer options:

Prove or bound the fallback before merge (recommended)
Add maintainer-acceptable real memory_search proof plus latency evidence on a non-trivial index, or adjust the fallback so CJK queries cannot perform unbounded synchronous scans.
Accept the scan tradeoff explicitly
Maintainers may choose to merge the correctness fix as-is if restoring CJK keyword contribution is worth the possible large-index latency cost.
Pause and consolidate the CJK fix
If the CJK split/ranking strategy needs product judgment, pause this branch and consolidate it with the related open PR and issue before landing one final shape.

Next step before merge

[P2] The remaining blockers are maintainer decisions on proof override, CJK fallback semantics, latency tolerance, and branch refresh rather than a narrow automated repair.

Security
Cleared: No concrete security or supply-chain concern was found; the diff only changes parameterized memory search SQL and colocated tests.

Review details

Best possible solution:

Land one maintainer-approved CJK fallback strategy, with focused regression tests plus real memory_search and latency proof, then close the linked CJK memory-search issues after merge.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main sends long CJK trigram tokens through MATCH and only falls back to LIKE on MATCH exceptions. I did not run a live OpenClaw memory_search instance with an embedding provider in this read-only review.

Is this the best way to solve the issue?

Unclear as merge-ready: the patch targets the right memory-core owner function and has useful tests, but the best final fix still needs maintainer-approved CJK split semantics plus real and latency proof.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 865e4db1cd0a.

Label changes

Label justifications:

P2: This is a normal-priority memory-core search correctness fix with limited blast radius but user-visible CJK recall impact.
merge-risk: 🚨 availability: The diff can route more memory_search queries through synchronous LIKE fallback scans, which may stall runtime responsiveness on larger indexes.
rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR has useful node:sqlite terminal proof, but it explicitly lacks after-fix OpenClaw memory_search proof and no maintainer proof override is present. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +50, Tests +97. Total +147 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	71	21	+50
Tests	1	97	0	+97
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	168	21	+147

What I checked:

AGENTS policy read: Read root AGENTS.md and extensions/AGENTS.md; their bundled-plugin, fallback-safety, dependency-contract, and PR proof guidance applies to this memory-core review. (AGENTS.md:1, 865e4db1cd0a)
Current main CJK routing: Current main only sends CJK trigram tokens shorter than three characters to substringTerms; longer CJK strings still become MATCH terms. (extensions/memory-core/src/memory/manager-search.ts:127, 865e4db1cd0a)
Current main fallback behavior: Current main only runs the LIKE fallback inside the MATCH exception path, so a successful MATCH returning zero rows does not retry LIKE. (extensions/memory-core/src/memory/manager-search.ts:366, 865e4db1cd0a)
Runtime caller: MemoryManager.search calls searchKeyword for both FTS-only and hybrid memory_search paths, so this keyword planning behavior affects user-visible memory recall scoring. (extensions/memory-core/src/memory/manager.ts:687, 865e4db1cd0a)
PR implementation: At PR head, CJK-bearing trigram tokens are split into LIKE terms, and empty successful MATCH results retry a shared LIKE fallback. (extensions/memory-core/src/memory/manager-search.ts:150, 2511c76abf1a)
PR regression tests: The PR adds focused trigram FTS tests for multi-character Chinese, Japanese kanji, empty MATCH fallback, mixed ASCII/CJK splitting, and BM25 preservation for mixed queries. (extensions/memory-core/src/memory/manager-search.test.ts:235, 2511c76abf1a)

Likely related people:

Takhoffman: Merged the configurable FTS5 tokenizer PR that added the current trigram/CJK search path and related tests. (role: introduced CJK tokenizer behavior; confidence: high; commits: 3ce48aff660a, 01dc8fdf4a3b; files: extensions/memory-core/src/memory/manager-search.ts, extensions/memory-core/src/memory/manager-search.test.ts, src/config/types.tools.ts)
AnonAmit: Authored the memory fallback lexical ranking PR that changed manager-search.ts, manager-search.test.ts, and the adjacent manager search flow. (role: fallback ranking contributor; confidence: medium; commits: 42590106ab6e, 1c20b1416cd4; files: extensions/memory-core/src/memory/manager-search.ts, extensions/memory-core/src/memory/manager-search.test.ts, extensions/memory-core/src/memory/manager.ts)
vincentkoc: Current blame on the implicated lines points to a recent release-support commit, and Vincent also contributed to the fallback-ranking PR touching this path. (role: recent area contributor; confidence: medium; commits: 94833b2c9072, 42590106ab6e, ad183778a826; files: extensions/memory-core/src/memory/manager-search.ts, extensions/memory-core/src/memory/manager.ts)
Peter Steinberger: Moved the memory engine into and behind the memory plugin boundary that owns this runtime path. (role: memory plugin boundary refactor contributor; confidence: medium; commits: cad83db8b2f7, dbf78de7c680; files: extensions/memory-core/src/memory/manager-search.ts, extensions/memory-core/src/memory/manager.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

draix · 2026-06-11T12:31:37Z

Thanks for the review — noting the patch-quality rating and the single remaining blocker (real behavior proof).

On the proof gate: a full live memory_search run requires an OpenClaw instance with an embedding provider (the issue used Ollama qwen3-embedding:4b) plus a populated CJK index, which isn't reproducible in my environment. I proved the failing mechanism at the layer where the bug actually lives — the real node:sqlite FTS5 trigram engine memory-core drives (details in the Real behavior proof section of the description): the quoted trigram phrase MATCH "飞书插件配置" returns 0 rows without throwing (reproducing textScore=0), while the per-character LIKE path returns the hit. Requesting a maintainer proof: override for the live-instance portion per CONTRIBUTING; happy to run any maintainer-specified validation.

On the red CI lanes: the failing check-lint / check-*-types / check-additional-* / build-artifacts jobs are all the same pre-existing failure on the base commit (9a6c71a4), [plugin-sdk boundary dts] src/agents/sessions/sdk.ts(279,19): error TS2322, in a file this PR does not touch — main's latest CI run is already red on it. This PR only changes extensions/memory-core; its extension-memory test lane (incl. the new CJK regression tests) passes locally (899/899). Branch has Allow edits by maintainers enabled.

openclaw-clownfish · 2026-06-15T23:15:39Z

Thanks @draix for the CJK trigram search fix in #92164. I am closing this as superseded by #92341 because #92341 is the open canonical path for the same #92061 textScore=0 root cause, covers the empty MATCH LIKE fallback as well, and currently has the stronger proof/check state.

Clownfish will keep your source PR and attribution in the canonical review context so the implementation credit is preserved. If #92341 misses a behavior that this PR still covers, please reply and we can reopen or split that work back out.

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 11, 2026

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 11, 2026

clawsweeper Bot mentioned this pull request Jun 15, 2026

memory-core: CJK LIKE fallback returns wrong results for multi-character queries #92728

Open

openclaw-clownfish Bot closed this Jun 15, 2026

openclaw-clownfish Bot added the clownfish Tracked by Clownfish automation label Jun 15, 2026

clawsweeper Bot mentioned this pull request Jun 20, 2026

fix(memory-core): CJK textScore=0 and empty MATCH LIKE fallback #92341

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory-core): score CJK keyword search instead of textScore=0 with trigram FTS5#92164

fix(memory-core): score CJK keyword search instead of textScore=0 with trigram FTS5#92164
draix wants to merge 1 commit into
openclaw:mainfrom
draix:fix/memory-core-cjk-fts-textscore

draix commented Jun 11, 2026

Uh oh!

clawsweeper Bot commented Jun 11, 2026 •

edited

Loading

Uh oh!

draix commented Jun 11, 2026

Uh oh!

openclaw-clownfish Bot commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

draix commented Jun 11, 2026

Summary

Root cause

Fix

Before / After

Minimal reproduction

Tests

Architecture note

Real behavior proof

Uh oh!

clawsweeper Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

draix commented Jun 11, 2026

Uh oh!

openclaw-clownfish Bot commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

clawsweeper Bot commented Jun 11, 2026 •

edited

Loading