Greptile Summary

This PR adds a configurable FTS5 tokenizer (unicode61 or trigram) to enable CJK full-text search, wiring the option through config/schema, persisting it in index metadata, and triggering a full reindex on tokenizer change. The plumbing is solid, but there are two logic bugs in the query-expansion layer that would leave Chinese search broken even after enabling the trigram tokenizer.

Key issues found:

• Chinese unigrams incompatible with trigram FTS5 (src/memory/query-expansion.ts lines 699–710): In trigram mode, the Chinese CJK branch emits individual 1-character tokens. SQLite's trigram FTS5 tokenizer requires at least 3 characters per query term; shorter terms silently return no results. The Japanese kanji path correctly keeps the whole contiguous block (e.g. "経済政策"), but the Chinese path falls back to unigrams. Chinese search will therefore still return zero results in trigram mode.
• expandQueryWithLlm fallback ignores tokenizer (src/memory/query-expansion.ts line 817): The function signature has no opts parameter, so when the LLM expander fails the fallback extractKeywords(query) always uses unicode61 behaviour (producing CJK bigrams). While this function is not currently called in production code, the export makes it part of the public API and the oversight should be fixed.

Confidence Score: 2/5

• Not safe to merge as-is: the primary CJK use-case (Chinese text search) remains broken in trigram mode due to unigram query tokens that the FTS5 trigram tokenizer cannot match.
• The config plumbing, schema changes, reindex logic, and backward-compatibility are all correct. However the core goal of the PR — making Chinese text searchable — is not achieved: the Chinese tokenization path in trigram mode produces 1-character tokens that SQLite's trigram FTS5 silently ignores, so Chinese queries still return no results. A second logic bug exists in expandQueryWithLlm. The two issues are in src/memory/query-expansion.ts and must be fixed for the feature to work as intended.
• src/memory/query-expansion.ts — both the Chinese CJK tokenization branch (lines 699–710) and the expandQueryWithLlm fallback (line 817) need fixes.

_{Last reviewed commit: 4bb8da8}

Addressing the two issues flagged by Greptile (both in "Comments Outside Diff"):

1. Chinese unigrams incompatible with trigram FTS5 (lines 699–710)

Fixed in 92be66c. The Chinese tokenization path now mirrors the Japanese kanji path: in trigram mode, it pushes the whole contiguous CJK block as a single token instead of individual characters. SQLite FTS5 trigram tokenizer requires >= 3 characters per query term; single characters silently return no results.

Before: "之前讨论的那个方案" → ["之", "前", "讨", "论", ...] (no FTS5 match)
After: "之前讨论的那个方案" → ["之前讨论的那个方案"] (valid trigram matches)

2. expandQueryWithLlm fallback ignores tokenizer (line 817)

Fixed in 92be66c. Both expandQueryForFts and expandQueryWithLlm now accept and propagate the ftsTokenizer option to extractKeywords.

✍️ Author: Claude Code with @carrotRakko (AI-written, human-approved)

Real-world validation from a heavy CJK user

I've been running OpenClaw with Chinese memory for months and independently discovered the same FTS5 issue. Here's our production data:

Before fix (unicode61 tokenizer)

• Search hit rate: 0/7 for typical Chinese queries
• FTS5 tokenizes entire Chinese sentences as single tokens (e.g., 元宝老婆刚怀孕 = 1 token)
• MATCH '怀孕' returns 0 results; only prefix match '怀孕*' works partially

Workaround we deployed (3 months ago)

Since we couldn't wait for upstream, we applied a multi-layered fix:

1. Write-side: space-separated Chinese keywords in memoryFlush prompt (forces FTS5 to tokenize per-word)
2. Tags line: every memory entry starts with tags: keyword1 keyword2 keyword3
3. Rebalanced weights: vectorWeight: 0.6, textWeight: 0.4 to lean on vector search
4. MMR dedup: lambda: 0.7 to avoid redundant results

After fix

• Search hit rate: 7/7 (same 7 test queries)
• Top score: 0.46 → 0.83

Supporting this PR

The trigram tokenizer approach is cleaner than our workaround. A few suggestions:

• Consider documenting the migration path (users need to rebuild FTS index)
• Adding a cjk_auto option that detects CJK content and applies trigram automatically would be ideal
• The write-side space-separation trick could be kept as defense-in-depth even with trigram

Strong +1 for merging this. CJK users are effectively flying blind without it. 🦞

@laolin5564 Thank you for the detailed real-world validation — production data with before/after hit rates is exactly the kind of evidence that helps upstream maintainers evaluate a PR. Much appreciated.

Responding to each suggestion:

1. Migration path: The code handles this automatically. manager-sync-ops.ts detects a tokenizer mismatch between the stored index metadata and the configured tokenizer, drops the FTS table, and rebuilds it on next sync. Users just need to set store.fts.tokenizer: "trigram" in config — no manual reindex step required.

2. cjk_auto option: Interesting idea. Auto-detecting CJK content and switching tokenizer would be convenient, but it adds complexity around mixed-content thresholds and per-entry vs per-index decisions (FTS5 tokenizer is index-wide, not per-row). An explicit config is the right starting point; auto-detection could be a future enhancement.

3. Write-side space-separation as defense-in-depth: Your workaround was clever and effective under unicode61 constraints. With the trigram tokenizer, FTS5 handles sub-string matching natively, so the write-side trick becomes unnecessary. That said, the space-separated tags approach has independent value for readability — no reason to remove it if it is already in place.

Your 0/7 → 7/7 hit rate improvement is a compelling data point. Thanks for sharing it here.

✍️ Author: Claude Code with @carrotRakko (AI-written, human-approved)

Maintainer follow-up:

I pulled/rebased this locally, fixed the remaining short-CJK trigram gap, and verified the memory-search surface. I cannot update the PR branch directly because the PR head is on delight-co/openclaw-public and GitHub returns 403 Permission denied to steipete for that fork.

Commits prepared locally:

• 4d531fe78838b5a5f4daefa82f2d8273a91fe5bc fix: support short CJK trigram memory queries
• 2e0d3e4219b2da7e3f96544e2ac012eecf1a25d0 test: cover additional short CJK trigram queries

Suggested apply path from the PR branch:

git fetch https://github.com/openclaw/openclaw.git feat/fts5-tokenizer-config
git cherry-pick 4d531fe78838b5a5f4daefa82f2d8273a91fe5bc 2e0d3e4219b2da7e3f96544e2ac012eecf1a25d0

What this fixes:

• trigram-backed memory search now falls back to substring matching for 1-2 character Chinese, Japanese, and Korean queries
• mixed long trigram terms plus short CJK terms still work in one query
• config schema baseline updated for memorySearch.store.fts.tokenizer

Focused verification I ran:

pnpm test -- extensions/memory-core/src/memory/manager-search.test.ts packages/memory-host-sdk/src/host/query-expansion.test.ts src/config/schema.base.generated.test.ts
pnpm config:schema:check
pnpm config:docs:check

New/expanded regression coverage includes:

• 1-char Chinese
• 2-char Chinese
• 2-char Japanese
• 2-char Korean
• mixed long MATCH + short LIKE query

Refactor suggestion after landing:

• move trigram query planning into one shared helper that returns a query plan (matchTokens, substringTokens) so tokenizer-aware keyword extraction and tokenizer-aware search stay in one place instead of being split across query expansion and SQL search code.

Once those two commits are on this PR branch, the remaining behavior gap I found should be closed.

Closing this as implemented after Codex review.

Current main already ships this feature and the follow-up short-CJK trigram handling, so this PR is obsolete on main.

What I checked:

• Config surface exists on main: agents.defaults.memorySearch.store.fts.tokenizer is present in the typed config surface and runtime schema as "unicode61" | "trigram". (src/config/types.tools.ts:403, 7ac35b4f6900)
• Runtime schema accepts trigram: The agent-runtime zod schema includes store.fts.tokenizer with unicode61 and trigram literals. (src/config/zod-schema.agent-runtime.ts:691, 7ac35b4f6900)
• Tokenizer-aware keyword extraction landed: Current query expansion is tokenizer-aware, emits whole CJK blocks in trigram mode, and propagates the tokenizer through extractKeywords, expandQueryForFts, and expandQueryWithLlm. (packages/memory-host-sdk/src/host/query-expansion.ts:675, 7ac35b4f6900)
• Short-CJK fallback also landed: Main contains the maintainer follow-up that plans trigram searches into FTS MATCH terms plus LIKE substring fallback for 1-2 character CJK queries. (extensions/memory-core/src/memory/manager-search.ts:11, 7ac35b4f6900)
• Regression tests cover the landed behavior: There is dedicated coverage for short Chinese, Japanese, and Korean trigram fallback plus mixed long-and-short queries. (extensions/memory-core/src/memory/manager-search.test.ts:13, 7ac35b4f6900)
• Docs and release notes confirm shipment: Docs expose store.fts.tokenizer, builtin-memory docs advertise CJK trigram support, and the changelog records the shipped feature under release 2026.4.5. (CHANGELOG.md:1452, 7ac35b4f6900)

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review notes: reviewed against 7ac35b4f6900; fix evidence: release v2026.4.5.