Problem

The memorySearch hybrid mode uses SQLite FTS5 with the default "simple" tokenizer, which does not tokenize CJK (Japanese, Korean, Chinese) text. This makes the BM25 component completely ineffective for CJK users.

How it fails

Japanese example:

1. Document: "今日は成語しりとりをした" (Today we played word chain)
2. FTS5 simple tokenizer indexes it as one single token
3. Query: "しりとり" (word chain) → no match

Korean example:

1. Document: "오늘 끝말잇기 게임을 했다" (Today we played word chain)
2. Query: "끝말잇기" → no match (entire Hangul run is one token)

Chinese example:

1. Document: "今天玩成语接龙游戏" (Today we played idiom chain)
2. Query: "成语接龙" → no match

The regex in buildFtsQuery() (/[\p{L}\p{N}_]+/gu) treats consecutive CJK characters as a single token, and FTS5 simple tokenizer does the same during indexing — but since the indexed token spans differ from query tokens, matching fails.

Reproduction

With any CJK content in memory/*.md files, memory_search returns empty results for CJK keywords even when the exact text exists.

Tested queries that return empty despite existing in files:

• Japanese: "しりとり" — exists in file, empty results
• Chinese: "成语接龙" — 3 occurrences in file, empty results
• Short CJK phrases in general fail to match against longer indexed runs

Root cause

• File: src/memory/memory-schema.ts — FTS5 table created without specifying tokenizer (defaults to "simple")
• File: src/memory/hybrid.ts — buildFtsQuery() regex treats CJK runs as single tokens

Suggested fixes (by complexity)

1. Quick fix: Modify buildFtsQuery() to split CJK characters individually for queries ("成語接龍" → "成" AND "語" AND "接" AND "龍")
2. Better: Add a ftsTokenizer config option supporting "icu" or "trigram" tokenizers
3. Best: Integrate a proper CJK tokenizer (e.g., MeCab for Japanese, jieba for Chinese)

Environment

• OpenClaw 2026.2.17
• macOS (Apple Silicon M4)
• CJK content in memory/*.md

Impact

This affects all CJK users (Japanese, Korean, Chinese). The hybrid mode effectively degrades to vector-only search for these languages. If the embedding model also struggles with short CJK queries, memory search becomes nearly unusable for CJK content.

Any CJK-speaking community members experiencing the same issue? Would love to hear your workarounds.