Summary

When using memory-core with the trigram FTS5 tokenizer, Chinese (CJK) text queries always return textScore: 0 in hybrid search results. The vector component works correctly, but the BM25/full-text component contributes nothing to the final score for CJK queries.

English queries work fine with both unicode61 and trigram tokenizers.

Environment

• OpenClaw version: 2026.6.5
• SQLite version: 3.51.0 (trigram tokenizer supported)
• Embedding provider: Ollama (qwen3-embedding:4b)
• Config: agents.defaults.memorySearch.store.fts.tokenizer: "trigram"

Root Cause Analysis

Two bugs in extensions/memory-core (manager-*.js → searchKeyword / planKeywordSearch):

Bug 1: CJK tokens not split for trigram matching

In planKeywordSearch, the regex /[\p{L}\p{N}_]+/gu treats consecutive CJK characters as a single token (e.g., "飞书插件配置" → 1 token of 6 chars). The code then checks:

if (SHORT_CJK_TRIGRAM_RE.test(token) && Array.from(token).length < 3) {
    substringTerms.push(token);  // LIKE path
    continue;
}
matchTerms.push(token);  // MATCH path

Since 6 >= 3, the entire CJK string goes to matchTerms and becomes MATCH "飞书插件配置" (quoted phrase). But this exact substring rarely exists in documents, so MATCH returns 0 results.

Fix: Split CJK tokens into individual characters for LIKE matching:

if (SHORT_CJK_TRIGRAM_RE.test(token)) {
    for (const char of Array.from(token)) substringTerms.push(char);
    continue;
}

Bug 2: No LIKE fallback on empty MATCH results

The LIKE fallback only triggers when MATCH throws an exception, not when it returns 0 rows:

if (plan.matchQuery) try {
    rows = db.prepare(`... MATCH ? ...`).all(...);
    usedMatch = true;
} catch (matchErr) {
    // LIKE fallback only here
}

If MATCH "飞书插件配置" returns empty without throwing, the LIKE path never runs, and all keyword results are empty → textScore=0 for all hybrid results.

Fix: Add LIKE fallback when MATCH returns empty:

if (plan.matchQuery) try {
    rows = db.prepare(`... MATCH ? ...`).all(...);
    usedMatch = true;
    if (rows.length === 0 && plan.substringTerms.length > 0) {
        // Fall back to LIKE with all terms
        rows = db.prepare(`... WHERE text LIKE ? ...`).all(...);
        usedMatch = false;
    }
} catch (matchErr) {
    // existing LIKE fallback
}

Reproduction

1. Set agents.defaults.memorySearch.store.fts.tokenizer: "trigram"
2. Index memory files containing Chinese text
3. Run memory_search with a Chinese query
4. Observe: textScore: 0 for all results, despite vector search returning correct matches

Workaround

Vector search still works for CJK text (vectorScore is correct). The textScore=0 reduces the final hybrid score but doesn't break recall entirely. Setting textWeight: 0 in query config would eliminate the misleading BM25 component.

Suggested Fix

Both bugs are in the searchKeyword / planKeywordSearch functions. The CJK token splitting and MATCH-empty-to-LIKE fallback should be added upstream.

Additional Note

The trigram tokenizer has an inherent limitation: it requires at least 3 characters to create a searchable token. Single/double CJK characters (e.g., "飞", "配置") cannot be matched via FTS5 MATCH at all, only via LIKE. The code already partially handles this (the < 3 check), but the threshold logic is incorrect for multi-character CJK strings.