Memory: add configurable FTS5 tokenizer for CJK text support by Takhoffman · Pull Request #56707 · openclaw/openclaw

Takhoffman · 2026-03-29T01:51:46Z

Supersedes #41810 because the original PR head lives on a fork branch that is not writable from this maintainer account.

Summary

add memorySearch.store.fts.tokenizer with unicode61 and trigram options
rebuild the FTS index when the configured tokenizer changes
route short 1-2 character CJK trigram queries through substring fallback so Chinese, Japanese, and Korean search still works for short terms and mixed long-and-short queries
add regression coverage for tokenizer-aware query expansion, real sqlite trigram search behavior, and refresh the generated base config schema

Changelog

added an Unreleased ### Fixes entry in CHANGELOG.md

Verification

pnpm build
pnpm check
pnpm test -- extensions/memory-core/src/memory/manager-search.test.ts packages/memory-host-sdk/src/host/query-expansion.test.ts
pnpm test -- extensions/memory-core/src/memory/index.test.ts -t "reindexes when extraPaths change"
pnpm test -- src/config/schema.base.generated.test.ts
pnpm test -- src/media-understanding/image.test.ts
pnpm test

Notes

a full pnpm test sweep hit one long-run timeout in src/media-understanding/image.test.ts; rerunning that file in isolation passed immediately, and a second isolated run remained green. No failures remained in the touched memory/tokenizer surfaces.
issue context: closes memorySearch BM25 (FTS5) does not work for CJK text #20730

@carrotRakko

Add a `memorySearch.store.fts.tokenizer` config option that allows selecting the FTS5 tokenizer. Default remains `unicode61` (backward compatible). Setting `"trigram"` enables substring matching for CJK (Chinese, Japanese, Korean) text that unicode61 cannot tokenize. Changes: - Add `store.fts.tokenizer` to MemorySearchConfig type and Zod schema - Pass tokenizer to ensureMemoryIndexSchema for FTS table creation - Detect tokenizer changes in MemoryIndexMeta and trigger full reindex - Reset FTS table with DROP + re-CREATE on tokenizer change - Skip CJK bigram splitting in query-expansion when using trigram - Auto-apply `case_sensitive 0` when trigram is selected Known limitation: trigram does not match queries shorter than 3 characters via FTS MATCH. In hybrid mode, vector search covers these cases. ✍️ Author: Claude Code with @carrotRakko (AI-written, human-approved)

@carrotRakko

Chinese tokenization path was emitting individual characters (unigrams) in trigram mode, but SQLite FTS5 trigram tokenizer requires >= 3 chars per query term. Single characters silently return no results, breaking Chinese search entirely. Now mirrors the Japanese kanji path: pushes the whole contiguous CJK block as a single token. Also propagates ftsTokenizer option through expandQueryForFts and expandQueryWithLlm fallback paths. ✍️ Author: Claude Code with @carrotRakko (AI-written, human-approved)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 323dfe507a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-29T01:56:21Z

+    if (SHORT_CJK_TRIGRAM_RE.test(token) && Array.from(token).length < 3) {
+      substringTerms.push(token);
+      continue;


Fallback short trigram tokens regardless of script

In trigram mode, only 1–2 character CJK tokens are moved to the LIKE fallback, while 1–2 character Latin/acronym tokens stay in the MATCH query. SQLite FTS5 trigram cannot satisfy MATCH terms shorter than 3 characters, so any such token makes the AND query unsatisfiable and drops otherwise valid hits (for example queries like AI tips or mixed queries that include a short acronym). This is a regression introduced by the new planner because short non-CJK terms now block matching in trigram-configured indexes.

Useful? React with 👍 / 👎.

greptile-apps · 2026-03-29T02:07:28Z

Greptile Summary

This PR adds a configurable FTS5 tokenizer (unicode61 / trigram) for CJK text support in memory search. When trigram mode is selected, the FTS5 table is rebuilt with SQLite's trigram tokenizer configuration, extractKeywords emits whole CJK blocks rather than individual bigrams/unigrams, and a short-term (<3 codepoints) CJK substring fallback using LIKE is layered on top of FTS5 MATCH so 2-character Chinese/Japanese/Korean queries work reliably. The reindex detection correctly compares the stored meta tokenizer against the configured value and triggers a full DROP + recreate cycle when they differ.

Key changes:

memory-schema.ts: appends the trigram tokenize clause to the FTS5 CREATE TABLE statement for trigram mode
manager-sync-ops.ts: stores ftsTokenizer in MemoryIndexMeta; replaces DELETE FROM fts_table with DROP TABLE IF EXISTS + ensureSchema() so tokenizer changes take effect on reindex
manager-search.ts: new planKeywordSearch function classifies query tokens into FTS5 MATCH terms and LIKE substring terms; builds a combined WHERE clause at query time
query-expansion.ts: tokenize(), extractKeywords(), expandQueryForFts(), and expandQueryWithLlm() now all accept an ftsTokenizer option; trigram mode skips CJK bigram splitting in favour of emitting the whole block
Config/schema: new memorySearch.store.fts.tokenizer field plumbed through Zod schema, generated JSON schema, and resolved config types

Two P2 edge cases to be aware of (neither blocks merge):

In trigram mode, non-CJK tokens shorter than 3 characters (e.g. \"ok\", \"vs\") land in matchTerms inside planKeywordSearch, but FTS5 trigram silently returns 0 rows for sub-3-character query terms. The FTS-only path is unaffected (filtered by isValidKeyword), but the hybrid path passes the raw query through without this guard.
In the hybrid path, buildFtsQuery is bypassed entirely in trigram mode; English stop words (e.g. \"the\", \"was\") are included verbatim in the MATCH clause. Vector search compensates in hybrid mode, but FTS precision for English queries can degrade slightly when trigram mode is active.

Confidence Score: 5/5

Safe to merge — no blocking correctness issues; all remaining findings are P2 edge cases with clear mitigating factors.

The tokenizer change, reindex trigger, schema rebuild, and short-CJK LIKE fallback are all correctly implemented and well-tested. The two P2 findings (sub-3-char non-CJK tokens silently producing empty MATCH results, and stop words bypassing buildFtsQuery in hybrid mode) are narrow edge cases that vector search compensates for in hybrid mode, and are already guarded in FTS-only mode by isValidKeyword. No data loss, incorrect indexing, or broken primary paths were found.

extensions/memory-core/src/memory/manager-search.ts — planKeywordSearch edge cases for short non-CJK tokens and stop-word bypass in trigram/hybrid mode.

Important Files Changed

Filename	Overview
extensions/memory-core/src/memory/manager-search.ts	Adds planKeywordSearch that routes short CJK tokens to LIKE fallback and long tokens to FTS5 MATCH in trigram mode. Two P2 edge cases: short non-CJK tokens (< 3 chars) silently match 0 rows in FTS5 trigram, and stop words bypass buildFtsQuery in the hybrid path.
extensions/memory-core/src/memory/manager-sync-ops.ts	Adds ftsTokenizer to MemoryIndexMeta and triggers a full reindex when the tokenizer changes. resetIndex now uses DROP TABLE + ensureSchema() instead of DELETE to allow the FTS5 table to be recreated with the new tokenizer config. Logic is correct.
packages/memory-host-sdk/src/host/memory-schema.ts	Appends trigram tokenize clause to the FTS5 CREATE TABLE statement for trigram mode. SQL is syntactically correct; the CREATE VIRTUAL TABLE IF NOT EXISTS guard is handled by the surrounding resetIndex/ensureSchema flow.
packages/memory-host-sdk/src/host/query-expansion.ts	In trigram mode, tokenize() emits whole contiguous CJK blocks instead of unigrams+bigrams. extractKeywords, expandQueryForFts, and expandQueryWithLlm all accept an ftsTokenizer opts parameter. Logic is sound and well-tested.
src/agents/memory-search.ts	Adds store.fts.tokenizer field (default unicode61) to ResolvedMemorySearchConfig and propagates it through mergeConfig correctly.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: extensions/memory-core/src/memory/manager-search.ts
Line: 60-66

Comment:
**Short non-CJK tokens silently match 0 rows in FTS5 trigram mode**

Any token that is NOT CJK but is shorter than 3 characters (e.g. `"go"`, `"ok"`, `"vs"`, `"US"`) falls through the `else` branch and is added to `matchTerms`. SQLite's FTS5 trigram tokenizer explicitly states that query terms shorter than 3 characters match no rows. So `MATCH '"go"'` in trigram mode silently returns an empty result set.

In the FTS-only path this is harmless because `isValidKeyword` already filters pure-ASCII tokens shorter than 3 chars before they reach `searchKeyword`. But in the **hybrid path** the full cleaned query is passed directly to `searchKeyword` without prior filtering, so 2-char abbreviations like `"go"` or `"ok"` reach `planKeywordSearch` and produce a `matchQuery` that yields zero FTS rows.

A simple fix is to extend the same length guard to non-CJK tokens in trigram mode:

```typescript
  for (const token of tokens) {
    if (SHORT_CJK_TRIGRAM_RE.test(token) && Array.from(token).length < 3) {
      substringTerms.push(token);
      continue;
    }
    // In trigram mode, FTS5 requires ≥3 chars per term; skip short non-CJK tokens
    if (params.ftsTokenizer === "trigram" && Array.from(token).length < 3) {
      continue;
    }
    matchTerms.push(token);
  }
```

Because vector search still handles these in hybrid mode the practical impact is limited, but the current code produces a silently-empty MATCH clause for any 2-char non-CJK token.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: extensions/memory-core/src/memory/manager-search.ts
Line: 42-47

Comment:
**English stop words bypass filtering in trigram mode's hybrid path**

In the non-trigram branch, `buildFtsQuery` is called — which performs stop word removal, query expansion, and sanitisation. In the trigram branch `buildFtsQuery` is never called; instead, all tokens extracted by `FTS_QUERY_TOKEN_RE` are put directly into `matchTerms`.

This means that for a hybrid-path query like `"what was that thing we discussed about the API"`, trigram mode produces a MATCH clause of `"what" AND "was" AND "that" AND "thing" AND "we" AND "discussed" AND "about" AND "the" AND "API"`. Because FTS5 in trigram mode scores by trigram overlap, stop words like `"the"` (exactly 3 characters, forming one trigram) will be present in almost every document, inflating the MATCH clause without improving precision.

In the FTS-only path this is fine because `extractKeywords` removes stop words before handing terms to `searchKeyword`. In the hybrid path the full cleaned query is passed through and stop words are never stripped.

Consider either (a) calling `buildFtsQuery` on the full query first and then layering the short-CJK LIKE logic on top of the resulting token set, or (b) running the same stop-word filter used by `extractKeywords` inside `planKeywordSearch` when `ftsTokenizer === "trigram"`.

This is mitigated by vector search providing the primary ranking in hybrid mode, but it can degrade FTS recall/precision for English-heavy queries when trigram mode is active.

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "Memory: update changelog and schema for ..." | Re-trigger Greptile}

greptile-apps · 2026-03-29T02:07:32Z

+  for (const token of tokens) {
+    if (SHORT_CJK_TRIGRAM_RE.test(token) && Array.from(token).length < 3) {
+      substringTerms.push(token);
+      continue;
+    }
+    matchTerms.push(token);
+  }


Short non-CJK tokens silently match 0 rows in FTS5 trigram mode

Any token that is NOT CJK but is shorter than 3 characters (e.g. "go", "ok", "vs", "US") falls through the else branch and is added to matchTerms. SQLite's FTS5 trigram tokenizer explicitly states that query terms shorter than 3 characters match no rows. So MATCH '"go"' in trigram mode silently returns an empty result set.

In the FTS-only path this is harmless because isValidKeyword already filters pure-ASCII tokens shorter than 3 chars before they reach searchKeyword. But in the hybrid path the full cleaned query is passed directly to searchKeyword without prior filtering, so 2-char abbreviations like "go" or "ok" reach planKeywordSearch and produce a matchQuery that yields zero FTS rows.

A simple fix is to extend the same length guard to non-CJK tokens in trigram mode:

for (const token of tokens) { if (SHORT_CJK_TRIGRAM_RE.test(token) && Array.from(token).length < 3) { substringTerms.push(token); continue; } // In trigram mode, FTS5 requires ≥3 chars per term; skip short non-CJK tokens if (params.ftsTokenizer === "trigram" && Array.from(token).length < 3) { continue; } matchTerms.push(token); }

Because vector search still handles these in hybrid mode the practical impact is limited, but the current code produces a silently-empty MATCH clause for any 2-char non-CJK token.

Prompt To Fix With AI

This is a comment left during a code review. Path: extensions/memory-core/src/memory/manager-search.ts Line: 60-66 Comment: **Short non-CJK tokens silently match 0 rows in FTS5 trigram mode** Any token that is NOT CJK but is shorter than 3 characters (e.g. `"go"`, `"ok"`, `"vs"`, `"US"`) falls through the `else` branch and is added to `matchTerms`. SQLite's FTS5 trigram tokenizer explicitly states that query terms shorter than 3 characters match no rows. So `MATCH '"go"'` in trigram mode silently returns an empty result set. In the FTS-only path this is harmless because `isValidKeyword` already filters pure-ASCII tokens shorter than 3 chars before they reach `searchKeyword`. But in the **hybrid path** the full cleaned query is passed directly to `searchKeyword` without prior filtering, so 2-char abbreviations like `"go"` or `"ok"` reach `planKeywordSearch` and produce a `matchQuery` that yields zero FTS rows. A simple fix is to extend the same length guard to non-CJK tokens in trigram mode: ```typescript for (const token of tokens) { if (SHORT_CJK_TRIGRAM_RE.test(token) && Array.from(token).length < 3) { substringTerms.push(token); continue; } // In trigram mode, FTS5 requires ≥3 chars per term; skip short non-CJK tokens if (params.ftsTokenizer === "trigram" && Array.from(token).length < 3) { continue; } matchTerms.push(token); } ``` Because vector search still handles these in hybrid mode the practical impact is limited, but the current code produces a silently-empty MATCH clause for any 2-char non-CJK token. How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2026-03-29T02:07:33Z

+  if (params.ftsTokenizer !== "trigram") {
+    return {
+      matchQuery: params.buildFtsQuery(params.query),
+      substringTerms: [],
+    };
+  }


English stop words bypass filtering in trigram mode's hybrid path

In the non-trigram branch, buildFtsQuery is called — which performs stop word removal, query expansion, and sanitisation. In the trigram branch buildFtsQuery is never called; instead, all tokens extracted by FTS_QUERY_TOKEN_RE are put directly into matchTerms.

This means that for a hybrid-path query like "what was that thing we discussed about the API", trigram mode produces a MATCH clause of "what" AND "was" AND "that" AND "thing" AND "we" AND "discussed" AND "about" AND "the" AND "API". Because FTS5 in trigram mode scores by trigram overlap, stop words like "the" (exactly 3 characters, forming one trigram) will be present in almost every document, inflating the MATCH clause without improving precision.

In the FTS-only path this is fine because extractKeywords removes stop words before handing terms to searchKeyword. In the hybrid path the full cleaned query is passed through and stop words are never stripped.

Consider either (a) calling buildFtsQuery on the full query first and then layering the short-CJK LIKE logic on top of the resulting token set, or (b) running the same stop-word filter used by extractKeywords inside planKeywordSearch when ftsTokenizer === "trigram".

This is mitigated by vector search providing the primary ranking in hybrid mode, but it can degrade FTS recall/precision for English-heavy queries when trigram mode is active.

Prompt To Fix With AI

This is a comment left during a code review. Path: extensions/memory-core/src/memory/manager-search.ts Line: 42-47 Comment: **English stop words bypass filtering in trigram mode's hybrid path** In the non-trigram branch, `buildFtsQuery` is called — which performs stop word removal, query expansion, and sanitisation. In the trigram branch `buildFtsQuery` is never called; instead, all tokens extracted by `FTS_QUERY_TOKEN_RE` are put directly into `matchTerms`. This means that for a hybrid-path query like `"what was that thing we discussed about the API"`, trigram mode produces a MATCH clause of `"what" AND "was" AND "that" AND "thing" AND "we" AND "discussed" AND "about" AND "the" AND "API"`. Because FTS5 in trigram mode scores by trigram overlap, stop words like `"the"` (exactly 3 characters, forming one trigram) will be present in almost every document, inflating the MATCH clause without improving precision. In the FTS-only path this is fine because `extractKeywords` removes stop words before handing terms to `searchKeyword`. In the hybrid path the full cleaned query is passed through and stop words are never stripped. Consider either (a) calling `buildFtsQuery` on the full query first and then layering the short-CJK LIKE logic on top of the resulting token set, or (b) running the same stop-word filter used by `extractKeywords` inside `planKeywordSearch` when `ftsTokenizer === "trigram"`. This is mitigated by vector search providing the primary ranking in hybrid mode, but it can degrade FTS recall/precision for English-heavy queries when trigram mode is active. How can I resolve this? If you propose a fix, please make it concise.

…w#56707) Verified: - pnpm build - pnpm check - pnpm test -- extensions/memory-core/src/memory/manager-search.test.ts packages/memory-host-sdk/src/host/query-expansion.test.ts - pnpm test -- extensions/memory-core/src/memory/index.test.ts -t "reindexes when extraPaths change" - pnpm test -- src/config/schema.base.generated.test.ts - pnpm test -- src/media-understanding/image.test.ts - pnpm test Co-authored-by: Mitsuyuki Osabe <[email protected]>

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core agents Agent runtime and tooling size: M maintainer Maintainer-authored PR labels Mar 29, 2026

Mitsuyuki Osabe and others added 4 commits March 28, 2026 20:52

Memory: support short CJK trigram queries

01dc8fd

Memory: update changelog and schema for trigram tokenizer

f3d4200

Takhoffman force-pushed the codex/pr-41810-short-cjk-fallback branch from 323dfe5 to f3d4200 Compare March 29, 2026 01:53

Takhoffman merged commit 3ce48af into main Mar 29, 2026
10 checks passed

Takhoffman deleted the codex/pr-41810-short-cjk-fallback branch March 29, 2026 01:53

chatgpt-codex-connector Bot reviewed Mar 29, 2026

View reviewed changes

Takhoffman mentioned this pull request Mar 29, 2026

test: refresh bundled plugin metadata #56712

Closed

greptile-apps Bot reviewed Mar 29, 2026

View reviewed changes

clawsweeper Bot mentioned this pull request May 1, 2026

[codex] add memory text-search fallback #65435

Closed

This was referenced May 15, 2026

[security fix] Fix SQL injection in memory schema management #65452

Closed

fix(memory): remove provider model filter from FTS keyword search #48328

Closed

Bug: memory_search hybrid mode not returning FTS matches #48300

Closed

This was referenced Jun 11, 2026

memory-core: CJK text FTS5 search returns textScore=0 with trigram tokenizer #92061

Closed

memory-core: CJK LIKE fallback returns wrong results for multi-character queries #92728

Open

clawsweeper Bot mentioned this pull request Jun 29, 2026

fix(memory): stop chunkMarkdown injecting newlines into single-line chunks #96747

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Memory: add configurable FTS5 tokenizer for CJK text support#56707

Memory: add configurable FTS5 tokenizer for CJK text support#56707
Takhoffman merged 4 commits into
mainfrom
codex/pr-41810-short-cjk-fallback

Takhoffman commented Mar 29, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Mar 29, 2026

Uh oh!

greptile-apps Bot commented Mar 29, 2026

Important Files Changed

Uh oh!

greptile-apps Bot Mar 29, 2026

Uh oh!

greptile-apps Bot Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Takhoffman commented Mar 29, 2026

Summary

Changelog

Verification

Notes

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Mar 29, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

greptile-apps Bot Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants