fix(memory-core): CJK textScore=0 and empty MATCH LIKE fallback by rrriiiccckkk · Pull Request #92341 · openclaw/openclaw

rrriiiccckkk · 2026-06-12T04:15:36Z

Summary

Fixes three related bugs that cause textScore: 0 in hybrid memory search for CJK queries (e.g. "飞书插件配置", "股票分析").

Bug 1: CJK tokens bypass LIKE path (trigram tokenizer)

In planKeywordSearch, CJK tokens with 3+ characters were sent to matchTerms as quoted phrase matches (MATCH "飞书插件配置"), which rarely match with the trigram tokenizer. The < 3 character threshold incorrectly routed long CJK strings to the MATCH path.

Fix: Remove the < 3 threshold so all CJK tokens go to substringTerms for LIKE matching.

Bug 2: No LIKE fallback when MATCH returns 0 rows

When FTS5 MATCH returned 0 rows (not an exception), no LIKE fallback triggered. This affected both CJK queries and short English tokens (e.g. AI, A) that the trigram tokenizer cannot index.

Fix: Add LIKE fallback with OR-based matching when MATCH returns empty.

Bug 3: mergeHybridResults discards textScore when chunk IDs don't overlap (#92337)

When keyword and vector results have different chunk IDs (common with CJK/trigram queries), the keyword search results were effectively discarded — textScore stayed 0 even though searchKeyword returned valid results.

Fix: Add path + line range overlap fallback in mergeHybridResults:

Build a vectorByPath secondary index grouped by file path
When no exact chunk ID match: find the vector chunk with the highest vectorScore on the same path with overlapping line ranges
Merge textScore into that vector entry (using Math.max, expanded line range)
If no overlap found: preserve original keyword-only behavior

Real behavior proof

Behavior or issue addressed: CJK memory search returns textScore: 0 for all results when querying with multi-character Chinese strings (e.g. "股票分析", "A股投研"). Root cause: (1) CJK tokens bypass LIKE path, (2) no LIKE fallback when MATCH returns 0 rows, (3) mergeHybridResults discards textScore when keyword/vector chunk IDs don't overlap.
Real environment tested: macOS arm64, OpenClaw 2026.6.5, Ollama qwen3-embedding:4b, memory store with 73 indexed files / 358 chunks, SQLite with trigram tokenizer.
Exact steps or command run after this patch:

openclaw memory search --json --max-results 10 "股票分析"
openclaw memory search --json --max-results 10 "A股投研"
openclaw memory search --json --max-results 10 "memory plugin"

Evidence after fix:

$ openclaw memory search --json --max-results 10 "股票分析"
  "textScore": 1,
  "vectorScore": 0.5289,
  "score": 0.6703

$ openclaw memory search --json --max-results 10 "A股投研"
  "textScore": 1,
  "vectorScore": 0.5008,
  "score": 0.6506

$ openclaw memory search --json --max-results 10 "memory plugin"
  "textScore": 0.7530,
  "vectorScore": 0.6297,
  "score": 0.6667

Observed result after fix:
- "股票分析": textScore=1, keyword+vector merged successfully
- "A股投研": textScore=1, keyword+vector merged successfully
- "memory plugin": textScore=0.75, keyword+vector merged successfully
- "飞书插件配置": textScore=0 — keyword results exist but from different files than vector results (LIKE fallback limitation, tracked in memory-core: CJK LIKE fallback returns wrong results for multi-character queries #92728)
What was not tested: CJK+English mixed queries (e.g. "OpenClaw配置") — LIKE fallback limitation tracked in memory-core: CJK LIKE fallback returns wrong results for multi-character queries #92728.

Files Changed

extensions/memory-core/src/memory/manager-search.ts — planKeywordSearch + searchKeyword (Bug 1 & 2)
extensions/memory-core/src/memory/hybrid.ts — mergeHybridResults (Bug 3)

Closes #92061
Fixes #92337

CJK tokens with 3+ characters were sent to matchTerms as quoted phrase matches (MATCH "飞书插件配置"), which rarely match with trigram tokenizer. Remove the character threshold so all CJK tokens go to substringTerms for LIKE matching with individual characters. Fixes openclaw#92061 (Bug 1)

clawsweeper · 2026-06-12T04:17:22Z

Codex review: found issues before merge. Reviewed June 26, 2026, 6:17 AM ET / 10:17 UTC.

Summary
The PR changes memory-core keyword search to route all CJK trigram tokens through LIKE, adds an empty-MATCH LIKE fallback, and merges keyword textScore into overlapping vector results.

PR surface: Source +69. Total +69 across 2 files.

Reproducibility: yes. for the source/dependency path: current main routes long CJK trigram tokens through MATCH and only falls back on exceptions, matching the reported zero-row behavior. I did not run a full live Ollama-backed memory search in this read-only review.

Review metrics: 1 noteworthy metric.

Regression coverage: 0 tests added. The PR changes memory retrieval scoring and selection, so CJK fallback precision and hybrid minScore behavior remain unguarded on the branch.

Stored data model
Persistent data-model change detected: unknown-data-model-change: extensions/memory-core/src/memory/hybrid.ts, vector/embedding metadata: extensions/memory-core/src/memory/hybrid.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #92061
Summary: This PR is the maintainer-routed validation thread for the original CJK textScore issue, but it partially overlaps separate hybrid preservation and remaining CJK LIKE precision tracks.

Members:

canonical: memory-core: CJK text FTS5 search returns textScore=0 with trigram tokenizer #92061 - A member closed the original issue as covered by this PR and kept this PR as the validation thread.
partial_overlap: memory-core: mergeHybridResults ignores textScore when keyword/vector chunk IDs don't overlap #92337 - The issue covers hybrid non-overlap/minScore behavior but was routed to a separate manager-boundary PR.
partial_overlap: fix(memory-core): preserve keyword-only hybrid results #92524 - The open PR is the active hybrid keyword-only preservation path and uses a different manager-boundary solution.
partial_overlap: memory-core: CJK LIKE fallback returns wrong results for multi-character queries #92728 - The open issue tracks CJK LIKE precision left unresolved by this PR's whole-token fallback.
superseded: fix(memory-core): score CJK keyword search instead of textScore=0 with trigram FTS5 #92164 - The earlier CJK PR proposed a fuller keyword-search repair but is closed unmerged and is not a viable canonical target.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🐚 platinum hermit
Patch quality: 🧂 unranked krab
Result: blocked by patch quality or review findings.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P2] Choose and implement the CJK fallback decomposition policy with focused manager-search coverage.
[P2] Align the hybrid non-overlap fix with the manager-boundary preservation path or remove that part from this PR.

Risk before merge

[P2] Merging the OR-based empty-MATCH fallback can select rows containing only one query token and score them as fully relevant memory hits.
[P1] The PR still leaves multi-character CJK LIKE matching as a whole-token exact substring, which the open follow-up issue shows is not the intended relevance behavior.
[P1] The overlap merge changes hybrid result identity and can conflict with the separate manager-boundary keyword-only preservation path under review.

Maintainer options:

Repair Retrieval Semantics Before Merge (recommended)
Require a maintainer-approved CJK decomposition strategy, stricter empty-MATCH filtering, and alignment with the manager-boundary hybrid preservation path before landing.
Use Separate Canonical Tracks
Pause or close this branch if maintainers prefer the open CJK LIKE issue for fallback semantics and the separate hybrid-preservation PR for minScore behavior.
Accept Partial Recall Risk
Maintainers can intentionally land the partial improvement, but would own inconsistent CJK memory relevance and context selection until follow-up work lands.

Next step before merge

[P2] Maintainers need to choose the CJK relevance semantics and whether this branch or the separate hybrid-preservation PR owns the non-overlap fix.

Security
Cleared: No concrete security or supply-chain concern was found; the diff is limited to parameterized memory search SQL and in-memory result merge logic.

Review findings

[P2] Split CJK fallback terms before exact LIKE matching — extensions/memory-core/src/memory/manager-search.ts:126-127
[P2] Keep empty-MATCH fallback as a stricter filter — extensions/memory-core/src/memory/manager-search.ts:387
[P2] Preserve keyword-only rows after overlap merges — extensions/memory-core/src/memory/hybrid.ts:143-153

Review details

Best possible solution:

Use one canonical manager-search fix for CJK decomposition and empty-MATCH semantics, and keep keyword-only hybrid preservation at the manager selection boundary chosen by maintainers.

Do we have a high-confidence way to reproduce the issue?

Yes for the source/dependency path: current main routes long CJK trigram tokens through MATCH and only falls back on exceptions, matching the reported zero-row behavior. I did not run a full live Ollama-backed memory search in this read-only review.

Is this the best way to solve the issue?

No. This is a plausible partial repair, but it is not the best fix until CJK term decomposition, empty-MATCH fallback semantics, and hybrid keyword-preservation boundaries are corrected and covered.

Full review comments:

[P2] Split CJK fallback terms before exact LIKE matching — extensions/memory-core/src/memory/manager-search.ts:126-127
The patch routes every CJK token to substringTerms, but still pushes a multi-character query such as 飞书插件配置 as one exact substring. The PR body and memory-core: CJK LIKE fallback returns wrong results for multi-character queries #92728 show that this whole-token LIKE shape can still miss or mis-rank the intended CJK memory hits, so the bug is not closed until a chosen split-term strategy is implemented.
Confidence: 0.9
[P2] Keep empty-MATCH fallback as a stricter filter — extensions/memory-core/src/memory/manager-search.ts:387
This new empty-MATCH branch uses OR-based LIKE predicates and then maps fallback rows to textScore = 1. If MATCH returns no rows because one term is unindexable, rows containing only one query token can now be treated as fully relevant memory hits.
Confidence: 0.86
[P2] Preserve keyword-only rows after overlap merges — extensions/memory-core/src/memory/hybrid.ts:143-153
When a keyword hit overlaps a vector hit, this code merges it into the vector entry and drops the original keyword-only row. If the merged row does not pass the strict threshold or no longer matches the manager's keyword source/path/line key, the lexical hit can still be omitted; the related manager-boundary PR keeps this preservation decision at the selection layer.
Confidence: 0.85

Overall correctness: patch is incorrect
Overall confidence: 0.87

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against ec737ee74d9b.

Label changes

Label justifications:

P2: This is a bounded memory-core retrieval correctness bug with real CJK relevance impact but limited blast radius.
merge-risk: 🚨 session-state: The diff changes retrieval and scoring of persisted memory results that can be selected into agent/session context.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🐚 platinum hermit and patch quality is 🧂 unranked krab.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body includes terminal-style after-fix OpenClaw memory search output from a real macOS setup for two CJK queries and one English query.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal-style after-fix OpenClaw memory search output from a real macOS setup for two CJK queries and one English query.

Evidence reviewed

PR surface:

Source +69. Total +69 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	80	11	+69
Tests	0	0	0	0
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	80	11	+69

What I checked:

Current main still has the reported CJK planner behavior: On current main, only CJK trigram tokens shorter than three characters are routed to substring LIKE; longer CJK queries still go through MATCH. (extensions/memory-core/src/memory/manager-search.ts:126, ec737ee74d9b)
PR merge result keeps whole CJK tokens as exact LIKE terms: The PR removes the length threshold but still pushes the full multi-character CJK token into substringTerms, so exact-phrase LIKE behavior remains for queries like 飞书插件配置. (extensions/memory-core/src/memory/manager-search.ts:126, f957ad7797b9)
PR merge result uses OR fallback with full fallback textScore: The empty-MATCH path builds OR LIKE predicates and then marks rows as fallback results with textScore 1, unlike the existing exception fallback's all-term AND behavior. (extensions/memory-core/src/memory/manager-search.ts:387, f957ad7797b9)
Manager boundary already owns keyword-only fallback selection: Current manager selection preserves keyword-backed rows by source/path/line only when no strict rows survive, so mutating ranges inside the hybrid merger can bypass this preservation key. (extensions/memory-core/src/memory/manager.ts:792, ec737ee74d9b)
Related CJK precision issue remains open: The open follow-up issue tracks the same whole-token LIKE shape left by this PR and describes wrong or missing results for multi-character CJK queries.
History ties CJK trigram behavior to configurable tokenizer work: Commit 3ce48af added configurable FTS5 tokenizer support and adjacent manager-search tests for CJK text support. (extensions/memory-core/src/memory/manager-search.ts:126, 3ce48aff660a)

Likely related people:

Takhoffman: Commit 3ce48af added configurable FTS5 trigram support and adjacent manager-search tests for CJK text support. (role: introduced CJK tokenizer behavior; confidence: high; commits: 3ce48aff660a; files: extensions/memory-core/src/memory/manager-search.ts, extensions/memory-core/src/memory/manager-search.test.ts)
AnonAmit: Commit 4259010 changed memory fallback lexical ranking in manager-search, its tests, and the manager search flow adjacent to this fallback behavior. (role: fallback ranking contributor; confidence: medium; commits: 42590106ab6e; files: extensions/memory-core/src/memory/manager-search.ts, extensions/memory-core/src/memory/manager-search.test.ts, extensions/memory-core/src/memory/manager.ts)
steipete: Commit cad83db moved the memory engine into the memory-core plugin across the manager-search, manager, and hybrid boundaries involved here. (role: memory plugin boundary refactor contributor; confidence: high; commits: cad83db8b2f7, dbf78de7c680; files: extensions/memory-core/src/memory/manager-search.ts, extensions/memory-core/src/memory/manager.ts, extensions/memory-core/src/memory/hybrid.ts)
vignesh07: Commit f57b466 introduced the earlier keyword-hit preservation path that the PR's hybrid overlap merge now interacts with. (role: adjacent keyword-hit fallback contributor; confidence: medium; commits: f57b4669e1d6; files: src/memory/manager.ts, extensions/memory-core/src/memory/manager.ts)
vincentkoc: Recent memory-core manager edits and live issue comments route the original CJK issue to this PR and the hybrid non-overlap issue to the separate manager-boundary PR. (role: recent area contributor and issue router; confidence: medium; commits: ca26489fe882, 620537914b8c; files: extensions/memory-core/src/memory/manager.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Two fixes: 1. Remove CJK < 3 char threshold: all CJK tokens now go to substringTerms for LIKE matching instead of being sent as quoted phrase MATCH. 2. Add LIKE fallback when FTS5 MATCH returns 0 rows: previously only triggered on MATCH exception. Now also triggers on empty results, using OR-based LIKE matching for better recall. Fixes openclaw#92061

rrriiiccckkk · 2026-06-13T02:34:43Z

Updated PR description with Real behavior proof. Ready for re-review.

…penclaw#92337 When keyword and vector results have different chunk IDs (common with CJK/trigram queries), fall back to matching by path + overlapping line range instead of discarding textScore. Changes: - Add linesOverlap() helper function - Add vectorByPath secondary index for O(1) path lookups - When no chunk ID match: find vector chunk with highest vectorScore on same path with overlapping lines - Merge textScore into that vector entry (max textScore, expanded line range) - If no overlap found: preserve original keyword-only behavior Fixes openclaw#92337

rrriiiccckkk · 2026-06-14T11:24:12Z

Lint fix pushed. All CI checks passing now. Ready for re-review. @clawsweeper

rrriiiccckkk · 2026-06-15T09:51:04Z

Hi @vincentkoc 👋 — this PR has been ready for review since June 14 (all CI green, mergeable, proof supplied). Could you take a look when you have a moment? Thanks!

openclaw-barnacle · 2026-07-15T04:25:11Z

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

openclaw-barnacle · 2026-07-22T04:33:57Z

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 12, 2026

clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label Jun 12, 2026

rrriiiccckkk changed the title ~~fix(memory-core): remove CJK length threshold for trigram LIKE matching~~ fix(memory-core): CJK textScore=0 + empty MATCH LIKE fallback Jun 12, 2026

rrriiiccckkk changed the title ~~fix(memory-core): CJK textScore=0 + empty MATCH LIKE fallback~~ fix(memory-core): CJK textScore=0 and empty MATCH LIKE fallback Jun 12, 2026

rrriiiccckkk added 3 commits June 13, 2026 10:38

chore: trigger CI re-run

74a90e8

chore: update PR body with required proof fields

2912edd

chore: fix proof format for CI

223e990

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 13, 2026

github-actions Bot mentioned this pull request Jun 13, 2026

🦞 OpenClaw 生态日报 2026-06-13 jasonalang/big_model_radar#82

Open

openclaw-barnacle Bot added size: S and removed size: XS labels Jun 13, 2026

rrriiiccckkk mentioned this pull request Jun 13, 2026

memory-core: CJK LIKE fallback returns wrong results for multi-character queries #92728

Open

openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 13, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 13, 2026

fix: lint error - unnecessary escape in ESCAPE clause

8835358

fix: add curly braces for eslint(curly) in hybrid.ts lines 144-145

9ece62d

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 14, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 14, 2026

clawsweeper Bot mentioned this pull request Jun 15, 2026

memory-core: CJK text FTS5 search returns textScore=0 with trigram tokenizer #92061

Closed

openclaw-clownfish Bot mentioned this pull request Jun 15, 2026

fix(memory-core): score CJK keyword search instead of textScore=0 with trigram FTS5 #92164

Closed

clawsweeper Bot mentioned this pull request Jun 19, 2026

fix(memory-core): preserve keyword-only hybrid results #92524

Open

openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 15, 2026

openclaw-barnacle Bot closed this Jul 22, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory-core): CJK textScore=0 and empty MATCH LIKE fallback#92341

fix(memory-core): CJK textScore=0 and empty MATCH LIKE fallback#92341
rrriiiccckkk wants to merge 8 commits into
openclaw:mainfrom
rrriiiccckkk:fix/cjk-textscore-92061

rrriiiccckkk commented Jun 12, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

rrriiiccckkk commented Jun 13, 2026

Uh oh!

rrriiiccckkk commented Jun 14, 2026

Uh oh!

rrriiiccckkk commented Jun 15, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 15, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

rrriiiccckkk commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Bug 1: CJK tokens bypass LIKE path (trigram tokenizer)

Bug 2: No LIKE fallback when MATCH returns 0 rows

Bug 3: mergeHybridResults discards textScore when chunk IDs don't overlap (#92337)

Real behavior proof

Files Changed

Uh oh!

clawsweeper Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rrriiiccckkk commented Jun 13, 2026

Uh oh!

rrriiiccckkk commented Jun 14, 2026

Uh oh!

rrriiiccckkk commented Jun 15, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 15, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rrriiiccckkk commented Jun 12, 2026 •

edited

Loading

clawsweeper Bot commented Jun 12, 2026 •

edited

Loading