Skip to content

Commit 3e72715

Browse files
committed
fix(review): 标签截断不切在代理对中间(#612 第二轮)
按 UTF-16 长度切会切在 emoji 的代理对中间,留下一个孤立高位代理;那一半写进 SQLite 会变成替换字符,落盘后原文再也拼不回来。切完回看末位 charCodeAt,落在 0xd800..0xdbff 再退一格。
1 parent 8ed63c5 commit 3e72715

6 files changed

Lines changed: 54 additions & 13 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-3E63DD?style=flat-square" alt="license"></a>
1111
<a href="https://github.com/slow-stack/mneme/actions"><img src="https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml?style=flat-square&label=CI" alt="CI"></a>
1212
<a href="https://nodejs.org"><img src="https://img.shields.io/badge/node-22%2B-3E63DD?style=flat-square&logo=nodedotjs&logoColor=white" alt="node"></a>
13-
<a href="https://github.com/slow-stack/mneme"><img src="https://img.shields.io/badge/tests-2919%20total-3E63DD?style=flat-square" alt="tests"></a>
13+
<a href="https://github.com/slow-stack/mneme"><img src="https://img.shields.io/badge/tests-2921%20total-3E63DD?style=flat-square" alt="tests"></a>
1414
<a href="https://codecov.io/gh/slow-stack/mneme"><img src="https://img.shields.io/codecov/c/github/slow-stack/mneme/main?style=flat-square" alt="coverage"></a>
1515
<a href="https://github.com/awesome-dsh-plugin/awesome-dsh-plugin"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="Awesome"></a>
1616
</p>
@@ -276,7 +276,7 @@ dsh-mneme-serve # 默认 ~/.dsh/memory + 127.0.0.1:8790
276276

277277
```bash
278278
cd dsh-mneme && npm install
279-
npm test # 2919 个测试
279+
npm test # 2921 个测试
280280
npm run stress # 三轴线压测
281281
npm run sync # src → lib 同步
282282
```
@@ -562,7 +562,7 @@ Community projects that wire mneme into more platforms — not part of this repo
562562

563563
```bash
564564
cd dsh-mneme && npm install
565-
npm test # 2919 tests
565+
npm test # 2921 tests
566566
npm run stress # three-axis stress test
567567
npm run sync # src → lib sync
568568
```

‎dsh-mneme/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
[![npm version](https://img.shields.io/npm/v/@modusensus/dsh-mneme?color=blue&label=npm)](https://www.npmjs.com/package/@modusensus/dsh-mneme)
66
[![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
77
[![Awesome](https://awesome-dsh-plugin.com/badge.svg)](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin)
8-
[![tests](https://img.shields.io/badge/tests-2919%20total-success)](https://github.com/slow-stack/mneme)
8+
[![tests](https://img.shields.io/badge/tests-2921%20total-success)](https://github.com/slow-stack/mneme)
99
[![CI](https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml)](https://github.com/slow-stack/mneme/actions)
1010
[![node](https://img.shields.io/badge/node-22%2B-blue)](https://nodejs.org)
1111
[![npm downloads](https://img.shields.io/npm/d18m/@modusensus/dsh-mneme.svg?color=blue&label=downloads)](https://www.npmjs.com/package/@modusensus/dsh-mneme)
@@ -733,7 +733,7 @@ src/
733733
├── api.js # HTTP 路由(Web 面板数据通道,含 /conflicts 冲突队列)
734734
└── index.js # 插件接线
735735
lib/ # src 的同步分发产物(npm run sync,prepack 钩子在发布时自动跑;一致性由 scripts/check-sync.js 在 CI 与 release.yml 校验——它只查不同步;唯一手写例外 lib/client.js——Web 面板 bundle,sync 不覆盖)
736-
test/ # 2919 个 node:test 测试(审计与三轴线压测不变量;src↔lib 一致性由 scripts/check-sync.js 发布闸门校验)
736+
test/ # 2921 个 node:test 测试(审计与三轴线压测不变量;src↔lib 一致性由 scripts/check-sync.js 发布闸门校验)
737737
scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压测 · sync-lib.js 同步 · check-sync.js 发布闸门 · benchmark-recall.js / benchmark-embed.js / benchmark-rerank.js 基准 · sync-test-badge.mjs 测试徽章 · build-runtime-manifest.mjs 运行时清单
738738
```
739739
@@ -742,7 +742,7 @@ scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压
742742
```bash
743743
cd dsh-mneme
744744
npm install # 安装 peer 依赖(以 devDependencies 形式,用于本地测试)
745-
npm test # 运行 2919 个测试
745+
npm test # 运行 2921 个测试
746746
npm run stress # 三轴线压测:长会话检索 / 冲突仲裁 / 多 Agent 并发(离线 mock LLM)
747747
npm run sync # 把 src/ 同步到 lib/(发布时由 prepack 钩子自动执行)
748748
```

‎dsh-mneme/docs/DISTILL.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -336,6 +336,11 @@ audit 行,归因会跟着悬空;而 `memory_provenance` 刻意活得比原
336336
控制字符、连续空白一律折成单个空格,再按 80 字截断(超出加 `…`)。**无元数据时仍返回空串**
337337
这条既有行为不变,由调用方退化到裸 `[图]`。
338338
339+
截断不能切在 Unicode 代理对中间(同一轮 CodeRabbit 的第二条):`slice` 按 UTF-16 单元走,
340+
emoji 占两个单元,切在中间就留下一个孤立的高位代理——那一半写进 SQLite 会变成 `�`,
341+
落盘后原文再也拼不回来。所以切完要回头看一眼末位 `charCodeAt`:落在 `0xd800..0xdbff`
342+
就再退一格。
343+
339344
**不做(本批)**:不调 vision、不读字节、不落原图。宿主侧其实齐全
340345
(`attachments.readImage()` / `supportsAcpImagePrompts()` 判 `inputModalities`),
341346
但记忆库已 273MB,原图档明确不做;描述档(多模态模型出 1 段描述 + attachmentId 溯源)
@@ -353,13 +358,14 @@ audit 行,归因会跟着悬空;而 `memory_provenance` 刻意活得比原
353358
`user/message` 与 `agent/inbox/spliced`(投递路径重叠),按事件数会算两遍,所以累加必须放在
354359
去重 `continue` 之后。
355360
356-
**验收**:`test/attachment-text.test.js` 12 例锁口径(占位非空、三级退化、宽松字符串形状
357-
逐字节不变、reasoning 仍被挡、图+文顺序、countImages,加上消毒三条:`]` 不能闭合占位、
358-
换行不能造新行、超长名被截断);
361+
**验收**:`test/attachment-text.test.js` 14 例锁口径(占位非空、三级退化、宽松字符串形状
362+
逐字节不变、reasoning 仍被挡、图+文顺序、countImages,加上消毒四条:`]` 不能闭合占位、
363+
换行不能造新行、超长名被截断、代理对中间截断不留孤立代理项);
359364
`test/image-memory.test.js` 5 例锁端到端(图+文进转录、图-only 现在也会蒸馏、
360-
两张图数出 2、无图不写 imageCount、被挡掉的注入消息里的图不计入)。变异自证两条:
365+
两张图数出 2、无图不写 imageCount、被挡掉的注入消息里的图不计入)。变异自证三条:
361366
①把 `blockToText` 的 image 分支改回 `return ""` → 图-only 那条立刻红(`calls.length` 0 ≠ 1);
362-
②把计量改回按事件累加 → 「注入消息里的图不计入」那条立刻红。两条各自抓不同的错,
367+
②把计量改回按事件累加 → 「注入消息里的图不计入」那条立刻红;
368+
③去掉代理对守卫 → 「代理对中间截断不留孤立代理项」那条立刻红。三条各自抓不同的错,
363369
恢复后全绿。
364370
`test/continuity.test.js` 那条静态 import 锁同步从「恰好一行」改成「恰好两行」——
365371
新增一行仍然转红,仍然要人看一眼它是不是纯的(`attachment-text.js` 不 import 任何东西,

‎dsh-mneme/lib/attachment-text.js‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,13 @@ function sanitizeLabelPart(value) {
3838
// eslint-disable-next-line no-control-regex
3939
const flattened = value.replace(/[\u0000-\u001f\u007f[\]\\]/g, " ").replace(/\s+/g, " ").trim();
4040
if (flattened.length <= LABEL_PART_MAX) return flattened;
41-
return `${flattened.slice(0, LABEL_PART_MAX - 1)}…`;
41+
// 按 UTF-16 长度切会切在代理对中间(emoji 占两个单元),留下一个孤立的高位代理——
42+
// 那一半写进 SQLite 就是 `�`,落盘后原文再也拼不回来(CodeRabbit 2026-11-08 指出)。
43+
// 所以先切到上限,再回头检查末位是不是高位代理:是就再退一格。
44+
let cut = flattened.slice(0, LABEL_PART_MAX - 1);
45+
const last = cut.charCodeAt(cut.length - 1);
46+
if (last >= 0xd800 && last <= 0xdbff) cut = cut.slice(0, -1);
47+
return `${cut}…`;
4248
}
4349

4450
/** 一个图片块读出的人类可读标签(不含方括号)。什么元数据都没有时返回空串。 */

‎dsh-mneme/src/attachment-text.js‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,13 @@ function sanitizeLabelPart(value) {
3838
// eslint-disable-next-line no-control-regex
3939
const flattened = value.replace(/[\u0000-\u001f\u007f[\]\\]/g, " ").replace(/\s+/g, " ").trim();
4040
if (flattened.length <= LABEL_PART_MAX) return flattened;
41-
return `${flattened.slice(0, LABEL_PART_MAX - 1)}…`;
41+
// 按 UTF-16 长度切会切在代理对中间(emoji 占两个单元),留下一个孤立的高位代理——
42+
// 那一半写进 SQLite 就是 `�`,落盘后原文再也拼不回来(CodeRabbit 2026-11-08 指出)。
43+
// 所以先切到上限,再回头检查末位是不是高位代理:是就再退一格。
44+
let cut = flattened.slice(0, LABEL_PART_MAX - 1);
45+
const last = cut.charCodeAt(cut.length - 1);
46+
if (last >= 0xd800 && last <= 0xdbff) cut = cut.slice(0, -1);
47+
return `${cut}…`;
4248
}
4349

4450
/** 一个图片块读出的人类可读标签(不含方括号)。什么元数据都没有时返回空串。 */

‎dsh-mneme/test/attachment-text.test.js‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,4 +114,27 @@ test("超长文件名被截断,占位不被撑开", () => {
114114
const text = blockToText(imageBlock({ name: "a".repeat(300) + ".png" }));
115115
assert.ok(text.length < 140, `占位长度应受控,实际 ${text.length}`);
116116
assert.ok(text.endsWith("]"));
117+
});
118+
test("超长文件名在代理对中间截断时不留孤立代理项(CodeRabbit 2026-11-08 指出)", () => {
119+
// emoji 占两个 UTF-16 单元。按长度硬切会把它切成半个,孤立高位代理写进 SQLite
120+
// 就是 `�`——落盘后原文再也拼不回来,而这条路径正是往库里写文本的那条。
121+
const name = "a".repeat(78) + "😀".repeat(10);
122+
const label = imageLabel({ attachment: { name } });
123+
for (let i = 0; i < label.length; i += 1) {
124+
const code = label.charCodeAt(i);
125+
if (code >= 0xd800 && code <= 0xdbff) {
126+
const next = label.charCodeAt(i + 1);
127+
assert.ok(next >= 0xdc00 && next <= 0xdfff, `第 ${i} 位是孤立高位代理(没有配对低位)`);
128+
}
129+
}
130+
assert.ok(label.endsWith("…"));
131+
assert.ok(label.length <= 81, `不该超过上限,实际 ${label.length}`);
132+
});
133+
134+
test("恰好卡在边界上的 emoji 也不会被切坏", () => {
135+
// 79 个 a + 一个 emoji:截断点正好落在 emoji 的高位代理上,是这条分支最锋利的样例。
136+
const label = imageLabel({ attachment: { name: "a".repeat(79) + "😀" } });
137+
const last = label.charCodeAt(label.length - 2);
138+
assert.ok(!(last >= 0xd800 && last <= 0xdbff), "末位之前不能是孤立高位代理");
139+
assert.ok(label.endsWith("…"));
117140
});

0 commit comments

Comments
 (0)