Skip to content

Commit a514985

Browse files
authored
Merge pull request #406 from slow-stack/chore/239-distill-window-metrics
chore(distill): record the distill window size in audit metadata (#239)
2 parents 84c6f50 + 8a777bd commit a514985

5 files changed

Lines changed: 95 additions & 40 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-3E63DD?style=flat-square" alt="license"></a>
1111
<a href="https://github.com/slow-stack/mneme/actions"><img src="https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml?style=flat-square&label=CI" alt="CI"></a>
1212
<a href="https://nodejs.org"><img src="https://img.shields.io/badge/node-22%2B-3E63DD?style=flat-square&logo=nodedotjs&logoColor=white" alt="node"></a>
13-
<a href="https://github.com/slow-stack/mneme"><img src="https://img.shields.io/badge/tests-1613%20passed-3E63DD?style=flat-square" alt="tests"></a>
13+
<a href="https://github.com/slow-stack/mneme"><img src="https://img.shields.io/badge/tests-1615%20passed-3E63DD?style=flat-square" alt="tests"></a>
1414
<a href="https://codecov.io/gh/slow-stack/mneme"><img src="https://img.shields.io/codecov/c/github/slow-stack/mneme/main?style=flat-square" alt="coverage"></a>
1515
<a href="https://github.com/awesome-dsh-plugin/awesome-dsh-plugin"><img src="https://awesome-dsh-plugin.com/badge.svg" alt="Awesome"></a>
1616
</p>
@@ -207,7 +207,7 @@ dsh-mneme-serve # 默认 ~/.dsh/memory + 127.0.0.1:8790
207207

208208
```bash
209209
cd dsh-mneme && npm install
210-
npm test # 1613 个测试
210+
npm test # 1615 个测试
211211
npm run stress # 三轴线压测
212212
npm run sync # src → lib 同步
213213
```
@@ -414,7 +414,7 @@ Community projects that wire mneme into more platforms — not part of this repo
414414

415415
```bash
416416
cd dsh-mneme && npm install
417-
npm test # 1613 tests
417+
npm test # 1615 tests
418418
npm run stress # three-axis stress test
419419
npm run sync # src → lib sync
420420
```

‎dsh-mneme/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
[![npm version](https://img.shields.io/npm/v/@modusensus/dsh-mneme?color=blue&label=npm)](https://www.npmjs.com/package/@modusensus/dsh-mneme)
66
[![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
77
[![Awesome](https://awesome-dsh-plugin.com/badge.svg)](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin)
8-
[![tests](https://img.shields.io/badge/tests-1613%20passed-success)](https://github.com/slow-stack/mneme)
8+
[![tests](https://img.shields.io/badge/tests-1615%20passed-success)](https://github.com/slow-stack/mneme)
99
[![CI](https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml)](https://github.com/slow-stack/mneme/actions)
1010
[![node](https://img.shields.io/badge/node-22%2B-blue)](https://nodejs.org)
1111
[![npm downloads](https://img.shields.io/npm/d18m/@modusensus/dsh-mneme.svg?color=blue&label=downloads)](https://www.npmjs.com/package/@modusensus/dsh-mneme)
@@ -613,7 +613,7 @@ src/
613613
├── api.js # HTTP 路由(Web 面板数据通道,含 /conflicts 冲突队列)
614614
└── index.js # 插件接线
615615
lib/ # src 的同步分发产物(npm run sync;发布前由 root prepack 的 check-sync.js 校验一致性;唯一手写例外 lib/client.js——Web 面板 bundle,sync 不覆盖)
616-
test/ # 1613 个 node:test 测试(审计与三轴线压测不变量;src↔lib 一致性由 scripts/check-sync.js 发布闸门校验)
616+
test/ # 1615 个 node:test 测试(审计与三轴线压测不变量;src↔lib 一致性由 scripts/check-sync.js 发布闸门校验)
617617
scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压测 · sync-lib.js 同步 · check-sync.js 发布闸门 · benchmark-recall.js / benchmark-embed.js / benchmark-rerank.js 基准 · sync-test-badge.mjs 测试徽章 · build-runtime-manifest.mjs 运行时清单
618618
```
619619
@@ -622,7 +622,7 @@ scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压
622622
```bash
623623
cd dsh-mneme
624624
npm install # 安装 peer 依赖(以 devDependencies 形式,用于本地测试)
625-
npm test # 运行 1613 个测试
625+
npm test # 运行 1615 个测试
626626
npm run stress # 三轴线压测:长会话检索 / 冲突仲裁 / 多 Agent 并发(离线 mock LLM)
627627
npm run sync # 把 src/ 同步到 lib/(发布时由 prepack 钩子自动执行)
628628
```

‎dsh-mneme/lib/summarize.js‎

Lines changed: 26 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -574,23 +574,28 @@ export function createSummarizer(ctx, service, config, deps = {}) {
574574
// 判定纯规则(字符数),无模型参与;被挡下的窗口照常消费游标,否则每个
575575
// turn/end 都会重新评估同一段短文本。skip 原因进审计,作为后续按真实负载
576576
// 决定下一级规则的依据。
577+
//
578+
// 字符数改为无条件计算并写进审计 metadata:它是「下一级预判该不该开、阈值
579+
// 该给多少」的唯一依据,而原先开关一关就一个数都不留(维护者本机 1240 条
580+
// summarize_compress 成功行的 metadata 里窗口字符数为 0 条),验收 4「默认配置
581+
// 下调用量可量化下降」因此无法对账。本改动只加计量,不动任何跳过决策——
582+
// summarizeMinWindowChars 仍为 0 时行为逐字节不变。
583+
const distillChars = messages.reduce(
584+
(total, message) => total + (Array.isArray(message.content) ? message.content : [])
585+
.reduce((n, part) => n + (typeof part?.text === "string" ? part.text.length : 0), 0),
586+
0
587+
);
577588
const minWindowChars = config.summarizeMinWindowChars ?? 0;
578-
if (minWindowChars > 0) {
579-
const distillChars = messages.reduce(
580-
(total, message) => total + (Array.isArray(message.content) ? message.content : [])
581-
.reduce((n, part) => n + (typeof part?.text === "string" ? part.text.length : 0), 0),
582-
0
583-
);
584-
if (distillChars < minWindowChars) {
585-
commitCursor(session.id, nextSeq);
586-
writeAudit({
587-
timestamp: new Date().toISOString(),
588-
model_id: route ? `${route.provider}:${route.model}` : "unknown",
589-
status: "skipped",
590-
error_message: "window-too-small"
591-
});
592-
return;
593-
}
589+
if (minWindowChars > 0 && distillChars < minWindowChars) {
590+
commitCursor(session.id, nextSeq);
591+
writeAudit({
592+
timestamp: new Date().toISOString(),
593+
model_id: route ? `${route.provider}:${route.model}` : "unknown",
594+
status: "skipped",
595+
error_message: "window-too-small",
596+
metadata: { distillChars, minWindowChars }
597+
});
598+
return;
594599
}
595600
// 只有真正准备发起一次 LLM 蒸馏时才占用最小间隔与 run 预算;失败路径仍按
596601
// 原语义占用(调用确实发生过),aborted 则由 finally 回滚。
@@ -809,8 +814,12 @@ export function createSummarizer(ctx, service, config, deps = {}) {
809814
// 空数组是合法成功:没有记忆写入,但本次事件窗口仍然应被持久消费。
810815
persistCursor(session.id, nextSeq);
811816
}
812-
if (audit && (capped > 0 || deduped > 0 || parsedResult.salvaged)) {
817+
// Issue #239 定标:窗口字符数无条件随成功行落进 metadata。原先只有
818+
// capped / deduped / salvaged 其中一项发生时才建 metadata,默认档位下
819+
// 一条都不留——下一级预判规则的阈值就无从定标。
820+
if (audit) {
813821
audit.metadata = {
822+
distillChars,
814823
parsed: parsed.length,
815824
...(parsedResult.salvaged ? { json_salvaged: true } : {}),
816825
...(capped > 0 ? { capped } : {}),

‎dsh-mneme/src/summarize.js‎

Lines changed: 26 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -574,23 +574,28 @@ export function createSummarizer(ctx, service, config, deps = {}) {
574574
// 判定纯规则(字符数),无模型参与;被挡下的窗口照常消费游标,否则每个
575575
// turn/end 都会重新评估同一段短文本。skip 原因进审计,作为后续按真实负载
576576
// 决定下一级规则的依据。
577+
//
578+
// 字符数改为无条件计算并写进审计 metadata:它是「下一级预判该不该开、阈值
579+
// 该给多少」的唯一依据,而原先开关一关就一个数都不留(维护者本机 1240 条
580+
// summarize_compress 成功行的 metadata 里窗口字符数为 0 条),验收 4「默认配置
581+
// 下调用量可量化下降」因此无法对账。本改动只加计量,不动任何跳过决策——
582+
// summarizeMinWindowChars 仍为 0 时行为逐字节不变。
583+
const distillChars = messages.reduce(
584+
(total, message) => total + (Array.isArray(message.content) ? message.content : [])
585+
.reduce((n, part) => n + (typeof part?.text === "string" ? part.text.length : 0), 0),
586+
0
587+
);
577588
const minWindowChars = config.summarizeMinWindowChars ?? 0;
578-
if (minWindowChars > 0) {
579-
const distillChars = messages.reduce(
580-
(total, message) => total + (Array.isArray(message.content) ? message.content : [])
581-
.reduce((n, part) => n + (typeof part?.text === "string" ? part.text.length : 0), 0),
582-
0
583-
);
584-
if (distillChars < minWindowChars) {
585-
commitCursor(session.id, nextSeq);
586-
writeAudit({
587-
timestamp: new Date().toISOString(),
588-
model_id: route ? `${route.provider}:${route.model}` : "unknown",
589-
status: "skipped",
590-
error_message: "window-too-small"
591-
});
592-
return;
593-
}
589+
if (minWindowChars > 0 && distillChars < minWindowChars) {
590+
commitCursor(session.id, nextSeq);
591+
writeAudit({
592+
timestamp: new Date().toISOString(),
593+
model_id: route ? `${route.provider}:${route.model}` : "unknown",
594+
status: "skipped",
595+
error_message: "window-too-small",
596+
metadata: { distillChars, minWindowChars }
597+
});
598+
return;
594599
}
595600
// 只有真正准备发起一次 LLM 蒸馏时才占用最小间隔与 run 预算;失败路径仍按
596601
// 原语义占用(调用确实发生过),aborted 则由 finally 回滚。
@@ -809,8 +814,12 @@ export function createSummarizer(ctx, service, config, deps = {}) {
809814
// 空数组是合法成功:没有记忆写入,但本次事件窗口仍然应被持久消费。
810815
persistCursor(session.id, nextSeq);
811816
}
812-
if (audit && (capped > 0 || deduped > 0 || parsedResult.salvaged)) {
817+
// Issue #239 定标:窗口字符数无条件随成功行落进 metadata。原先只有
818+
// capped / deduped / salvaged 其中一项发生时才建 metadata,默认档位下
819+
// 一条都不留——下一级预判规则的阈值就无从定标。
820+
if (audit) {
813821
audit.metadata = {
822+
distillChars,
814823
parsed: parsed.length,
815824
...(parsedResult.salvaged ? { json_salvaged: true } : {}),
816825
...(capped > 0 ? { capped } : {}),

‎dsh-mneme/test/summarize.test.js‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,43 @@ test("#239 窗口过短:零 LLM 预判拦下调用,留 skip 审计并消费
274274
assert.equal(service.listLlmAudits().length, 1, "游标已消费:同一窗口不重复评估、不重复审计");
275275
});
276276

277+
// #239 定标埋点:窗口字符数必须落进审计,否则下一级预判规则的阈值无从选起。
278+
// 第二条才是本改动的目的——预判开关关着(默认档位)时行为逐字节不变,但从这一刻
279+
// 起成功行开始留数,验收 4「默认配置下调用量可量化下降」才有账可对。
280+
test("#239 计量:window-too-small 的 skip 行带 distillChars 与阈值", async () => {
281+
const { events, service } = setup({ summarizeMinWindowChars: 1000 });
282+
const handler = events.find((e) => e.name === "session/event").fn;
283+
const session = {
284+
id: "s239-metric-skip",
285+
requestHeader: () => ({ config: { provider: "deepseek", model: "deepseek-chat" } }),
286+
events: [userMessage("好", 1), { seq: 2, type: "turn/end" }]
287+
};
288+
await handler(session, { seq: 2, type: "turn/end" });
289+
const row = service.listLlmAudits()[0];
290+
assert.equal(row.error_message, "window-too-small");
291+
// 计的是**真正发给模型的转录文本**长度(含角色前缀:一条 1 字的「好」实测记 4),
292+
// 与闸门比较的口径同一个数,定标才对得上。这里不锁字面量,锁语义:短于阈值才该 skip。
293+
assert.ok(Number.isInteger(row.metadata?.distillChars) && row.metadata.distillChars > 0,
294+
"skip 行必须带窗口字符数");
295+
assert.equal(row.metadata.minWindowChars, 1000, "阈值随行落档,事后能复核是哪条闸门拦的");
296+
assert.ok(row.metadata.distillChars < row.metadata.minWindowChars, "skip 的前提:窗口短于阈值");
297+
});
298+
299+
test("#239 计量:预判开关关闭时成功行仍记录 distillChars", async () => {
300+
const { events, service } = setup();
301+
const handler = events.find((e) => e.name === "session/event").fn;
302+
const session = {
303+
id: "s239-metric-success",
304+
requestHeader: () => ({ config: { provider: "deepseek", model: "deepseek-chat" } }),
305+
events: [userMessage("今天决定把窗口字符数记进审计", 1), { seq: 2, type: "turn/end" }]
306+
};
307+
await handler(session, { seq: 2, type: "turn/end" });
308+
const row = service.listLlmAudits()[0];
309+
assert.equal(row.status, "success");
310+
assert.ok(Number.isInteger(row.metadata?.distillChars) && row.metadata.distillChars > 0,
311+
"默认档位下也要留数,否则验收 4 无从对账");
312+
});
313+
277314
test("#239 窗口达到阈值时预判放行(不误杀)", async () => {
278315
const { events, calls } = setup({ summarizeMinWindowChars: 5 });
279316
const handler = events.find((e) => e.name === "session/event").fn;

0 commit comments

Comments
 (0)