Skip to content

Commit 7412400

Browse files
authored
Merge pull request #286 from z2Ace0107/codex/issue-250-llm-audit-coverage
feat(audit): 实体抽取与 Sleep Mode 计入 llm_audit_logs(#250)
2 parents d423706 + 180e75c commit 7412400

9 files changed

Lines changed: 602 additions & 29 deletions

File tree

‎dsh-mneme/README.md‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,11 +189,37 @@ v0.3.0 起新增**记忆基因**层:从记忆里抽取**命名实体**、**带
189189

190190
### LLM 消耗审计 📊(v0.4.6,默认开)
191191

192-
每次**后台 LLM 调用**(autoDream 整理 + 摘要、autoSummarize 压缩)都会写入 `llm_audit_logs` 表:`tokens` / `duration` / `status` / `source`(由哪个触发产生)。失败调用记为 `status=error`,绝不阻塞功能本体;`retentionDays`(默认 90)在启动时清理超期行。新增两个只读 API:
192+
每次**后台 LLM 调用**都会写入 `llm_audit_logs` 表:`tokens` / `duration` / `status` / `source`(由哪个触发产生)。失败调用记为 `status=error`,绝不阻塞功能本体;`retentionDays`(默认 90)在启动时清理超期行。
193+
194+
覆盖的后台调用(`source` 即触发源,`operation_type` 区分同一源下的不同调用):
195+
196+
| `source` | `operation_type` | 说明 |
197+
|---|---|---|
198+
| `autoDream` | `dream_consolidate` / `dream_summarize` | 巩固的整理裁决与总览摘要,各一行 |
199+
| `autoDream` | `dream_narrative` | 叙述条合成,需 `dreamNarrativeEnabled`(默认关)才跑 |
200+
| `autoSummarize` | `summarize_compress` | 空闲蒸馏的压缩 |
201+
| `sleep` | `sleep_conflict` / `sleep_pattern` | Sleep Mode 的冲突裁决与模式挖掘,两阶段各一行 |
202+
| `entityExtract` | `entity_extract` | 实体抽取,**每次写入记忆都会跑** |
203+
204+
> 后两条链路见 #250——它们是结构上漏接的后台路径(sleep 的 `streamText` 副本没有 `onUsage`、实体抽取适配器拿不到 `service`),不是刻意收窄口径。
205+
206+
新增两个只读 API:
193207

194208
- `GET /api/dsh-mneme/semantic/llm-audit?page=&pageSize=&source=` — 分页查询 + 按 source 过滤
195209
- `GET /api/dsh-mneme/semantic/llm-audit/stats?days=` — 近 N 天按 source 汇总预算(tokens / 次数 / 失败数)
196210

211+
`/llm-audit/stats` 的 `by_source` 是服务端 `GROUP BY trigger_source`,不做任何白名单过滤,因此新链路自动进入汇总。
212+
213+
面板侧现状(#250 之后,未改面板):
214+
215+
- 「LLM 消耗」卡读 `/llm-audit/stats`,但只渲染 `total_calls` / `total_tokens` 两个总数。新链路的 token 与次数**已经计入这两个总数**,所以「mneme 总共花了多少」现在是对的;
216+
- 按链路的拆分(`by_source`)目前只能从 API 读,面板不展示——「哪条链路最费」在面板上还答不了;
217+
- 状态页活动流仍只显示 `autoDream` / `autoSummarize` 两个硬编码来源(`lib/client.js`),新链路不会出现在活动流里。
218+
219+
这三处要动面板(活动流还有一处来源白名单加一处二元标签兜底,且 `pageSize=12` 会被每次写入都跑的实体抽取占满),留到面板批次,不在 #250 内。
220+
221+
> 实体抽取是**每次写入记忆都会跑**的链路,计入审计后 `llm_audit_logs` 的行数会随写入量线性增长(`retentionDays` 只约束上界、不约束增速)。若实测增速过快,再考虑按 `operation_type` 采样或单独保留期。
222+
197223
> 只读端点,与 list/search/semantic 一样在设置 `apiToken` 后仍保持开放。
198224
199225
### 记忆库主区视图与记忆图谱 🕸️(v0.5.0)

‎dsh-mneme/lib/config.js‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -500,11 +500,18 @@ export const Config = z.object({
500500
}).default({}),
501501

502502
// --- LLM audit trail (Bug8) ------------------------------------------------
503-
// Records every background LLM call (autoDream consolidation + summary,
504-
// autoSummarize compression) into llm_audit_logs: tokens, duration, status
505-
// and which trigger produced it. Failures are recorded as status=error and
506-
// never block the feature. retentionDays bounds the table: older rows are
507-
// purged on boot.
503+
// Records every background LLM call into llm_audit_logs: tokens, duration,
504+
// status and which trigger produced it. Failures are recorded as
505+
// status='error' and never block the feature. retentionDays bounds the table:
506+
// older rows are purged on boot.
507+
//
508+
// Covered trigger_source values: autoDream (consolidation + summary),
509+
// autoSummarize (compression), sleep (conflict + pattern) and entityExtract
510+
// (issue #250 — the last two were structurally missing the hook, not a
511+
// deliberate narrowing). entityExtract runs on every memory write, so row
512+
// growth scales with write volume; retentionDays caps the ceiling but not the
513+
// rate. If that turns out too fast, sample by operation_type or give it a
514+
// separate retention — do not add a second gate: this flag is the single one.
508515
llmAudit: z.object({
509516
enabled: z.boolean().default(true),
510517
retentionDays: z.natural().min(1).max(3650).default(90)

‎dsh-mneme/lib/dream/sleep.js‎

Lines changed: 110 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -49,12 +49,22 @@ function parseJsonArray(text) {
4949
}
5050
}
5151

52-
/** Same stream consumption contract as dream.js. */
53-
async function streamText(ctx, options, onStreamError) {
52+
/**
53+
* Same stream consumption contract as dream.js.
54+
*
55+
* `onUsage`(optional, issue #250)receives any usage chunk for token accounting.
56+
* dream.js 那份一直有它;sleep 这份副本没有,导致 conflict / pattern 两条链路的
57+
* token 从未进 llm_audit_logs(面板因此只看到 autoDream 与 autoSummarize)。
58+
*/
59+
async function streamText(ctx, options, onUsage, onStreamError) {
5460
if (!ctx?.llm?.stream) return undefined;
5561
let text = "";
5662
for await (const chunk of ctx.llm.stream(options)) {
5763
if (chunk.type === "text-delta" && typeof chunk.text === "string") text += chunk.text;
64+
// DSH 的 StreamChunk 契约把用量嵌在 chunk.usage(TokenUsage:inputTokens /
65+
// outputTokens / totalTokens);兼容直接平铺在 chunk 上的旧形态。归一放在
66+
// 这里,调用侧的 reporter 只需认平铺形状——与 dream.js:239 同一处理。
67+
if (chunk.type === "usage" && typeof onUsage === "function") onUsage(chunk.usage ?? chunk);
5868
if (chunk.type === "finish" && (chunk.reason?.kind === "error" || chunk.reason?.kind === "aborted")) {
5969
// Same rc.1 error-as-finish-chunk behavior as dream.js — surface the
6070
// cause instead of discarding it.
@@ -67,6 +77,79 @@ async function streamText(ctx, options, onStreamError) {
6777
return text;
6878
}
6979

80+
/**
81+
* Issue #250:给 sleep 的一条 LLM 调用记一笔 llm_audit_logs(token / 时长 / 状态 /
82+
* 触发源)。与 dream.js 的 runAuditedLlm 同口径但各自独立一份——dream 那两个
83+
* checker 的调用形状绑定着它自己的排程细节,硬抽共享抽象只会收敛回 dream 的形态。
84+
*
85+
* 记账是 best-effort:写审计行失败只 warn,绝不反噬 sleep 本体(CONTRIBUTING 的
86+
* fail-safe 硬约定)。`body(reportUsage)` 负责真实消费流,usage chunk 经
87+
* reportUsage 上报。
88+
*
89+
* 审计诚实性:流被中止/报错时 body 返回 undefined,记 status='error';流式成功但
90+
* 输出不可用时 `spec.auditError` 给出原因,同样记 error——否则同一轮调用会在
91+
* 「phase 报 failed」与「audit 报 success」之间自相矛盾(CONTRIBUTING 的审计诚实性
92+
* 硬约定)。`auditError` 必须是**纯判据函数**:调用方的控制流不能依赖它的副作用,
93+
* 因为审计关闭时这里整段早退、钩子根本不会被调用(dream 正是那样踩的,见 PR 正文)。
94+
*/
95+
async function auditedSleepLlm(ctx, service, config, logger, spec, body) {
96+
if (config?.llmAudit?.enabled === false || typeof service?.saveLlmAudit !== "function") {
97+
return body(() => {});
98+
}
99+
const startedAt = Date.now();
100+
const timestamp = new Date(startedAt).toISOString();
101+
let inputTokens = 0;
102+
let outputTokens = 0;
103+
let status = "success";
104+
let errorMessage = null;
105+
try {
106+
const result = await body((usage) => {
107+
if (!usage) return;
108+
const i = usage.input_tokens ?? usage.inputTokens ?? usage.prompt_tokens ?? usage.promptTokens;
109+
const o = usage.output_tokens ?? usage.outputTokens ?? usage.completion_tokens ?? usage.completionTokens;
110+
if (Number.isFinite(i)) inputTokens = i;
111+
if (Number.isFinite(o)) outputTokens = o;
112+
});
113+
if (result === undefined) {
114+
status = "error";
115+
const streamErr = typeof spec.streamError === "function" ? String(spec.streamError() ?? "") : "";
116+
errorMessage = streamErr ? `llm stream aborted or errored (${streamErr})` : "llm stream aborted or errored";
117+
} else if (typeof spec.auditError === "function") {
118+
// 流式成功但输出不可用,仍然是一次失败的调用——记 error,不与调用方的
119+
// failed 返回值打架。
120+
const message = spec.auditError(result);
121+
if (message) {
122+
status = "error";
123+
errorMessage = message;
124+
}
125+
}
126+
return result;
127+
} catch (error) {
128+
status = "error";
129+
errorMessage = String(error?.message ?? error);
130+
throw error;
131+
} finally {
132+
try {
133+
service.saveLlmAudit({
134+
timestamp,
135+
trigger_source: spec.triggerSource,
136+
operation_type: spec.operationType,
137+
model_id: spec.modelId,
138+
input_tokens: inputTokens,
139+
output_tokens: outputTokens,
140+
total_tokens: inputTokens + outputTokens,
141+
cost_usd: 0,
142+
duration_ms: Date.now() - startedAt,
143+
status,
144+
error_message: errorMessage,
145+
related_memory_ids: spec.relatedMemoryIds ?? []
146+
});
147+
} catch (auditError) {
148+
logger?.warn?.(`dsh-mneme sleep: llm audit write failed: ${String(auditError)}`);
149+
}
150+
}
151+
}
152+
70153
/** LLM route (Issue #25): explicit sleepProvider/Model wins, then the dream
71154
* route as a shared explicit fallback, then the agent default model. Sleep
72155
* can pin a cheaper model for its bulk passes without disturbing the dream
@@ -232,7 +315,19 @@ async function phaseConflicts(ctx, service, config, logger, runId, semantic = nu
232315
const conflictPrompt = actionSet === "full" ? STR.prompts.conflictFull[language] : STR.prompts.conflict[language];
233316
const runConflict = (withEffort) => {
234317
conflictStreamFailure = "";
235-
return streamText(ctx, {
318+
return auditedSleepLlm(ctx, service, config, logger, {
319+
triggerSource: "sleep",
320+
operationType: "sleep_conflict",
321+
modelId: `${route.provider}:${route.model}`,
322+
// 候选对的双方 id:面板的活动流用 related_memory_ids 解析「沉淀条数」。
323+
relatedMemoryIds: samePairs.flatMap((p) => [p.a.id, p.b.id]),
324+
streamError: () => conflictStreamFailure,
325+
// 纯判据:输出里没有 JSON 数组 ⇒ 这轮是失败的调用,审计记 error。**不要**把
326+
// 解析结果存进闭包给下面的控制流用——审计关闭时 auditedSleepLlm 整段早退、
327+
// 这个钩子不会被调用,控制流会跟着失效(dream 的既有坑)。代价是审计开启时
328+
// 多解析一次,换来两条路径彻底解耦。
329+
auditError: (text) => (parseJsonArray(text) ? null : "no json array in llm output")
330+
}, (reportUsage) => streamText(ctx, {
236331
provider: route.provider,
237332
model: route.model,
238333
purpose: "sleep-conflict",
@@ -242,7 +337,7 @@ async function phaseConflicts(ctx, service, config, logger, runId, semantic = nu
242337
{ role: "system", content: [{ type: "text", text: conflictPrompt }], source: { kind: "plugin", plugin: "dsh-mneme" } },
243338
{ role: "user", content: [{ type: "text", text: listText }], source: { kind: "plugin", plugin: "dsh-mneme" } }
244339
]
245-
}, (reason) => { conflictStreamFailure = describeStreamFailure(reason); });
340+
}, reportUsage, (reason) => { conflictStreamFailure = describeStreamFailure(reason); }));
246341
};
247342
const text = await withEffortFallback(ctx, sleepEffort, () => runConflict(true), () => runConflict(false), () => conflictStreamFailure);
248343
if (text === undefined) {
@@ -385,7 +480,16 @@ async function phasePatterns(ctx, service, config, logger, runId, signal = null)
385480
let patternStreamFailure = "";
386481
const runPattern = (withEffort) => {
387482
patternStreamFailure = "";
388-
return streamText(ctx, {
483+
return auditedSleepLlm(ctx, service, config, logger, {
484+
triggerSource: "sleep",
485+
operationType: "sleep_pattern",
486+
modelId: `${route.provider}:${route.model}`,
487+
// 本轮扫描的记忆 id:与 dream 的 relatedMemoryIds 同义,供面板解析沉淀条数。
488+
relatedMemoryIds: memories.map((m) => m.id),
489+
streamError: () => patternStreamFailure,
490+
// 纯判据,同 conflict 阶段:不把解析结果存进闭包供控制流使用。
491+
auditError: (text) => (parseJsonArray(text) ? null : "no json array in llm output")
492+
}, (reportUsage) => streamText(ctx, {
389493
provider: route.provider,
390494
model: route.model,
391495
purpose: "sleep-pattern",
@@ -395,7 +499,7 @@ async function phasePatterns(ctx, service, config, logger, runId, signal = null)
395499
{ role: "system", content: [{ type: "text", text: STR.prompts.pattern[language].replace("N", String(maxPatterns)) }], source: { kind: "plugin", plugin: "dsh-mneme" } },
396500
{ role: "user", content: [{ type: "text", text: listText }], source: { kind: "plugin", plugin: "dsh-mneme" } }
397501
]
398-
}, (reason) => { patternStreamFailure = describeStreamFailure(reason); });
502+
}, reportUsage, (reason) => { patternStreamFailure = describeStreamFailure(reason); }));
399503
};
400504
const text = await withEffortFallback(ctx, sleepEffort, () => runPattern(true), () => runPattern(false), () => patternStreamFailure);
401505
if (text === undefined) {

‎dsh-mneme/lib/index.js‎

Lines changed: 60 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,15 @@ export { Config };
4343
// unit-testable; the extractor only ever sees a callLLM(messages, options)
4444
// => Promise<string>. The route always carries a real provider/model (dsh-llm
4545
// GenerateOptions requires both) — never a bare stream.
46-
export function createEntityStreamAdapter({ llm, agentDefaultModel, logger }) {
46+
//
47+
// Issue #250: it also accounts for itself. Entity extraction is the background
48+
// LLM path that runs on every memory write, yet it never reached
49+
// llm_audit_logs — the adapter only ever held an llm handle, so it had no way
50+
// to call service.saveLlmAudit. It now takes service (+ config for the shared
51+
// llmAudit.enabled gate) and writes one row per stream attempt, matching the
52+
// contract of runAuditedLlm in dream.js: a rejected effort attempt records its
53+
// own error row and the retry records its own success row.
54+
export function createEntityStreamAdapter({ llm, agentDefaultModel, logger, service, config }) {
4755
return async function streamEntityText(messages, options = {}) {
4856
let route = {};
4957
if (options.provider) route.provider = options.provider;
@@ -58,8 +66,38 @@ export function createEntityStreamAdapter({ llm, agentDefaultModel, logger }) {
5866
} catch { /* fall through to whatever route we already have */ }
5967
}
6068
const effort = options.reasoningEffort;
69+
// 没有解析出 provider/model 就没有可归属的模型——与 dream 的 resolveRoute
70+
// 无路由早退同口径,那种情况不写审计行。
71+
const modelId = route.provider && route.model ? `${route.provider}:${route.model}` : "";
6172
const tryStream = (withEffort) => {
6273
let text = "";
74+
let inputTokens = 0;
75+
let outputTokens = 0;
76+
const startedAt = Date.now();
77+
const timestamp = new Date(startedAt).toISOString();
78+
// 记账是 best-effort:写审计行失败只 warn,绝不反噬抽取本身(CONTRIBUTING
79+
// 的 fail-safe 硬约定)。
80+
const writeAudit = (status, errorMessage) => {
81+
if (config?.llmAudit?.enabled === false || !modelId || typeof service?.saveLlmAudit !== "function") return;
82+
try {
83+
service.saveLlmAudit({
84+
timestamp,
85+
trigger_source: "entityExtract",
86+
operation_type: "entity_extract",
87+
model_id: modelId,
88+
input_tokens: inputTokens,
89+
output_tokens: outputTokens,
90+
total_tokens: inputTokens + outputTokens,
91+
cost_usd: 0,
92+
duration_ms: Date.now() - startedAt,
93+
status,
94+
error_message: errorMessage,
95+
related_memory_ids: []
96+
});
97+
} catch (auditError) {
98+
logger?.warn?.(`dsh-mneme: entity extraction llm audit write failed: ${String(auditError)}`);
99+
}
100+
};
63101
return (async () => {
64102
for await (const chunk of llm.stream({
65103
...route,
@@ -68,11 +106,26 @@ export function createEntityStreamAdapter({ llm, agentDefaultModel, logger }) {
68106
messages
69107
})) {
70108
if (chunk.type === "text-delta" && typeof chunk.text === "string") text += chunk.text;
71-
if (chunk.type === "finish" && (chunk.reason?.kind === "error" || chunk.reason?.kind === "aborted")) return undefined;
109+
// DSH 的 StreamChunk 契约把用量嵌在 chunk.usage(TokenUsage);兼容直接
110+
// 平铺在 chunk 上的旧形态。归一放在这里,下面只需认平铺形状——与
111+
// dream.js 的 streamText 同一处理。
112+
if (chunk.type === "usage") {
113+
const u = chunk.usage ?? chunk;
114+
const i = u.input_tokens ?? u.inputTokens ?? u.prompt_tokens ?? u.promptTokens;
115+
const o = u.output_tokens ?? u.outputTokens ?? u.completion_tokens ?? u.completionTokens;
116+
if (Number.isFinite(i)) inputTokens = i;
117+
if (Number.isFinite(o)) outputTokens = o;
118+
}
119+
if (chunk.type === "finish" && (chunk.reason?.kind === "error" || chunk.reason?.kind === "aborted")) {
120+
writeAudit("error", "llm stream aborted or errored");
121+
return undefined;
122+
}
72123
}
124+
writeAudit("success", null);
73125
return text;
74126
})().catch((err) => {
75127
logger?.warn?.(`[dsh-mneme] entity extraction llm stream failed: ${String(err)}`);
128+
writeAudit("error", String(err?.message ?? err));
76129
return undefined;
77130
});
78131
};
@@ -445,7 +498,11 @@ export const apply = (ctx, config) => {
445498
const streamEntityText = createEntityStreamAdapter({
446499
llm: ctx.llm,
447500
agentDefaultModel: ctx.agentDefaultModel,
448-
logger: ctx.logger
501+
logger: ctx.logger,
502+
// Issue #250: the adapter needs service to write its llm_audit_logs row
503+
// (it never had it) and config for the shared llmAudit.enabled gate.
504+
service,
505+
config: cfg
449506
});
450507
service.setEntityExtractor((memory) =>
451508
extractEntities(memory, { store, config: cfg, callLLM: streamEntityText, logger: ctx.logger })

‎dsh-mneme/src/config.js‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -500,11 +500,18 @@ export const Config = z.object({
500500
}).default({}),
501501

502502
// --- LLM audit trail (Bug8) ------------------------------------------------
503-
// Records every background LLM call (autoDream consolidation + summary,
504-
// autoSummarize compression) into llm_audit_logs: tokens, duration, status
505-
// and which trigger produced it. Failures are recorded as status=error and
506-
// never block the feature. retentionDays bounds the table: older rows are
507-
// purged on boot.
503+
// Records every background LLM call into llm_audit_logs: tokens, duration,
504+
// status and which trigger produced it. Failures are recorded as
505+
// status='error' and never block the feature. retentionDays bounds the table:
506+
// older rows are purged on boot.
507+
//
508+
// Covered trigger_source values: autoDream (consolidation + summary),
509+
// autoSummarize (compression), sleep (conflict + pattern) and entityExtract
510+
// (issue #250 — the last two were structurally missing the hook, not a
511+
// deliberate narrowing). entityExtract runs on every memory write, so row
512+
// growth scales with write volume; retentionDays caps the ceiling but not the
513+
// rate. If that turns out too fast, sample by operation_type or give it a
514+
// separate retention — do not add a second gate: this flag is the single one.
508515
llmAudit: z.object({
509516
enabled: z.boolean().default(true),
510517
retentionDays: z.natural().min(1).max(3650).default(90)

0 commit comments

Comments
 (0)