Skip to content

Commit da19278

Browse files
committed
fix(organize): apply 只重放 dryRun 校验过的快照,回执与数据同事务(#231)
对照 CodeRabbit 复核逐条修: - input 改存规范化快照(保留原始索引 + payload 级 scope),apply 只认快照里的索引: 被 dryRun 跳过的候选没有资格落库,也不再出现「按 scope A 比对、写成全局 scope」。 - 提交后补重嵌入:service 注入 finalize(document.js 先例),事务里的 scheduleEmbed 被 txDepth 挡掉,不补这一步新写的记忆就没有向量。 - 成功回执写进同一个事务(回执写不进去 → 一起回滚);失败回执报 applied 0 与空 byId, 不再把已回滚的累积值写进审计。 - 写入期异常一律上抛(回滚 + failed):单条宽容只覆盖输入级错误,那一步在 normalize。 - dryRun 拒收 apply 回执当报告(靠 outcome.dry_run_id 区分)与 failed 报告;嵌入器抛错 时先留 failed 回执再上抛。 - organize(payload, opts) 不再丢 opts:hiddenIds 由可信调用方传入(document 的 hiddenEvidenceIds 同款契约),不取 payload 字段。 - 回归:新增 8 例(scope 随快照落地、越权索引、apply 回执链、抛错嵌入器、写失败回滚、 审计失败回滚、finalize 时机、hiddenIds)。
1 parent 1c7624b commit da19278

5 files changed

Lines changed: 442 additions & 182 deletions

File tree

‎dsh-mneme/lib/organize.js‎

Lines changed: 136 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -8,15 +8,20 @@
88
// 独立成模块(AGENTS.md 尺寸约定,同 document.js / recall-stats.js 先例):
99
// service.js 只做依赖注入 + barrel 出口,调用方零改动。
1010
//
11-
// 三条硬规则,都是「宁可什么都不做」的形态:
11+
// 四条硬规则,都是「宁可什么都不做」的形态:
1212
// 1. dryRun 不写记忆表:它只读 + 写一行审计(dream_runs)。比对是报告,不是动作。
13-
// 2. apply 必须引用一次真实 dryRun(run_id 且 run_type='organize')——没有比对过
14-
// 的候选一律不落库,堵死「跳过报告直接写」这条绕过路径。
15-
// 3. 筛除 = 归档(store.setArchived),绝不物理删除:归档可恢复,删除不可逆。
13+
// 2. apply 必须引用一次真实 dryRun——run_id 且 run_type='organize',且不是 apply
14+
// 自己的回执(靠 outcome.dry_run_id 区分)、也不是 failed 的报告。没有比对过的
15+
// 候选一律不落库,堵死「跳过报告直接写」这条绕过路径。
16+
// 3. apply 只重放 dryRun 校验过的那份规范化快照,且只认快照里的索引:被 dryRun
17+
// 跳过的候选没有资格落库;payload 级 scope 也随快照一起走,不会出现「报告按
18+
// scope A 比对、落地却写成全局」的错位。
19+
// 4. 筛除 = 归档(store.setArchived),绝不物理删除:归档可恢复,删除不可逆。
1620
//
17-
// 宽容形态(仓库红线 4,同 issue #89):单条非法候选/决策跳过 + 应用合法子集 +
18-
// run 记 degraded,不整单拒绝。审计口径复用 dream_runs 的 receipt 语义
19-
// (buildReceipt / parseReceipt 同一格式),不新建审计面。
21+
// 宽容形态(仓库红线 4,同 issue #89)只管**输入**:单条非法候选/决策跳过 + 应用
22+
// 合法子集 + run 记 degraded,不整单拒绝。基础设施错误(嵌入器抛错、存储异常)
23+
// 相反——记 failed 并原样上抛,绝不静默降级成「没找到重复」。审计口径复用
24+
// dream_runs 的 receipt 语义(buildReceipt / parseReceipt 同一格式),不新建审计面。
2025

2126
import { createHash, randomUUID } from "node:crypto";
2227
import { scopeKeyOf } from "./scope.js";
@@ -32,28 +37,33 @@ const CANDIDATE_LIMIT = 200;
3237
// 单次 dryRun 的候选硬上限:主场景是「长段工作收尾整理」,不是批量导入;不设界
3338
// 的话一次调用会把整库拉进比对。
3439
const MAX_ITEMS = 50;
40+
// scope 标注只做搬运(判定在 store / service 侧),省得整理接口成为第二条绕过
41+
// scope 治理的写入通道。载荷级给的值是候选级的缺省——比对按它判 scope,落地就必须
42+
// 带上它,否则报告与写入落在两个 scope 里。
43+
const SCOPE_KEYS = ["sensitivity", "agent_scope", "workspace_scope", "agent_scope_source", "workspace_scope_source"];
3544

3645
/**
3746
* Build the organizer.
3847
*
3948
* Injected deps are service.js closure members (store / embedQuery / saveWithDedupe
40-
* / transaction) so this module stays free of service-internal wiring; `saveWithDedupe`
41-
* is the normal write path, so its own epilogue (mirror sync / notify / re-embed)
42-
* is never duplicated here.
49+
* / transaction / finalize) so this module stays free of service-internal wiring;
50+
* `saveWithDedupe` is the normal write path, so its own epilogue (mirror sync and
51+
* notify) is never duplicated here. `finalize` is the post-commit re-embed hook
52+
* (document.js 同款): transaction 提交后补重嵌入,见 service.js › transaction 注释。
4353
*
44-
* @returns {{ organize: (payload: object) => Promise<object> }}
45-
* `organize({ mode: "dryRun"|"apply", ... })` — 一个入口带 mode 参数(维护者
46-
* 倾向的形态),内部两个具名函数各自可测。
54+
* @returns {{ organize: (payload: object, opts?: object) => Promise<object> }}
55+
* `organize({ mode: "dryRun"|"apply", ... }, opts)` — 一个入口带 mode 参数
56+
* (维护者倾向的形态),内部两个具名函数各自可测。
4757
*/
48-
export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction }) {
58+
export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction, finalize }) {
4959
const hashOf = (value) => createHash("sha256").update(JSON.stringify(value ?? null)).digest("hex");
5060

5161
/**
5262
* 候选规范化。返回 `{ candidate }` 或 `{ error }`——错误在 dryRun 里进 skipped,
5363
* 不抛整单(红线 4)。document 刻意在此拦下:它的唯一铸造口是 registerDocument
5464
* (#230),service 的通用 save 路径也会拒绝,这里提前拦截只是为了报告不撒谎。
5565
*/
56-
function normalizeCandidate(raw) {
66+
function normalizeCandidate(raw, defaults = {}) {
5767
const type = String(raw?.type ?? "").trim();
5868
const title = String(raw?.title ?? "").trim();
5969
const content = String(raw?.content ?? "").trim();
@@ -67,26 +77,24 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
6777
tags: [...new Set((Array.isArray(raw?.tags) ? raw.tags : []).map((t) => String(t ?? "").trim()).filter(Boolean))],
6878
source: String(raw?.source ?? "organize")
6979
};
70-
// scope 标注只做搬运(判定在 store / service 侧),省得整理接口成为第二条
71-
// 绕过 scope 治理的写入通道。
72-
for (const key of ["sensitivity", "agent_scope", "workspace_scope", "agent_scope_source", "workspace_scope_source"]) {
73-
if (raw?.[key] !== undefined) candidate[key] = raw[key];
80+
for (const key of SCOPE_KEYS) {
81+
const value = raw?.[key] ?? defaults[key];
82+
if (value !== undefined) candidate[key] = value;
7483
}
7584
return { candidate };
7685
}
7786

7887
/** 标题归一(同 findSessionDuplicate 的 title 档):大小写与标点不构成「不同标题」。 */
7988
const normalizeTitle = (s) => String(s ?? "").toLowerCase().replace(/[\s\p{P}]+/gu, "");
8089

81-
/** 向量档最佳命中;拿不到向量/嵌入器时返回 null(无信号 = 不判定,不是报错)。 */
90+
/**
91+
* 向量档最佳命中。拿不到向量/嵌入器时返回 null(无信号 = 不判定,不是报错);
92+
* 但**抛出的**错误不再吞掉——嵌入器故障下的空信号会被读成「全是新条目」,让
93+
* agent 把重复条写进库,比整单失败更糟,交 dryRun 记 failed 后上抛。
94+
*/
8295
async function bestNear(candidate, rows, vecs) {
8396
if (!vecs?.size) return null;
84-
let probe = null;
85-
try {
86-
probe = await embedQuery([candidate.title, candidate.content].filter(Boolean).join("\n"));
87-
} catch {
88-
return null; // 嵌入器故障只降级比对,不阻塞报告
89-
}
97+
const probe = await embedQuery([candidate.title, candidate.content].filter(Boolean).join("\n"));
9098
if (!probe) return null;
9199
let best = null;
92100
for (const m of rows) {
@@ -111,14 +119,18 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
111119
const raws = Array.isArray(payload.candidates) ? payload.candidates : null;
112120
if (!raws) throw new Error("organize.dryRun: candidates must be an array");
113121
const bounded = raws.slice(0, MAX_ITEMS);
122+
const defaults = Object.fromEntries(SCOPE_KEYS.filter((k) => payload?.[k] !== undefined).map((k) => [k, payload[k]]));
114123

115124
const skipped = [];
116125
const accepted = [];
117126
bounded.forEach((raw, index) => {
118-
const { candidate, error } = normalizeCandidate(raw);
127+
const { candidate, error } = normalizeCandidate(raw, defaults);
119128
if (error) skipped.push({ index, error });
120129
else accepted.push({ index, candidate });
121130
});
131+
// 审计存规范化快照(带原始索引),不存调用方原始输入:apply 要重放的就是这份
132+
// ——原始输入里混着被跳过的候选,也没有随行的 payload 级 scope。
133+
const snapshot = accepted.map(({ index, candidate }) => ({ index, ...candidate }));
122134

123135
// scope 同等才比对(跨 scope 永不互判——防泄漏,与 saveWithDedupe 的去重键
124136
// 「type+title+scope」同口径)。行扫描与向量读取按 type 缓存:一批候选通常
@@ -142,20 +154,49 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
142154
return vecsByType.get(type);
143155
};
144156

157+
const runId = randomUUID();
158+
const createdAt = new Date().toISOString();
159+
const snapshotHash = hashOf(bounded);
160+
const record = (status, { error = null, decisions = items, skipped: skipList = skipped, applied = 0 } = {}) =>
161+
store.saveDreamRun({
162+
id: runId,
163+
created_at: createdAt,
164+
status,
165+
error,
166+
provider: null,
167+
model: null,
168+
snapshot_hash: snapshotHash,
169+
input_count: bounded.length,
170+
input: snapshot,
171+
decisions,
172+
applied,
173+
summary_stored: 0,
174+
receipt: buildReceipt({ runId, status, snapshotHash, inputCount: bounded.length, applied, summaryStored: false }),
175+
policy_epoch: 0,
176+
run_type: "organize",
177+
...(skipList.length ? { skipped: skipList } : {})
178+
});
179+
145180
const items = [];
146-
for (const { index, candidate } of accepted) {
147-
const rows = rowsFor(candidate.type);
148-
let item = { index, type: candidate.type, title: candidate.title, verdict: "new", match: null };
149-
const exact = rows.find((m) => normalizeTitle(m.title) === normalizeTitle(candidate.title));
150-
if (exact) {
151-
item = { ...item, verdict: "exact", match: { id: exact.id, title: exact.title, sim: null } };
152-
} else {
153-
const near = await bestNear(candidate, rows, vecsFor(candidate.type, rows));
154-
if (near) {
155-
item = { ...item, verdict: "near", match: { id: near.memory.id, title: near.memory.title, sim: near.sim } };
181+
try {
182+
for (const { index, candidate } of accepted) {
183+
const rows = rowsFor(candidate.type);
184+
let item = { index, type: candidate.type, title: candidate.title, verdict: "new", match: null };
185+
const exact = rows.find((m) => normalizeTitle(m.title) === normalizeTitle(candidate.title));
186+
if (exact) {
187+
item = { ...item, verdict: "exact", match: { id: exact.id, title: exact.title, sim: null } };
188+
} else {
189+
const near = await bestNear(candidate, rows, vecsFor(candidate.type, rows));
190+
if (near) {
191+
item = { ...item, verdict: "near", match: { id: near.memory.id, title: near.memory.title, sim: near.sim } };
192+
}
156193
}
194+
items.push(item);
157195
}
158-
items.push(item);
196+
} catch (e) {
197+
// 审计先留痕、再上抛:一份没有回执的失败报告,事后无法与「没调用过」区分。
198+
record("failed", { error: String(e?.message ?? e) });
199+
throw e;
159200
}
160201

161202
const counts = {
@@ -168,26 +209,7 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
168209
};
169210
// 有被拒的候选 → degraded(如实标记),而不是 ok 掩盖过去;一条都没报 → noop。
170211
const status = skipped.length ? "degraded" : items.length ? "ok" : "noop";
171-
const runId = randomUUID();
172-
const createdAt = new Date().toISOString();
173-
const snapshotHash = hashOf(bounded);
174-
store.saveDreamRun({
175-
id: runId,
176-
created_at: createdAt,
177-
status,
178-
provider: null,
179-
model: null,
180-
snapshot_hash: snapshotHash,
181-
input_count: bounded.length,
182-
input: bounded,
183-
decisions: items,
184-
applied: 0,
185-
summary_stored: 0,
186-
receipt: buildReceipt({ runId, status, snapshotHash, inputCount: bounded.length, applied: 0, summaryStored: false }),
187-
policy_epoch: 0,
188-
run_type: "organize",
189-
...(skipped.length ? { skipped } : {})
190-
});
212+
record(status);
191213
return { run_id: runId, created_at: createdAt, status, counts, items, skipped };
192214
}
193215

@@ -199,7 +221,9 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
199221
*
200222
* @param {object} payload `{ run_id, decisions }`
201223
* @param {object} [opts] `{ hiddenIds }` — 调用者 scope 看不见的行按「不存在」
202-
* 处理(#170 复核项 4:无存在性泄漏),与 document 的 hiddenEvidenceIds 同形。
224+
* 处理(#170 复核项 4:无存在性泄漏),与 document 的 hiddenEvidenceIds 同形:
225+
* **可见性由可信调用方算好传入**(tools 层先例:register_document),不取 payload
226+
* 里的字段——那是调用方可自选的内容,不是授权。
203227
* @returns {{run_id, dry_run_id, status, saved, discarded, archived, memory_ids, skipped, degraded}}
204228
*/
205229
function apply(payload = {}, { hiddenIds = [] } = {}) {
@@ -211,14 +235,25 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
211235
if (!report || report.run_type !== "organize") {
212236
throw new Error(`organize.apply: unknown organize run "${dryRunId}"`);
213237
}
238+
// apply 回执与 dryRun 报告共用 run_type='organize':只有 outcome.dry_run_id 能
239+
// 把它们分开。不查这一条,apply 的 run_id 可以被当成 dry_run_id,串成
240+
// apply → apply 的审计链,报告那一环就此消失。
241+
if (report.outcome?.dry_run_id) {
242+
throw new Error(`organize.apply: "${dryRunId}" is an apply receipt, not a dryRun report`);
243+
}
244+
if (report.status === "failed") {
245+
throw new Error(`organize.apply: dryRun "${dryRunId}" failed — re-run dryRun before applying`);
246+
}
214247
const decisions = Array.isArray(payload?.decisions) ? payload.decisions : null;
215248
if (!decisions) throw new Error("organize.apply: decisions must be an array");
216249

217-
const input = Array.isArray(report.input) ? report.input : [];
250+
const snapshot = Array.isArray(report.input) ? report.input : [];
251+
const byIndex = new Map(snapshot.map((entry) => [Number(entry?.index), entry]));
218252
const hidden = new Set((Array.isArray(hiddenIds) ? hiddenIds : []).map((id) => String(id)));
219253
const skipped = [];
220254
const byId = {};
221255
const memoryIds = [];
256+
const savedRows = [];
222257
let saved = 0;
223258
let discarded = 0;
224259
let archived = 0;
@@ -228,20 +263,21 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
228263
const action = String(decision?.action ?? "").trim();
229264
if (action === "save") {
230265
const index = Number(decision?.index);
231-
const candidate = Number.isInteger(index) ? input[index] : undefined;
232-
if (!candidate) {
233-
skipped.push({ index: i, action, error: "candidate index is out of range" });
266+
const entry = Number.isInteger(index) ? byIndex.get(index) : undefined;
267+
if (!entry) {
268+
// 索引不在快照里 = dryRun 从没校验过它(越界,或当时就被 normalize 拒了)。
269+
skipped.push({ index: i, action, error: "candidate index was not accepted by this dryRun" });
234270
return;
235271
}
236-
try {
237-
const result = saveWithDedupe(candidate);
238-
saved++;
239-
memoryIds.push(result.memory.id);
240-
byId[result.memory.id] = action;
241-
} catch (e) {
242-
// 单条失败不拖垮整批(红线 4);原因进 skipped 明细,run 记 degraded。
243-
skipped.push({ index: i, action, error: String(e?.message ?? e) });
244-
}
272+
const { index: _originalIndex, ...candidate } = entry;
273+
// 这里不吞异常:能走到这一步的候选是 dryRun 已经规范化过的合法输入,抛出
274+
// 来的只能是基础设施错误 → 让它冒到事务外,整批回滚 + failed(红线 4 的
275+
// 「单条跳过」只覆盖输入级错误,那一步在 normalizeCandidate)。
276+
const result = saveWithDedupe(candidate);
277+
saved++;
278+
savedRows.push(result.memory);
279+
memoryIds.push(result.memory.id);
280+
byId[result.memory.id] = action;
245281
return;
246282
}
247283
if (action === "discard") {
@@ -268,50 +304,61 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
268304
});
269305
};
270306

271-
// 批量写在一个事务里:要么整批可见,要么全不可见(不留半批)。单条业务失败
272-
// 已在上面吞掉,所以这里抛出的都是基础设施级错误 → failed + 原样上抛。
273307
const applyRunId = randomUUID();
274308
const createdAt = new Date().toISOString();
275309
const snapshotHash = report.snapshot_hash;
276-
const record = (status, error) => store.saveDreamRun({
310+
// 回执的 applied/byId 由调用方传入:失败路径必须报 0(事务已回滚,累积的
311+
// saved/archived 是尝试值,写进回执就是撒谎)。
312+
const record = (status, error, { byId: outcomeById = byId, applied = saved + archived } = {}) => store.saveDreamRun({
277313
id: applyRunId,
278314
created_at: createdAt,
279315
status,
280316
error,
281317
provider: null,
282318
model: null,
283319
snapshot_hash: snapshotHash,
284-
input_count: input.length,
285-
input,
320+
input_count: snapshot.length,
321+
input: snapshot,
286322
decisions,
287323
// dry_run_id 是这条回执与它所指报告的连接键:审计能还原「报告 → 判断 → 落地」
288324
// 三步,而不用新建审计面。
289-
outcome: { dry_run_id: dryRunId, byId },
290-
applied: saved + archived,
325+
outcome: { dry_run_id: dryRunId, byId: outcomeById },
326+
applied,
291327
summary_stored: 0,
292328
receipt: buildReceipt({
293329
runId: applyRunId,
294330
status,
295331
snapshotHash,
296-
inputCount: input.length,
297-
applied: saved + archived,
332+
inputCount: snapshot.length,
333+
applied,
298334
summaryStored: false
299335
}),
300336
policy_epoch: 0,
301337
run_type: "organize",
302338
...(skipped.length ? { skipped } : {})
303339
});
304340

341+
let status;
305342
try {
306-
transaction(runDecisions);
343+
transaction(() => {
344+
runDecisions();
345+
// 只有 discard 的批次什么都没改 → noop(绝不虚报 ok)。
346+
status = skipped.length ? "degraded" : saved + archived > 0 ? "ok" : "noop";
347+
// 成功回执与数据同事务:审计写不进去就一起回滚,不留「库改了却没有回执」
348+
// 的窗口(回执是这条链上唯一的证据)。
349+
record(status, null);
350+
});
307351
} catch (e) {
308-
record("failed", String(e?.message ?? e));
352+
// 事务已回滚 → 失败回执单独写(它本身失败就让它抛,不掩盖原始错误)。
353+
record("failed", String(e?.message ?? e), { byId: {}, applied: 0 });
309354
throw e;
310355
}
311356

312-
// 只有 discard 的批次什么都没改 → noop(绝不虚报 ok)。
313-
const status = skipped.length ? "degraded" : saved + archived > 0 ? "ok" : "noop";
314-
record(status, null);
357+
// transaction 只负责镜像与通知,重嵌入留给调用方(service.js › transaction
358+
// 注释);document.js 用同一个 finalize 先例。放在提交之后,事务里 scheduleEmbed
359+
// 会被 txDepth 挡掉,不补这一步新写的记忆就没有向量。
360+
if (savedRows.length && typeof finalize === "function") finalize(savedRows);
361+
315362
return {
316363
run_id: applyRunId,
317364
dry_run_id: dryRunId,
@@ -325,10 +372,11 @@ export function createOrganizer({ store, embedQuery, saveWithDedupe, transaction
325372
};
326373
}
327374

328-
async function organize(payload = {}) {
375+
/** 一个入口带 mode 参数(维护者口径);opts 是可信调用上下文,不取 payload 里的字段。 */
376+
async function organize(payload = {}, opts = {}) {
329377
const mode = String(payload?.mode ?? "dryRun").trim();
330378
if (mode === "dryRun") return dryRun(payload);
331-
if (mode === "apply") return apply(payload);
379+
if (mode === "apply") return apply(payload, opts);
332380
throw new Error(`organize: unknown mode "${mode}" (dryRun | apply)`);
333381
}
334382

0 commit comments

Comments
 (0)