Skip to content

Commit 7fbd60f

Browse files
committed
feat(sleep): auto-resolve near-duplicate pairs and stop the demotion ledger from lying
sleep: a new opt-in tier (conflictAutoResolveEnabled, default off) adjudicates same-scope pairs at or above conflictAutoResolveMinSimilarity (default 0.95) instead of parking them for manual review. The newer side wins; the older one is superseded, so the traceability note lands on the archived copy and the survivor keeps its body untouched. Cross-scope pairs are never auto-adjudicated - ownership is the user's call (#170) - and a failed adjudication falls back to the manual queue rather than disappearing. trustEpistemicWeighting is forced off on this path: letting it swap sides would make "keep the newer one" drift with the data. Measured on the live queue before choosing 0.95: of 345 historical rows carrying a similarity, 20 (5.8%) sit at or above 0.95 and 28 (8.1%) at or above 0.94. The tier cuts the peak, it does not empty the queue - the threshold stays configurable so it can be tuned against the actual distribution instead of a guess. demotion: phaseDemotion pushed every candidate into demoted without reading what demoteToSummary returned, and demoteToSummary returns undefined for a row that was already demoted, no longer exists, or got touched after the snapshot. The ledger therefore re-reported the same rows every run (43%-99% overstatement across twelve measured runs, one run reporting 81 rows where all 81 were repeats). Already demoted rows are now skipped before the decision chain, and a row the freshness guard spares is reported as demoteSkipped instead of demoted. config/docs: both keys are whitelisted for the panel and documented in CONFIGURATION.md.
1 parent 62606bc commit 7fbd60f

10 files changed

Lines changed: 381 additions & 13 deletions

File tree

‎dsh-mneme/docs/CONFIGURATION.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -236,6 +236,8 @@
236236
| `reflectionFailureRetentionDays` | `90` | `failure_memories`(失败/纠正记录)的启动期清理保留天数(1–3650) | 只读 bundle 配置:未进 feature_flags 白名单,面板改不了。`dsh-mneme reclaim` 不读这个键(它对三个表用同一条 `--older-than`,本表不在其中)。此前这个 90 天是写死的(#275 伞下的欠账) |
237237
| `conflictFreezeEnabled` | `false` | 冲突不自动合并、冻结待人工复核 | opt-in |
238238
| `conflictFreezeMaxPending` | `100` | 冻结队列上限(≤1000) | — |
239+
| `conflictAutoResolveEnabled` | `false` | 相似度极高的**同 scope** 重复对自动裁决(保留新条、归档旧条),不进人工队列 | opt-in。会归档记忆,所以默认关;跨 scope 对永不自动(归属是用户的决定,#170)。用 supersede 而非 conflict:注记写在被归档侧,幸存记忆正文不动 |
240+
| `conflictAutoResolveMinSimilarity` | `0.95` | 自动裁决的相似度门槛(0.85–1) | 实测 live 库 345 条有相似度的历史行里 ≥0.95 只占 5.8%、≥0.94 占 8.1%——这一档减的是峰值,不是把队列清零。低于 0.85 就不是「重复」而是「相似」,不该自动归档任何一条 |
239241

240242
## 实体
241243

‎dsh-mneme/lib/config.js‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -525,6 +525,15 @@ export const Config = z.object({
525525
conflictFreezeEnabled: z.boolean().default(false),
526526
// Maximum number of frozen conflicts to keep pending for manual review.
527527
conflictFreezeMaxPending: z.natural().min(1).max(1000).default(100),
528+
// Issue #603 第 4 层(自动裁决档):freeze 模式里相似度极高的同 scope 对直接
529+
// 裁决(保留新条、归档旧条),不进人工队列——人只看模糊地带。默认关:这是
530+
// 会归档记忆的自动动作,必须显式打开。
531+
// 实测口径(live 库 345 条有相似度的历史行):>=0.95 只有 20 条(5.8%),
532+
// >=0.94 是 28 条(8.1%)。所以这一档减量约一半的峰值,减不到个位数;
533+
// 阈值留在配置里就是为了让人按自己队列的分布调,而不是信一个拍脑袋的数。
534+
// 跨 scope 对永不自动裁决:归属是用户的决定(v0.8.1 #170 的既有约束)。
535+
conflictAutoResolveEnabled: z.boolean().default(false),
536+
conflictAutoResolveMinSimilarity: z.number().min(0.85).max(1).default(0.95),
528537

529538
// --- entity gene (v0.3.0) -----------------------------------------------
530539
// Opt-in: when false (default) nothing in the pipeline extracts entities.

‎dsh-mneme/lib/dream/sleep.js‎

Lines changed: 100 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -442,9 +442,87 @@ async function phaseConflicts(ctx, service, config, logger, runId, semantic = nu
442442
// 只报「有位置但被每轮上限截住」的那部分:队列满时再追一轮也没用,
443443
// 报成 backlog 会让调度器空转(catchup 上限 24 轮)。
444444
const perRunSkipped = withinRoom.length - pool.length;
445+
// Issue #603 第 4 层(自动裁决档,默认关):相似度极高的**同 scope** 对不再
446+
// 进人工队列,直接保留新条、归档旧条。人只看模糊地带。
447+
//
448+
// 为什么阈值留在配置里而不是写死 0.95:live 库 345 条有相似度的历史行里,
449+
// >=0.95 只有 20 条(5.8%)、>=0.94 是 28 条(8.1%)。这一档减的是峰值,
450+
// 不是把队列清零——把阈值当常量会让人以为「开了就没事了」。
451+
//
452+
// 为什么跨 scope 永不自动:归属是用户的决定(v0.8.1 #170),归档败者可能
453+
// 销毁某归属下的唯一副本。这条约束比减负重要。
454+
//
455+
// 为什么用 supersede 而不是 conflict:conflict 会把「(已否决旧信息:…)」
456+
// 追加到**赢家**正文,而赢家正是留下来的那条;近重复对再叠一段互相引用的
457+
// 注记,等于用冗余污染幸存记忆。supersede 把可追溯性留在被归档的一侧,
458+
// 赢家正文一字不动(#126 的原设计)。
459+
const autoEnabled = config.conflictAutoResolveEnabled === true;
460+
const autoMin = typeof config.conflictAutoResolveMinSimilarity === "number"
461+
? config.conflictAutoResolveMinSimilarity
462+
: 0.95;
463+
const autoPool = [];
464+
const parkPool = [];
465+
for (const p of pool) {
466+
const eligible = autoEnabled
467+
&& !isCrossScopePair(p)
468+
&& typeof p.similarity === "number"
469+
&& p.similarity >= autoMin;
470+
(eligible ? autoPool : parkPool).push(p);
471+
}
472+
let autoResolved = 0;
473+
let autoFailed = 0;
474+
// 这一档不走 LLM,没有批次时间戳可借(judgedAt 在下面那条 LLM 通路里才定义),
475+
// 自己取一个:台账要能回答「什么时候判的」,不能留空。
476+
const autoJudgedAt = new Date().toISOString();
477+
// 保留较新的那条:updated_at 相等时用 id 兜底,保证同一对每次判出同一个
478+
// 赢家(否则重放会来回翻,归档的可能是刚刚保住的那条)。
479+
const newerFirst = (p) => {
480+
const au = p.a.updated_at ?? "";
481+
const bu = p.b.updated_at ?? "";
482+
if (au !== bu) return au > bu ? [p.a, p.b] : [p.b, p.a];
483+
return p.a.id > p.b.id ? [p.a, p.b] : [p.b, p.a];
484+
};
485+
for (const p of autoPool) {
486+
const [winner, loser] = newerFirst(p);
487+
const snapshot = new Map();
488+
for (const m of [winner, loser]) {
489+
snapshot.set(m.id, { updated_at: m.updated_at, content: m.content, title: m.title });
490+
}
491+
// trustEpistemicWeighting 显式关掉:它会在 applySupersede 里按 epistemic_status
492+
// 交换赢家,那会让「保留新条」这条契约随数据漂移。自动档的承诺必须是确定的。
493+
const { applied, conflicts } = applyDecisions(
494+
[{ action: "supersede", winner: winner.id, loser: loser.id }],
495+
service, logger, snapshot, { ...config, trustEpistemicWeighting: false }
496+
);
497+
if (applied > 0) {
498+
autoResolved++;
499+
// 记进机器侧台账:自动动作也要能事后回答「这一对什么时候、被谁判的」。
500+
try {
501+
service.saveConflictReviewed?.({
502+
run_id: runId, memory_a: p.a.id, memory_b: p.b.id,
503+
outcome: "supersede", judged_at: autoJudgedAt
504+
});
505+
} catch (error) {
506+
logger?.warn?.(`dsh-mneme sleep: failed to record auto-resolved pair ${p.a.id}/${p.b.id}: ${String(error)}`);
507+
}
508+
continue;
509+
}
510+
// 自动档没落地 → 退回人工队列。绝不因为「本来想自动处理」就把这一对丢掉,
511+
// 那才是真正的静默丢失。两种没落地的原因都要说清楚:
512+
// · CAS 冲突(applyDecisions 记在 conflicts 里)——两侧在快照之后被改过,
513+
// 自动判的赢家可能已经不是该保的那条,人看一眼最稳;
514+
// · 其余异常(记在 failures 里)——写库失败之类,重试无意义就交给人。
515+
autoFailed++;
516+
logger?.warn?.(
517+
`dsh-mneme sleep: auto-resolve skipped ${p.a.id}/${p.b.id}`
518+
+ (conflicts.length ? ` (concurrent edit)` : "")
519+
+ `, falling back to manual review`
520+
);
521+
parkPool.push(p);
522+
}
445523
let frozen = 0;
446524
let parkFailed = 0;
447-
for (const p of pool) {
525+
for (const p of parkPool) {
448526
const outcome = parkPair(p, isCrossScopePair(p) ? scopeReason(p) : simReason(p));
449527
if (outcome === "fresh") frozen++;
450528
else if (outcome === "failed") parkFailed++;
@@ -453,8 +531,12 @@ async function phaseConflicts(ctx, service, config, logger, runId, semantic = nu
453531
logger?.warn?.(`dsh-mneme sleep: conflict freeze queue full (${pendingNow}/${maxPending}), skipped ${pendingSkipped} conflict(s)`);
454532
}
455533
return {
456-
status: frozen > 0 ? "ok" : "noop",
534+
status: frozen > 0 || autoResolved > 0 ? "ok" : "noop",
457535
frozen,
536+
// 自动裁决掉的不进队列,但确实是这一轮干的活——分开报,别混进 frozen
537+
// (frozen 的语义是「给人工队列新增了几行」,混进去面板就会虚高)。
538+
autoResolved,
539+
autoFailed,
458540
// 写库抛错的对既没入队也不算落地:clean=false,别让调度器当「追完了」。
459541
clean: parkFailed === 0,
460542
scopeCandidates: crossPairs.length,
@@ -794,6 +876,9 @@ function phaseDemotion(service, config, logger, runId, signal = null) {
794876
const compressCut = Date.now() - compressDays * 86400000;
795877
const demoted = [];
796878
const archived = [];
879+
// 被判定为「该降级」但 demoteToSummary 实际没动手的条数。单独报出来:
880+
// 它正是「台账说降级了 N 条、实际一条没变」那个缺口,藏起来就等于没修。
881+
let demoteSkipped = 0;
797882
// 政策轴在这里尤其要紧:下面两行会真改归档面(setArchived / demoteToSummary)。
798883
// 把过滤放在循环之外、而不是在循环体里 continue:一旦将来有人在循环体里加
799884
// 副作用(现已有),漏掉那条 continue 就是静默改到被按下的行。
@@ -803,6 +888,10 @@ function phaseDemotion(service, config, logger, runId, signal = null) {
803888
for (const m of withoutPaused(service, service.all().filter((m) => !isImportedRow(m)))) {
804889
if (signal?.aborted) break;
805890
if (m.archived || m.forgotten) continue;
891+
// 已降级的行不必再进判定链:demoteToSummary 见到 _full_content 会直接返回,
892+
// 但这行每轮仍会被选中、空转一次事务,再被下面无条件 push 进 demoted。
893+
// 提前挡掉既省事务,也让「这一轮到底动了什么」从源头算得准(#603 台账)。
894+
if (m._full_content) continue;
806895
const ref = m.last_accessed_at ?? m.created_at;
807896
if (!ref) continue;
808897
const t = new Date(ref).getTime();
@@ -825,14 +914,20 @@ function phaseDemotion(service, config, logger, runId, signal = null) {
825914
} else if (t < archiveCut) {
826915
// minRefTimeMs re-checks freshness inside demoteToSummary's transaction:
827916
// a recall touch landing after this snapshot must not demote the memory.
828-
service.demoteToSummary(m.id, makeSummary(m), { minRefTimeMs: archiveCut });
829-
demoted.push(m.id);
917+
//
918+
// 返回值必须看:demoteToSummary 在三种情况下返回 undefined(行不存在、
919+
// 已降级、快照后被 recall 摸热),而这里原来无条件 push,于是台账把没发生
920+
// 的降级记成发生了——live 库实测 12 轮里虚报 43%–99%(#603)。
921+
const updated = service.demoteToSummary(m.id, makeSummary(m), { minRefTimeMs: archiveCut });
922+
if (updated) demoted.push(m.id);
923+
else demoteSkipped++;
830924
}
831925
}
832926
return {
833927
status: demoted.length || archived.length ? "ok" : "noop",
834928
demoted,
835-
archived
929+
archived,
930+
demoteSkipped
836931
};
837932
}
838933

‎dsh-mneme/lib/settings.js‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,11 @@ const FEATURE_FLAG_BOOLEANS = [
159159
"reflectionFailureTracking",
160160
"bm25SearchEnabled",
161161
"conflictFreezeEnabled",
162+
// Issue #603 第 4 层:自动裁决档(默认关)。相似度极高的同 scope 重复对直接
163+
// 保留新条、归档旧条,不进人工队列;模糊地带照旧留给人。会归档记忆,所以必须
164+
// 显式打开,面板可启停。阈值见 FEATURE_FLAG_NUMBER_RANGES 的
165+
// conflictAutoResolveMinSimilarity。
166+
"conflictAutoResolveEnabled",
162167
"trustEpistemicWeighting",
163168
// Plan #2: attach per-source {keyword, vector, bm25, final} signals to each
164169
// search result for transparency/debugging. Default off, purely decorative.
@@ -319,7 +324,10 @@ const FEATURE_FLAG_NUMBER_RANGES = {
319324
// Issue #125:hybrid 判"高相似"的阈值。
320325
dreamCandidateMinSim: [0.5, 0.99],
321326
// Issue #24 块2:单次有效触达的边权抬升幅度(与 config.js 的闭区间对齐)。
322-
graphWeightDelta: [0, 1]
327+
graphWeightDelta: [0, 1],
328+
// Issue #603 第 4 层:自动裁决的相似度门槛。下界 0.85 与冲突检测阈值同源——
329+
// 低于它就不是「重复」而是「相似」,不该自动归档任何一条。
330+
conflictAutoResolveMinSimilarity: [0.85, 1]
323331
};
324332
// 自由字符串开关(与 config.js 的 z.string() 同名同型):trim 后 ≤200 字符,
325333
// 空串合法(= 跟随主对话模型/默认路径,面板显示 placeholder)。

‎dsh-mneme/src/config.js‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -525,6 +525,15 @@ export const Config = z.object({
525525
conflictFreezeEnabled: z.boolean().default(false),
526526
// Maximum number of frozen conflicts to keep pending for manual review.
527527
conflictFreezeMaxPending: z.natural().min(1).max(1000).default(100),
528+
// Issue #603 第 4 层(自动裁决档):freeze 模式里相似度极高的同 scope 对直接
529+
// 裁决(保留新条、归档旧条),不进人工队列——人只看模糊地带。默认关:这是
530+
// 会归档记忆的自动动作,必须显式打开。
531+
// 实测口径(live 库 345 条有相似度的历史行):>=0.95 只有 20 条(5.8%),
532+
// >=0.94 是 28 条(8.1%)。所以这一档减量约一半的峰值,减不到个位数;
533+
// 阈值留在配置里就是为了让人按自己队列的分布调,而不是信一个拍脑袋的数。
534+
// 跨 scope 对永不自动裁决:归属是用户的决定(v0.8.1 #170 的既有约束)。
535+
conflictAutoResolveEnabled: z.boolean().default(false),
536+
conflictAutoResolveMinSimilarity: z.number().min(0.85).max(1).default(0.95),
528537

529538
// --- entity gene (v0.3.0) -----------------------------------------------
530539
// Opt-in: when false (default) nothing in the pipeline extracts entities.

0 commit comments

Comments
 (0)