Add native dreaming sweep engine#898
Conversation
rwmjhb
left a comment
There was a problem hiding this comment.
Requesting changes. The feature direction may be useful, but this implementation currently mutates memory state in ways that do not line up with existing retrieval/governance behavior.
Must fix
Light-phase archived duplicates can still be recalled.
The dreaming engine archives duplicates by patching metadata to state: "archived" and memory_layer: "archive", but normal retrieval/search still only uses active/expiry filtering. The store excludeInactive path delegates to isMemoryActiveAt, which checks temporal validity such as valid_from / invalidated_at, not state: archived or memory_layer: archive. Manual memory_recall and OpenClaw capability search can therefore still return a duplicate after the light phase has supposedly archived it.
Please either make retrieval/store inactive filtering exclude archived/archive-layer memories consistently, or have the dreaming engine set the same invalidation metadata that existing retrieval already respects. Add a regression test that archives a duplicate through the light phase and then proves retriever/search no longer returns it.
Also worth addressing before this grows further
- Default deep promotion requires unique-query metadata, but existing recall/access writers only maintain
access_count/last_accessed_at; normal recalled memories will not satisfy the defaultminUniqueQueries=3gate. dreaming_phaseis used as an audit marker on real memories, butisDreamingGenerated()treats anydreaming_phaseas generated output, causing real memories touched by light/deep phases to be skipped in later sweeps.- Source filters advertised in the manifest (
daily,sessions,recall,logs,memory,deep) are compared directly to storedmetadata.source/ category values, so configured filters may silently match nothing. - The scheduler can reschedule after service shutdown if
stophappens while a sweep is in flight;DreamingEngine.stop()is currently a no-op. - Unsupported cron/frequency strings silently fall back to daily scheduling instead of failing closed or warning clearly.
Given the scope drift from the original unsupported-setting issue, I would also prefer splitting capability/config compatibility from automatic archive/promotion/reflection mutation unless maintainers explicitly want the full native sweep engine in this PR.
|
@rwmjhb Thanks, agreed on the scope concern. I pushed a narrow fix for the blocking issue: light-phase duplicate archival now sets For the remaining bullets, I agree those should not keep expanding this PR unless maintainers explicitly want the full native sweep engine here. My preference is to split this back down to the original capability/config compatibility work, then handle the native archive/promotion/reflection engine separately with the unique-query metadata, source-filter mapping, scheduler shutdown, and invalid-frequency behavior designed together. Happy to trim/split this PR in that direction if that’s what you prefer. |
…reaming-engine # Conflicts: # scripts/ci-test-manifest.mjs
rwmjhb
left a comment
There was a problem hiding this comment.
Requesting changes on the latest head. Build and focused tests are clean, but the implementation still has several correctness issues in the enabled dreaming mutation path.
Independent verification on e3fea38 passed:
npm run build --if-presentnode --test test/dreaming-engine.test.mjsnode test/plugin-manifest-regression.mjsnode test/cli-smoke.mjs- generated/source files remained clean after build
Must fix
-
Dreaming can reschedule after service shutdown. The scheduled callback clears
dreamingTimer, awaitsrunDreamingSweep(), then schedules the next sweep unconditionally.stop()only clears an existing timer, andDreamingEngine.stop()is effectively a no-op. If shutdown happens while a sweep is in flight, the callback can schedule new background work after plugin stop/reload and continue mutating memory. Please add a stopped/abort state and check it before and after the awaited sweep. -
Manifest source filters do not map to the plugin's stored source values.
entryMatchesSources()compares configured values directly tometadata.sourceor category, while the manifest advertises values such asdaily,sessions,recall,logs,memory, anddeep. The stored normalized sources are values likemanual,auto-capture,reflection,dreaming-engine,session-summary, andlegacy. A valid advertised filter can therefore silently match zero memories. Please map these schema values explicitly or reject unsupported values during config normalization. -
The default deep promotion gate is not fed by normal recall/access paths. Deep promotion defaults to
minUniqueQueries = 3, but normal recall/access updatesaccess_countandlast_accessed_at, not unique-query metadata. The current test promotes by pre-seedingunique_query_count, so the default real path can no-op despite repeated recalls. Please either record bounded unique query metadata during recall or make the default gate use metadata the plugin already writes.
Also worth fixing
- Light dedupe limits candidates before applying the recent lookback window, so older rows can fill the limited batch and recent duplicates can be missed.
- Invalid/unsupported dreaming frequency silently falls back to a 24-hour schedule for a mutating feature.
- Deep promotion updates metadata and importance in separate writes; if the second write fails, the memory can be left partially promoted.
Given this feature archives/promotes/creates memories, I do not think these should ship as follow-ups.
rwmjhb
left a comment
There was a problem hiding this comment.
Requesting changes on the latest head. The shutdown/source-filter/deep-promotion fixes are moving in the right direction, and build/focused tests are clean, but there is still a lifecycle blocker.
Independent verification on fb63742 passed:
npm run build --if-presentnode --test test/dreaming-engine.test.mjsnode test/plugin-manifest-regression.mjsnode test/cli-smoke.mjs- build left source/dist/package/manifest/test files clean
Must fix
Dreaming cannot restart after service stop. The service stop path calls dreamingEngine.stop(), which sets the engine-local stopped flag. runSweep() then returns immediately whenever that flag is true. On a later service start, the outer scheduler flag is reset, but the engine-local stopped flag is never cleared. The result is that timers can be scheduled while every sweep silently no-ops until the plugin state is recreated.
Please make the engine lifecycle reversible: add a start()/reset() path that clears the engine stop state on service start, or recreate the engine when starting after a stop. Add a regression test for stop -> start -> sweep on the same registered service instance.
Also worth fixing
- Stop still does not abort mutations already inside the current light/deep/REM phase; it only prevents later phase starts.
- The advertised
sessionssource remains unreliable: session-summary rows can keep the legacy source when no explicit source is patched, while raw session-summary metadata can default to archived and be filtered out. - The light sweep path can materialize and sort the full scoped memory table before applying phase limits.
fetchForCompaction()ordering now changes existing compaction selection behavior; that may be intended, but it should be explicit because it affects non-dreaming compaction too.
|
Fixed the restart lifecycle issue. The dreaming engine now has a reversible start path, and service start clears the engine-local stopped state before scheduling sweeps.\n\nI added a stop -> start -> sweep regression on the same engine instance and reran:\n- npm run build --if-present\n- node --test test/dreaming-engine.test.mjs\n- node test/plugin-manifest-regression.mjs\n- node test/cli-smoke.mjs |
rwmjhb
left a comment
There was a problem hiding this comment.
Review paused at the conflict gate. GitHub currently reports mergeable=CONFLICTING / merge_state_status=DIRTY, so deep review is deferred until this branch rebases cleanly onto the base.
Please rebase or merge the latest base branch, resolve the conflicts, and push the updated branch. I will re-run the review once the PR is mergeable again.
…aming-engine # Conflicts: # dist/index.js # index.ts
|
Merged latest master to clear the conflict |
rwmjhb
left a comment
There was a problem hiding this comment.
Requesting changes based on the latest orchestrator review on head 6cee108.
Must fix before merge:
-
Full
npm testdid not complete. Verification ended with[TIMEOUT] npm test exceeded 180s. This PR adds scheduled background memory mutation behavior and touchesindex.ts/src/store.ts, so the full suite needs to complete successfully or the timeout needs to be isolated as pre-existing/environmental. -
The shared dreaming engine lifecycle is unsafe across multiple registrations. The plugin creates/reuses a singleton dreaming engine, but each registered service stop calls
dreamingEngine.stop()before the final-registration check. Stopping one registration can stop the shared engine while other registrations remain active, and multiple registrations can also schedule independent timers against the same singleton store/engine. Please move scheduler/engine lifecycle ownership into shared singleton state and only stop it when the final registration stops. -
stop -> start -> stopon the same service object leaves the second stop as a no-op.stop()setsregistrationStopped=true, whilestart()restarts dreaming but does not reset that flag or restore registration ownership. A restarted service can schedule background work that the next stop will not clean up. Please make this lifecycle reversible, or make stopped service objects terminal and prevent restart scheduling.
Also worth addressing while here:
sources=["sessions"]does not reliably reach normal session-summary rows.stop()does not interrupt mutations already inside a light/deep/REM phase.fetchForCompactionordering now changes existing non-dreaming compaction callers.- Invalid dreaming config can throw and abort plugin registration.
- There is unrelated line-ending churn in
index.tsaround thedualMemoryHintLoggedblock.
|
Pushed a lifecycle fix for the shared dreaming engine. Scheduler ownership now lives with the singleton state, stopping one registration no longer stops the shared engine while another registration is active, and stopped service objects do not restart background dreaming work. Checks now pass:
|
|
Fixed the Node 22 packaging/workflow failure too. The request-dimensions assertion now matches the embedding request for the submitted text, so delayed background embedding probes cannot make it inspect the wrong payload. Additional check run locally with Node 22:
|
rwmjhb
left a comment
There was a problem hiding this comment.
Review paused again at the conflict gate for the latest head 8744aac. GitHub currently reports mergeable=CONFLICTING / merge_state_status=DIRTY, so deep review is deferred until this branch rebases cleanly onto the base.
Please rebase or merge the latest base branch, resolve the conflicts, and push the updated branch. I will re-run the review once the PR is mergeable again.
|
Merged latest master to clear the conflict. CI is green now. |
rwmjhb
left a comment
There was a problem hiding this comment.
Approved on head 26f3a03. Orchestrator verdict: approve.
Independent verification run on the same head:
- npm ci --silent
- npm run build --if-present
- npm test
- git diff --exit-code -- dist src index.ts package.json package-lock.json openclaw.plugin.json README.md scripts test
All passed. Non-blocking follow-ups to consider: exclude expired valid_until memories from dreaming inputs, clarify/cover sources=["sessions"] behavior, wait for or abort in-flight sweeps before destroying the store, and confirm the fetchForCompaction ordering change is intentional for existing compaction callers.
Closes #565.
Adds a default-off native dreaming sweep runner for memory-lancedb-pro with scheduled light, deep, and REM phases. It archives high-similarity recent duplicates, promotes high-value recalled memories, and stores REM reflection memories with a durable dreaming-engine source tag.
Also wires dreaming config parsing/scheduling into plugin startup and covers the behavior with focused regression tests.