Skip to content

Add native dreaming sweep engine#898

Merged
rwmjhb merged 9 commits into
CortexReach:masterfrom
TurboTheTurtle:codex/issue-565-dreaming-engine
Jun 27, 2026
Merged

Add native dreaming sweep engine#898
rwmjhb merged 9 commits into
CortexReach:masterfrom
TurboTheTurtle:codex/issue-565-dreaming-engine

Conversation

@TurboTheTurtle

Copy link
Copy Markdown
Contributor

Closes #565.

Adds a default-off native dreaming sweep runner for memory-lancedb-pro with scheduled light, deep, and REM phases. It archives high-similarity recent duplicates, promotes high-value recalled memories, and stores REM reflection memories with a durable dreaming-engine source tag.

Also wires dreaming config parsing/scheduling into plugin startup and covers the behavior with focused regression tests.

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The feature direction may be useful, but this implementation currently mutates memory state in ways that do not line up with existing retrieval/governance behavior.

Must fix

Light-phase archived duplicates can still be recalled.

The dreaming engine archives duplicates by patching metadata to state: "archived" and memory_layer: "archive", but normal retrieval/search still only uses active/expiry filtering. The store excludeInactive path delegates to isMemoryActiveAt, which checks temporal validity such as valid_from / invalidated_at, not state: archived or memory_layer: archive. Manual memory_recall and OpenClaw capability search can therefore still return a duplicate after the light phase has supposedly archived it.

Please either make retrieval/store inactive filtering exclude archived/archive-layer memories consistently, or have the dreaming engine set the same invalidation metadata that existing retrieval already respects. Add a regression test that archives a duplicate through the light phase and then proves retriever/search no longer returns it.

Also worth addressing before this grows further

  • Default deep promotion requires unique-query metadata, but existing recall/access writers only maintain access_count / last_accessed_at; normal recalled memories will not satisfy the default minUniqueQueries=3 gate.
  • dreaming_phase is used as an audit marker on real memories, but isDreamingGenerated() treats any dreaming_phase as generated output, causing real memories touched by light/deep phases to be skipped in later sweeps.
  • Source filters advertised in the manifest (daily, sessions, recall, logs, memory, deep) are compared directly to stored metadata.source / category values, so configured filters may silently match nothing.
  • The scheduler can reschedule after service shutdown if stop happens while a sweep is in flight; DreamingEngine.stop() is currently a no-op.
  • Unsupported cron/frequency strings silently fall back to daily scheduling instead of failing closed or warning clearly.

Given the scope drift from the original unsupported-setting issue, I would also prefer splitting capability/config compatibility from automatic archive/promotion/reflection mutation unless maintainers explicitly want the full native sweep engine in this PR.

@TurboTheTurtle

Copy link
Copy Markdown
Contributor Author

@rwmjhb Thanks, agreed on the scope concern.

I pushed a narrow fix for the blocking issue: light-phase duplicate archival now sets invalidated_at, and the regression runs the light phase then verifies normal retriever/search filtering no longer returns the archived duplicate. I also fixed the adjacent dreaming_phase audit-marker issue so real memories touched by light/deep phases are not treated as dreaming-generated output.

For the remaining bullets, I agree those should not keep expanding this PR unless maintainers explicitly want the full native sweep engine here. My preference is to split this back down to the original capability/config compatibility work, then handle the native archive/promotion/reflection engine separately with the unique-query metadata, source-filter mapping, scheduler shutdown, and invalid-frequency behavior designed together.

Happy to trim/split this PR in that direction if that’s what you prefer.

…reaming-engine

# Conflicts:
#	scripts/ci-test-manifest.mjs

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on the latest head. Build and focused tests are clean, but the implementation still has several correctness issues in the enabled dreaming mutation path.

Independent verification on e3fea38 passed:

  • npm run build --if-present
  • node --test test/dreaming-engine.test.mjs
  • node test/plugin-manifest-regression.mjs
  • node test/cli-smoke.mjs
  • generated/source files remained clean after build

Must fix

  1. Dreaming can reschedule after service shutdown. The scheduled callback clears dreamingTimer, awaits runDreamingSweep(), then schedules the next sweep unconditionally. stop() only clears an existing timer, and DreamingEngine.stop() is effectively a no-op. If shutdown happens while a sweep is in flight, the callback can schedule new background work after plugin stop/reload and continue mutating memory. Please add a stopped/abort state and check it before and after the awaited sweep.

  2. Manifest source filters do not map to the plugin's stored source values. entryMatchesSources() compares configured values directly to metadata.source or category, while the manifest advertises values such as daily, sessions, recall, logs, memory, and deep. The stored normalized sources are values like manual, auto-capture, reflection, dreaming-engine, session-summary, and legacy. A valid advertised filter can therefore silently match zero memories. Please map these schema values explicitly or reject unsupported values during config normalization.

  3. The default deep promotion gate is not fed by normal recall/access paths. Deep promotion defaults to minUniqueQueries = 3, but normal recall/access updates access_count and last_accessed_at, not unique-query metadata. The current test promotes by pre-seeding unique_query_count, so the default real path can no-op despite repeated recalls. Please either record bounded unique query metadata during recall or make the default gate use metadata the plugin already writes.

Also worth fixing

  • Light dedupe limits candidates before applying the recent lookback window, so older rows can fill the limited batch and recent duplicates can be missed.
  • Invalid/unsupported dreaming frequency silently falls back to a 24-hour schedule for a mutating feature.
  • Deep promotion updates metadata and importance in separate writes; if the second write fails, the memory can be left partially promoted.

Given this feature archives/promotes/creates memories, I do not think these should ship as follow-ups.

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on the latest head. The shutdown/source-filter/deep-promotion fixes are moving in the right direction, and build/focused tests are clean, but there is still a lifecycle blocker.

Independent verification on fb63742 passed:

  • npm run build --if-present
  • node --test test/dreaming-engine.test.mjs
  • node test/plugin-manifest-regression.mjs
  • node test/cli-smoke.mjs
  • build left source/dist/package/manifest/test files clean

Must fix

Dreaming cannot restart after service stop. The service stop path calls dreamingEngine.stop(), which sets the engine-local stopped flag. runSweep() then returns immediately whenever that flag is true. On a later service start, the outer scheduler flag is reset, but the engine-local stopped flag is never cleared. The result is that timers can be scheduled while every sweep silently no-ops until the plugin state is recreated.

Please make the engine lifecycle reversible: add a start()/reset() path that clears the engine stop state on service start, or recreate the engine when starting after a stop. Add a regression test for stop -> start -> sweep on the same registered service instance.

Also worth fixing

  • Stop still does not abort mutations already inside the current light/deep/REM phase; it only prevents later phase starts.
  • The advertised sessions source remains unreliable: session-summary rows can keep the legacy source when no explicit source is patched, while raw session-summary metadata can default to archived and be filtered out.
  • The light sweep path can materialize and sort the full scoped memory table before applying phase limits.
  • fetchForCompaction() ordering now changes existing compaction selection behavior; that may be intended, but it should be explicit because it affects non-dreaming compaction too.

@TurboTheTurtle

Copy link
Copy Markdown
Contributor Author

Fixed the restart lifecycle issue. The dreaming engine now has a reversible start path, and service start clears the engine-local stopped state before scheduling sweeps.\n\nI added a stop -> start -> sweep regression on the same engine instance and reran:\n- npm run build --if-present\n- node --test test/dreaming-engine.test.mjs\n- node test/plugin-manifest-regression.mjs\n- node test/cli-smoke.mjs

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review paused at the conflict gate. GitHub currently reports mergeable=CONFLICTING / merge_state_status=DIRTY, so deep review is deferred until this branch rebases cleanly onto the base.

Please rebase or merge the latest base branch, resolve the conflicts, and push the updated branch. I will re-run the review once the PR is mergeable again.

…aming-engine

# Conflicts:
#	dist/index.js
#	index.ts
@TurboTheTurtle

TurboTheTurtle commented Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

Merged latest master to clear the conflict

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes based on the latest orchestrator review on head 6cee108.

Must fix before merge:

  1. Full npm test did not complete. Verification ended with [TIMEOUT] npm test exceeded 180s. This PR adds scheduled background memory mutation behavior and touches index.ts / src/store.ts, so the full suite needs to complete successfully or the timeout needs to be isolated as pre-existing/environmental.

  2. The shared dreaming engine lifecycle is unsafe across multiple registrations. The plugin creates/reuses a singleton dreaming engine, but each registered service stop calls dreamingEngine.stop() before the final-registration check. Stopping one registration can stop the shared engine while other registrations remain active, and multiple registrations can also schedule independent timers against the same singleton store/engine. Please move scheduler/engine lifecycle ownership into shared singleton state and only stop it when the final registration stops.

  3. stop -> start -> stop on the same service object leaves the second stop as a no-op. stop() sets registrationStopped=true, while start() restarts dreaming but does not reset that flag or restore registration ownership. A restarted service can schedule background work that the next stop will not clean up. Please make this lifecycle reversible, or make stopped service objects terminal and prevent restart scheduling.

Also worth addressing while here:

  • sources=["sessions"] does not reliably reach normal session-summary rows.
  • stop() does not interrupt mutations already inside a light/deep/REM phase.
  • fetchForCompaction ordering now changes existing non-dreaming compaction callers.
  • Invalid dreaming config can throw and abort plugin registration.
  • There is unrelated line-ending churn in index.ts around the dualMemoryHintLogged block.

@TurboTheTurtle

Copy link
Copy Markdown
Contributor Author

Pushed a lifecycle fix for the shared dreaming engine. Scheduler ownership now lives with the singleton state, stopping one registration no longer stops the shared engine while another registration is active, and stopped service objects do not restart background dreaming work.

Checks now pass:

  • npm run build --if-present
  • node --test test/dreaming-engine.test.mjs
  • node test/plugin-manifest-regression.mjs
  • npm test

@TurboTheTurtle

Copy link
Copy Markdown
Contributor Author

Fixed the Node 22 packaging/workflow failure too. The request-dimensions assertion now matches the embedding request for the submitted text, so delayed background embedding probes cannot make it inspect the wrong payload.

Additional check run locally with Node 22:

  • node scripts/verify-ci-test-manifest.mjs && node scripts/run-ci-tests.mjs --group packaging-and-workflow

@TurboTheTurtle
TurboTheTurtle requested a review from rwmjhb June 25, 2026 23:45

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review paused again at the conflict gate for the latest head 8744aac. GitHub currently reports mergeable=CONFLICTING / merge_state_status=DIRTY, so deep review is deferred until this branch rebases cleanly onto the base.

Please rebase or merge the latest base branch, resolve the conflicts, and push the updated branch. I will re-run the review once the PR is mergeable again.

@TurboTheTurtle

Copy link
Copy Markdown
Contributor Author

Merged latest master to clear the conflict. CI is green now.

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on head 26f3a03. Orchestrator verdict: approve.

Independent verification run on the same head:

  • npm ci --silent
  • npm run build --if-present
  • npm test
  • git diff --exit-code -- dist src index.ts package.json package-lock.json openclaw.plugin.json README.md scripts test

All passed. Non-blocking follow-ups to consider: exclude expired valid_until memories from dreaming inputs, clarify/cover sources=["sessions"] behavior, wait for or abort in-flight sweeps before destroying the store, and confirm the fetchForCompaction ordering change is intentional for existing compaction callers.

@rwmjhb
rwmjhb merged commit 373117e into CortexReach:master Jun 27, 2026
8 checks passed
@TurboTheTurtle
TurboTheTurtle deleted the codex/issue-565-dreaming-engine branch June 28, 2026 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"memory-lancedb-pro" does not support dreaming settings

2 participants