fix(config): delta-only save against comparand to close fragment/env drag-in#258
Merged
fix(config): delta-only save against comparand to close fragment/env drag-in#258
Conversation
…drag-in Save paths running in-process (Web UI PATCH, /memory-dirs/add|remove, MCP mem_config) read cfg.<field> directly and wrote whatever the runtime config held — which, after load_config_d, included fragment- and env-merged values. A user PATCHing an unrelated field silently copied config.d/ fragment contents into config.json's REPLACE layer, freezing subsequent fragment edits. See project_fragment_dragin_gap.md for Phase 0 reproduction (5 scenarios + 3 measurements, all PASS) and the feedback_silent_policy_enforcement_gap.md "Known instances" entry. Delta-only write: save_config_overrides now builds a fresh comparand (defaults + env + fragments + env-dependent factories via _build_comparand) and persists only fields where cfg differs from it. Three silent-persistence patterns collapse into one mechanism: 1. default-equal (PR #256 drop-default) — default is a comparand case 2. env-sourced — BaseSettings reads MEMTOMEM_* at construction, so the comparand matches cfg and the env value drops cleanly 3. fragment-sourced — load_config_d merges fragments into comparand PR #256's _EXTRA_PERSIST_FIELDS defensive exemption is superseded (same concern, different angle): factory output is part of the comparand, so "equals default" on machine A drops cleanly while machine B rebuilds its own factory at load time. The set was renamed to _EXTRA_MUTATION_FIELDS (not deleted) because it retains a distinct role — marking fields that save_config_overrides persists even though they are absent from MUTABLE_FIELDS (managed by dedicated endpoints). load_config_d gains a quiet=True flag that suppresses warning-level logs only. Used by _build_comparand to avoid repeating "malformed fragment" / "unknown section" warnings on every save. Errors still raise. Comparand build costs 1.88 ms (100-iteration mean, real ~/.memtomem/config.d/). Save frequency under typical use (interactive PATCH) is well below the threshold where this overhead matters; rate- limited UI never approaches N req/s where it would. _field_equals_default is deleted (grep confirmed 0 external consumers). Tests: 11 existing TestSaveConfigOverrides rewritten as 16 (3 renamed for semantic generalization default→comparand, 1 deleted because Z flips its intent, 7 kept as-is, 6 new: fragment_not_dragged, env_not_dragged, memory_dirs_factory_default_not_persisted, save_is_idempotent, machine_migration_requires_active_reset, comparand_build_suppresses_warnings). All tests now share an `isolated` fixture that patches both _override_path and _config_d_path to tmp_path — without it, comparand build reads the developer's real ~/.memtomem/config.d/ fragments and tests pass accidentally (caught during Phase 1 rewrite). docs/guides/configuration.md: adds a "Delta-only save semantics" subsection replacing the stale "Web UI dumps every mutable field" narrative (now incorrect post-#256/#257), and a "Moving config.json between machines" note for Scenario-5 migration guidance. Refs: PR #249 (config.d/ precedence + the "config_set intentionally skips load_config_d" design that kept CLI safe), #256 (drop-default on save; _EXTRA_PERSIST_FIELDS exemption, now superseded), #257 (nested list coerce on mutation path)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Save paths running in-process (Web UI
PATCH /api/config,/memory-dirs/*, MCPmem_config) readcfg.<field>directly and wrote whatever the runtime config held. Afterload_config_d, that included fragment- and env-merged values. A user PATCHing an unrelated field silently copiedconfig.d/fragment contents intoconfig.json's REPLACE layer, freezing subsequent fragment edits. Tracked inproject_fragment_dragin_gap.mdand thefeedback_silent_policy_enforcement_gap.md"Known instances" entry (instance 2 — the canonical[]wipe, instance 1, was already resolved by #256 + #257).Reproduction (Phase 0, 2026-04-19, 5 scenarios + 3 measurements, all PASS):
config.d/noise.jsondefinesexclude_patterns;mm config set search.default_top_k 42via the web flow drags the fragment values intoconfig.json, and a subsequent fragment edit no longer takes effect. Phase 0 validated the Z design end-to-end without touching real code.Design — delta-only save against a comparand
save_config_overridesnow builds a fresh comparand via_build_comparand():Mem2MemConfig()construction readsMEMTOMEM_*env and runsdefault_factorycallables, so the comparand inherently reflectsdefaults + env + fragments + env-dependent factory output. Save persists only fields wherecfg.<field> != comparand.<field>.Three silent-persistence patterns collapse into one mechanism:
MEMTOMEM_MMR__ENABLED=trueno longer drag-pins intoconfig.json.load_config_dmerges fragments into the comparand; fragment values stop copying intoconfig.jsonon unrelated saves.Same concern, different angle
PR #256 introduced
_EXTRA_PERSIST_FIELDSas a defensive exemption for env-dependent factories (documented infeedback_env_dependent_factory_equality.md). This PR addresses the same concern differently — by including factory output in the comparand itself. Machine A saves: factory matches → drops. Machine B loads: factory runs fresh, returns the right local value. No exemption needed; no machine-A-to-B pin either.The set was renamed to
_EXTRA_MUTATION_FIELDS, not deleted, because it retains a distinct role beyond the pre-Z always-persist semantics: marking fields thatsave_config_overridespersists even though they are absent fromMUTABLE_FIELDS(managed by dedicated endpoints/memory-dirs/add|remove— generic mutation would bypass filesystem validation and indexing triggers). The rename reflects the semantic shift; an inline comment preserves the pre-Z history so future readers don't re-litigate.Secondary changes
load_config_d(quiet=False)parameter added — suppresses WARNING-level logs only (malformed fragment, unknown section, etc.). Used by_build_comparandto prevent repeated warnings on every save. Errors still raise._field_equals_defaultdeleted — grep confirmed 0 external consumers. Delta comparison is a plaincfg_val != comp_valinline.docs/guides/configuration.md— new "Delta-only save semantics" subsection replacing the stale "Web UI dumps every mutable field" narrative (incorrect post-fix(config): drop default-valued fields on save to prevent silent leftover pins #256/fix(config): coerce list[BaseSettings] on mutation path #257), and a "Movingconfig.jsonbetween machines" note for Scenario-5 migration guidance.Performance
Comparand build: 1.88 ms mean per call (100 iterations, real
~/.memtomem/config.d/with 3 fragments). Save frequency under interactive use is far below any rate where this overhead matters; UI PATCH is rate-limited by user interaction.Migration safety
Existing
config.jsonfiles with pinned values load normally. On the next save:memory_dirscarried to machine B) → stay pinned until user actively resets. Tested:test_machine_migration_requires_active_reset. Documented with an explicit reset recipe ("remove theindexingsection" ormm init --fresh).Test plan
1699 tests pass (pre-branch 1694 → +5 net: +6 new, −1 deleted).
TestSaveConfigOverridesrewritten. Matrix:test_default_valued_field_not_persisted→test_comparand_equal_field_not_persistedtest_existing_default_entry_pruned_on_save→test_existing_comparand_equal_entry_prunedtest_section_with_only_defaults_dropped_entirely→test_section_with_only_comparand_equal_fields_droppedtest_memory_dirs_equal_to_default_still_persistedtest_memory_dirs_factory_default_not_persistedtest_fragment_value_not_dragged_to_config_jsontest_env_value_not_draggedtest_memory_dirs_factory_default_not_persistedtest_save_is_idempotenttest_machine_migration_requires_active_resettest_comparand_build_suppresses_warningsquiet=Truecontract (suppresses warnings, errors still raise)New
isolatedfixtureAdded to support multi-source test setup (config.json + config.d/ fragments + env vars in one place). Without it,
_build_comparandreads the developer's real~/.memtomem/config.d/and tests pass accidentally — caught during Phase 1 rewrite. Centralizing the patching prevents every new test from repeating the boilerplate.Test plan checklist
uv run pytest -m "not ollama"— 1699 passeduv run ruff check+ruff format --check— cleanuv run mypy packages/memtomem/src— 0 issues_field_equals_defaultgrep in entire tree → 0 references (safe deletion)Refs
config.d/precedence + the "config_set intentionally skipsload_config_d" design that kept CLI safe from this flavor of drag-in._EXTRA_PERSIST_FIELDSdefensive exemption (now superseded by comparand inclusion).list[BaseSettings]coerce on mutation path; orthogonal.