Skip to content

fix(config): delta-only save against comparand to close fragment/env drag-in#258

Merged
memtomem merged 1 commit intomainfrom
fix/save-delta-only
Apr 18, 2026
Merged

fix(config): delta-only save against comparand to close fragment/env drag-in#258
memtomem merged 1 commit intomainfrom
fix/save-delta-only

Conversation

@memtomem
Copy link
Copy Markdown
Owner

Summary

Save paths running in-process (Web UI PATCH /api/config, /memory-dirs/*, MCP mem_config) read cfg.<field> directly and wrote whatever the runtime config held. After load_config_d, that included fragment- and env-merged values. A user PATCHing an unrelated field silently copied config.d/ fragment contents into config.json's REPLACE layer, freezing subsequent fragment edits. Tracked in project_fragment_dragin_gap.md and the feedback_silent_policy_enforcement_gap.md "Known instances" entry (instance 2 — the canonical [] wipe, instance 1, was already resolved by #256 + #257).

Reproduction (Phase 0, 2026-04-19, 5 scenarios + 3 measurements, all PASS): config.d/noise.json defines exclude_patterns; mm config set search.default_top_k 42 via the web flow drags the fragment values into config.json, and a subsequent fragment edit no longer takes effect. Phase 0 validated the Z design end-to-end without touching real code.

Design — delta-only save against a comparand

save_config_overrides now builds a fresh comparand via _build_comparand():

comparand = Mem2MemConfig()   # defaults + env (BaseSettings auto)
load_config_d(comparand, quiet=True)  # + fragments

Mem2MemConfig() construction reads MEMTOMEM_* env and runs default_factory callables, so the comparand inherently reflects defaults + env + fragments + env-dependent factory output. Save persists only fields where cfg.<field> != comparand.<field>.

Three silent-persistence patterns collapse into one mechanism:

  1. Default-equal (PR fix(config): drop default-valued fields on save to prevent silent leftover pins #256 drop-default) — the default is simply a comparand case.
  2. Env-sourced — env is part of the comparand, so MEMTOMEM_MMR__ENABLED=true no longer drag-pins into config.json.
  3. Fragment-sourcedload_config_d merges fragments into the comparand; fragment values stop copying into config.json on unrelated saves.

Same concern, different angle

PR #256 introduced _EXTRA_PERSIST_FIELDS as a defensive exemption for env-dependent factories (documented in feedback_env_dependent_factory_equality.md). This PR addresses the same concern differently — by including factory output in the comparand itself. Machine A saves: factory matches → drops. Machine B loads: factory runs fresh, returns the right local value. No exemption needed; no machine-A-to-B pin either.

The set was renamed to _EXTRA_MUTATION_FIELDS, not deleted, because it retains a distinct role beyond the pre-Z always-persist semantics: marking fields that save_config_overrides persists even though they are absent from MUTABLE_FIELDS (managed by dedicated endpoints /memory-dirs/add|remove — generic mutation would bypass filesystem validation and indexing triggers). The rename reflects the semantic shift; an inline comment preserves the pre-Z history so future readers don't re-litigate.

Secondary changes

  • load_config_d(quiet=False) parameter added — suppresses WARNING-level logs only (malformed fragment, unknown section, etc.). Used by _build_comparand to prevent repeated warnings on every save. Errors still raise.
  • _field_equals_default deleted — grep confirmed 0 external consumers. Delta comparison is a plain cfg_val != comp_val inline.
  • docs/guides/configuration.md — new "Delta-only save semantics" subsection replacing the stale "Web UI dumps every mutable field" narrative (incorrect post-fix(config): drop default-valued fields on save to prevent silent leftover pins #256/fix(config): coerce list[BaseSettings] on mutation path #257), and a "Moving config.json between machines" note for Scenario-5 migration guidance.

Performance

Comparand build: 1.88 ms mean per call (100 iterations, real ~/.memtomem/config.d/ with 3 fragments). Save frequency under interactive use is far below any rate where this overhead matters; UI PATCH is rate-limited by user interaction.

Migration safety

Existing config.json files with pinned values load normally. On the next save:

  • Values that now match the local comparand → pruned automatically (ambient cleanup, same mechanism as fix(config): drop default-valued fields on save to prevent silent leftover pins #256 post-merge).
  • Values that don't match (e.g. machine-A paths in memory_dirs carried to machine B) → stay pinned until user actively resets. Tested: test_machine_migration_requires_active_reset. Documented with an explicit reset recipe ("remove the indexing section" or mm init --fresh).

Test plan

1699 tests pass (pre-branch 1694 → +5 net: +6 new, −1 deleted).

TestSaveConfigOverrides rewritten. Matrix:

Action Test Note
RENAME test_default_valued_field_not_persistedtest_comparand_equal_field_not_persisted default ⊂ comparand
RENAME test_existing_default_entry_pruned_on_savetest_existing_comparand_equal_entry_pruned same
RENAME test_section_with_only_defaults_dropped_entirelytest_section_with_only_comparand_equal_fields_dropped same
DELETE test_memory_dirs_equal_to_default_still_persisted Z flips this semantic; replaced by test_memory_dirs_factory_default_not_persisted
KEEP 7 tests (memory_dirs_survives, invalid_value ×2, existing_not_clobbered, non_default_persists, legacy_repr, namespace_rules_round_trip) delta-unrelated
NEW test_fragment_value_not_dragged_to_config_json drag-in regression guard
NEW test_env_value_not_dragged env drag-in guard
NEW test_memory_dirs_factory_default_not_persisted Z core behavior (replaces the deleted test)
NEW test_save_is_idempotent 2× save byte-identical; order-dependency guard
NEW test_machine_migration_requires_active_reset Scenario-5 migration edge case, doc reference
NEW test_comparand_build_suppresses_warnings quiet=True contract (suppresses warnings, errors still raise)

New isolated fixture

Added to support multi-source test setup (config.json + config.d/ fragments + env vars in one place). Without it, _build_comparand reads the developer's real ~/.memtomem/config.d/ and tests pass accidentally — caught during Phase 1 rewrite. Centralizing the patching prevents every new test from repeating the boilerplate.

Test plan checklist

  • uv run pytest -m "not ollama" — 1699 passed
  • uv run ruff check + ruff format --check — clean
  • uv run mypy packages/memtomem/src — 0 issues
  • Manual: _field_equals_default grep in entire tree → 0 references (safe deletion)
  • Manual: comparand build benchmark → 1.88 ms/call, well under 5 ms
  • Manual: Phase 0 script (5 scenarios + 3 measurements) re-run after implementation → all PASS

Refs

…drag-in

Save paths running in-process (Web UI PATCH, /memory-dirs/add|remove,
MCP mem_config) read cfg.<field> directly and wrote whatever the runtime
config held — which, after load_config_d, included fragment- and
env-merged values. A user PATCHing an unrelated field silently copied
config.d/ fragment contents into config.json's REPLACE layer, freezing
subsequent fragment edits. See project_fragment_dragin_gap.md for
Phase 0 reproduction (5 scenarios + 3 measurements, all PASS) and the
feedback_silent_policy_enforcement_gap.md "Known instances" entry.

Delta-only write: save_config_overrides now builds a fresh comparand
(defaults + env + fragments + env-dependent factories via
_build_comparand) and persists only fields where cfg differs from it.
Three silent-persistence patterns collapse into one mechanism:

  1. default-equal (PR #256 drop-default) — default is a comparand case
  2. env-sourced — BaseSettings reads MEMTOMEM_* at construction, so the
     comparand matches cfg and the env value drops cleanly
  3. fragment-sourced — load_config_d merges fragments into comparand

PR #256's _EXTRA_PERSIST_FIELDS defensive exemption is superseded (same
concern, different angle): factory output is part of the comparand, so
"equals default" on machine A drops cleanly while machine B rebuilds
its own factory at load time. The set was renamed to _EXTRA_MUTATION_FIELDS
(not deleted) because it retains a distinct role — marking fields that
save_config_overrides persists even though they are absent from
MUTABLE_FIELDS (managed by dedicated endpoints).

load_config_d gains a quiet=True flag that suppresses warning-level
logs only. Used by _build_comparand to avoid repeating "malformed
fragment" / "unknown section" warnings on every save. Errors still
raise.

Comparand build costs 1.88 ms (100-iteration mean, real
~/.memtomem/config.d/). Save frequency under typical use (interactive
PATCH) is well below the threshold where this overhead matters; rate-
limited UI never approaches N req/s where it would.

_field_equals_default is deleted (grep confirmed 0 external consumers).

Tests: 11 existing TestSaveConfigOverrides rewritten as 16 (3 renamed
for semantic generalization default→comparand, 1 deleted because Z flips
its intent, 7 kept as-is, 6 new: fragment_not_dragged, env_not_dragged,
memory_dirs_factory_default_not_persisted, save_is_idempotent,
machine_migration_requires_active_reset, comparand_build_suppresses_warnings).
All tests now share an `isolated` fixture that patches both _override_path
and _config_d_path to tmp_path — without it, comparand build reads the
developer's real ~/.memtomem/config.d/ fragments and tests pass
accidentally (caught during Phase 1 rewrite).

docs/guides/configuration.md: adds a "Delta-only save semantics"
subsection replacing the stale "Web UI dumps every mutable field"
narrative (now incorrect post-#256/#257), and a "Moving config.json
between machines" note for Scenario-5 migration guidance.

Refs: PR #249 (config.d/ precedence + the "config_set intentionally
skips load_config_d" design that kept CLI safe), #256 (drop-default on
save; _EXTRA_PERSIST_FIELDS exemption, now superseded), #257 (nested
list coerce on mutation path)
@memtomem memtomem merged commit 5c0dab9 into main Apr 18, 2026
7 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 18, 2026
@memtomem memtomem deleted the fix/save-delta-only branch April 18, 2026 22:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants