Skip to content

feat: path-level mining control via paths config in mempalace.yaml#986

Open
roip wants to merge 1 commit intoMemPalace:mainfrom
roip:feat/path-descriptions
Open

feat: path-level mining control via paths config in mempalace.yaml#986
roip wants to merge 1 commit intoMemPalace:mainfrom
roip:feat/path-descriptions

Conversation

@roip
Copy link
Copy Markdown

@roip roip commented Apr 17, 2026

Summary

  • Adds a paths section to mempalace.yaml with a level attribute (ignore, describe, full) for per-pattern mining control
  • level: describe — creates a single drawer with user-provided description text instead of full content chunking
  • level: ignore — skips the file entirely during mining
  • level: full — default behavior, mine everything (implicit when no rule matches)

This replaces the binary mine-or-exclude model with three-tier awareness. Real-world test on a project with 849 files reduced drawers from 24,353 to 9,836 (60% reduction) by describing data files instead of chunking them.

Config format

\\yaml
paths:
'data/results/*.json':
level: describe
description: 'Experiment result JSONs — centroid tracking, spectral features'
'tests/':
level: describe
description: 'Pytest test suite'
room: testing
'pnpm-lock.yaml':
level: ignore
\\

Mining behavior

  • describe: skip content chunking, create one drawer with [Path description] relative/path: description, tagged with \mining_level: describe\ metadata
  • ignore: skip entirely, report count in summary output
  • Patterns use fnmatch glob syntax (same as .gitignore)
  • Respects mtime — only re-creates if description changed
  • Room can be overridden per pattern or auto-detected

Test plan

  • 25 tests pass (19 existing + 6 new)
  • test_match_path_description_glob — glob matching and room override
  • test_match_path_description_skips_non_describe_levels — full-level entries ignored
  • test_mine_with_path_descriptions — end-to-end describe mining
  • test_match_path_rule_ignore — ignore level flagging
  • test_compile_path_rules_skips_full_and_invalid — validation
  • test_mine_with_ignore_produces_no_drawers — end-to-end ignore mining

Resolves #981

Add a paths section to mempalace.yaml with level attribute (ignore, describe,
full) so users can control mining granularity per file pattern. Files matching
level=describe produce a single drawer with user-provided description text
instead of full content chunking. Files matching level=ignore are skipped
entirely. This replaces the binary mine-or-exclude model with a three-tier
awareness system.

Tested with 25 passing tests including 6 new ones for path rules.

Resolves MemPalace#981
@igorls igorls added enhancement New feature or request area/mining File and conversation mining labels Apr 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/mining File and conversation mining enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants