Hey everyone! I made my first push! (to clarify, I've made commits in the past, but this is the first real overhaul concept for MemPalace, done with Claude CLI obviously as I'm not a coder, but making sure I was there every step of the way to tests each piece of script, code, how it would affect the version people are using and how to try to make any merges as seamless as possible, that's why I always put my Claude agent Lu at the bottom, to make it clear that I used AI for complete transparency).

I know it's nothing for you all, but it's taken the last few weeks to understand what I PERSONALLY would need for my own file organizing process, refine and test with TDD and then run it against the current version at every step to make sure the integration will be (fingers crossed) as seamless as I could figure out how to make it.

In a nutshell, that's why it's so long.

I figured just finish the concept/idea first and then push it all at once so its coherent and not split into a bunch of commits.

I'm looking forward to all your thoughts, criticism etc...And most importantly, I hope it will help you all too (though I'm pretty positive your files are all A LOT more organized than mine lol), so figured whatever I added that could help me would also make it that much more user friendly for the uber intelligent GitHub community.
Plus, you all have an inordinate amount more experience to really judge the work.
anyway it's what I've been envisioning as a potential, new version so I would love for you to give it s whirl!

thank you so much everyone for the time and energy you've put into this project... It's amazing to be a part of such an insanely talented community. -Milla & Lu✨

Congrats on the first push, Milla — the persona-bucket idea is clever and the live Haiku verification is real work.

Quick question on direction, not blocking: the diff in CLAUDE.md softens Local-first, zero API to Local-first, zero external API by default, and the new flow makes LLM-assisted classification on-by-default. Was the design-principle change discussed somewhere I should catch up on, or is the call open here? I just want to understand whether init reaching out to a localhost LLM (Ollama default) is now the expected baseline — that's a meaningful shift from where the README pitches the project.

Either way I'll defer to @bensig / @igorls — happy to keep contributing in whichever direction the project lands.

By Zero API, it has always meant zero external APIs and zero keys required. When we talked about API, it was more about API keys for external providers that could potentially undermine your privacy. Now we are simply being more explicit about it.

@arnoldwender thanks for the question! I think to use this with zero LLM was not what my original idea was about. unless you're REALLY good working without an LLM, which I'm sure most people on GitHub are, but might not be something a "newb" would understand so well,I wanted to create a path that was easier for people to understand. that using "agentic intelligence" to always be an option. for devs on here, I'm sure its local models, but I wanted their to be a clear warning that if you are signing in through an API key, the user HAS to understand that their privacy by nature of something non local is also not an option. Also, I found it challenging to use the completely LLM free version on the types of files I have (as some are numbered, some have dates, some have no dates, some are script notes, some are stories, music lyrics etc... too many formats for a 0 model to handle. and honestly, I didn't create all this code on my own obviously as I'm an actress not a coder. And as I haven;t really decided on a local model that I really love yet, I run the pip install through Claude. I just wanted it to be a an easier experience for people on here who DO rely on some sort of model, whether local or not. -m

@milla-jovovich totally agree with the goal — real-world files are messy and onboarding should just handle that.

One thing I'd flag before making LLM-assisted init the default: MemPalace stores everything verbatim by design, which means it holds some of the most sensitive data a user has. If we route that through a free-tier cloud LLM by default, the users who understand privacy least are the ones most exposed — free-tier providers typically use inputs for training.

Worth noting that the OSS ecosystem already handles multi-format classification locally. unstructured (Apache 2.0, 14k stars, pip install unstructured) does format detection + per-type routing for 60+ file types — scripts, lyrics, dated notes, all covered — zero API key, runs fully offline. fastText handles content-based classification the same way. Both are genuinely free, no cloud, no per-page pricing.

The approach I'd lean toward: local heuristics as the default path, LLM as an explicit opt-in with a clear "your data will leave this machine" warning.

Happy to prototype the router layer if the direction sounds right — just wanted to surface the prior art before we touch the local-first defaults.

@arnoldwender let me double check, but I think most people on here start with their own locl model... so for someone using an api key like I am for instance, would (hopefully) already know that anthropic has all my data... and I'm sure people with experience understand that, but I wanted an extra safety rail for some people who might not know. but thank for the heads up and will take a look to triple check that what the concept actually suggests is using a cloud service as default. if so, then that's an oversight on my part... I don't want to speculate, but I think what it SHOULD be doing is scanning what you are using already and if it's a local agent, would use that as a default unless you say otherwise. but if you're using a cloud based agent like Claude Cli via api like I am, I intrinsically understand that my information is already in their systems and mempalace as well should give some warning about that. I wanted this new concept for an update consideration to make it more CLEAR to people, which obviously it's not since you brought it up, so let me check this now. I really appreciate the feedback! -m

@milla-jovovich the corpus-origin module is well-architected (two-tier with the co-occurrence rule for Claude/Gemini/Haiku is exactly the right call to avoid French-novel/astrology false-positives), the persona-reclassification sweep in discover_entities is clean, and backwards compat via corpus_origin=None is preserved tightly. 1355/1356 tests pass locally, lint and format are clean, all CI jobs green.

Two things to address before this is mergeable:

1. Branch is behind develop and would revert #1208's data-loss safety fix.

git diff origin/develop..feat/corpus-origin --stat shows:

• mempalace/repair.py — 148 lines removed
• tests/test_repair.py — 120 lines removed

That's the entire TruncationDetected / check_extraction_safety / CHROMADB_DEFAULT_GET_LIMIT block that landed via #1208 / #1210, plus its regression tests. GitHub reports mergeable: CLEAN because the feature commit doesn't touch repair.py directly — but a merge would still drop the safety code along the way. A rebase onto current develop fixes this with no other rework needed.

2. test_init_pass_zero_uses_full_file_content_not_front_sampled doesn't isolate the LLM tier.

The test builds an argparse.Namespace without no_llm, so cmd_init tries to acquire a provider. CI passes because CI has no Ollama (check_available() returns False, LLM tier is skipped, heuristic correctly flags the AI tail). Locally with Ollama + gemma4:e4b running, the model returns "not AI-dialogue, confidence 0.90" and the test fails. Same risk for the other Pass-0 integration tests that rely on heuristic-only behavior — worth a sweep.

Fix is small: patch("mempalace.cli.get_provider", ...) in the affected tests, or add no_llm=True to the Namespace.

That second one also surfaces a worth-discussing product nuance: with --llm now default-on and gemma4:e4b as the Ollama default, the LLM tier replaced a correct Tier-1 answer with a wrong one on this fixture. Might be worth letting Tier 1 win when it returns high confidence (≥0.9) and only invoking Tier 2 for the weak-signal / persona-extraction case.

@milla-jovovich — you're right, I misread the intent. The diff defaults to ollama + gemma4:e4b (local), not a cloud provider — get_provider + check_available() guard makes sure it only activates if Ollama is actually running. My cloud-default concern was off.

The remaining nit is cosmetic: users without Ollama installed get a silent heuristic-only run with no indication that the LLM tier was attempted and skipped. A one-liner like "LLM not available — running heuristics only. Install Ollama or pass --no-llm to silence this." would close that loop. Happy to add it if you'd like, or I can leave it as-is.

@igorls thank you so much! already fixed and I'm pretty sure you've merged now. will test out the option 3 I brought up and add is as a rebase. or an add on... anyway, haven't had time to do practice my GitHub skill or terminology as this week has been spent doing the "feature init" overhaul for the Mempalace update. will have time to do some GitHub learning when the kids are in school on Monday lol! thank you again for your tireless work over the last few weeks. you've been a rockstar and I'm so grateful to have you on the team!🙏

@arnoldwender yes I know lol! I've been deep into the figuring out the concept and model structure all week for the update that I missed the few last commits, but fixed those already via @igorls who is part of our tiny MemPalace squad, so the issues were already being fixed as I ws answering you earlier!

about your users and Ollama fix, it's already been flagged internally and I've already got a rebase that I will be adding in a few minutes to remain "agent agnostic" on the MemPalace platform!

Saying that, thank you again for being so vigilant. that's why this community is incredible. I'm so happy that MemPalace resonates with so many people and they take the time to point out these kinds of issues. it's so thoughtful. best, m

@arnoldwender So about the nit cosmetic: Actually we already shipped that exact one-liner:

see mempalace/cli.py:252-262 in #1211 (lines 254-256 for the check_available()=False path, 259-261 for the LLMError exception path).
Both print "... Running heuristics-only — pass --no-llm to silence this." and include the underlying reason ({msg}) so users see what actually went wrong (wrong port, missing key, etc.).

Happy to discuss the wording if you think the message could be clearer in some other way!

You're right, my mistake — I missed those two branches on my pass through cli.py. Both print the message with the underlying reason inline, which is exactly what I was after. No change needed.

Thanks for the patience walking me through it, and congrats on the merge — the persona-bucket work is going to land well with users.