Bound indexing resource usage with configurable limits #1347

Problem

An accidental index_repository request against an aggregate workspace or unexpectedly large tree can consume unbounded CPU, memory, temporary storage, and database space. Discovery currently has no unified admission policy shared with supervised workers and persistence.

Proposed behavior

Resolve one immutable resource policy per ordinary indexing request and enforce it across:

• repository-root admission and bounded discovery (files, directories, entries, depth, source bytes, per-file bytes, scan deadline)
• worker execution (CPU worker count, physical concurrency, resident memory, lower priority, wall-clock duration)
• persistence (database, staging, task temporary output, cache admission, reserved free disk)
• structured resource_limit_exceeded responses that preserve the last published database

Built-in exact denied roots would include the filesystem root, current user home, and the index cache. Operators could add exact aggregate roots with index_denied_roots; descendants would remain indexable.

cross-repo-intelligence would retain its existing database-linking path because it does not walk source files.

Proposed defaults

The prototype uses conservative finite defaults: 100k files, 20k directories, 500k entries, depth 64, 4 GiB accepted source, 64 MiB per file, 30-second discovery, 4 worker threads, one physical job, 8 GiB memory (also clamped to detected budget), 16 GiB database, 20 GiB staging, 24 GiB task temp, 32 GiB cache admission, 4 GiB free-disk reserve, and one-hour duration.

Prototype

A complete C implementation with tests and documentation is available for design review:

https://github.com/liuchong/codebase-memory-mcp/tree/fix/index-resource-guards

Focused ASan/UBSan suites: 640 passed, 3 platform skips. make -f Makefile.cbm lint-ci passes.

Design questions

1. Is a unified index_* configuration namespace acceptable?
2. Are the proposed built-in denied roots and defaults appropriate?
3. Should the implementation be reviewed as one cross-cutting PR, or split along discovery/supervision/persistence boundaries despite the shared policy contract?