Skip to content

Commit 008b6c3

Browse files
authored
Merge pull request DeusData#1724 from liuchong/feat/index-resource-worker
feat(index): add worker resource watchdogs
2 parents 1fc7300 + 6ab3653 commit 008b6c3

28 files changed

Lines changed: 2916 additions & 134 deletions

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1024,10 +1024,12 @@ codebase-memory-mcp config set watcher_enabled false # stop the watcher thre
10241024
codebase-memory-mcp config set watch_non_git true # also poll non-git project roots (default: false)
10251025
codebase-memory-mcp config set index_max_files 250000 # optional per-index source-file limit
10261026
codebase-memory-mcp config set index_max_source_mb 16384 # optional per-index source-size limit
1027+
codebase-memory-mcp config set index_max_rss_mb 8192 # optional worker-tree charged-memory limit
1028+
codebase-memory-mcp config set index_max_duration_seconds 3600 # optional per-request duration
10271029
codebase-memory-mcp config reset auto_index # reset to default
10281030
```
10291031

1030-
The two `index_max_*` settings default to `off`. Exceeding one fails the complete
1032+
The four `index_max_*` settings default to `off`. Exceeding one fails the complete
10311033
index attempt rather than publishing a partial graph; an existing serving index
10321034
is preserved. See [Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).
10331035

‎docs/CONFIGURATION.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,8 @@ Current keys:
9494
| `watch_non_git` | `false` | Also poll project roots that are **not git repositories**. By default the watcher only follows git projects, so a project indexed from a plain directory is never refreshed after its first index — reindex it manually with `index_repository`. Set `true` to poll such roots on the same adaptive cadence with a file-tree scan: the indexer's own discovery walk (same skip lists, `.gitignore` and `.cbmignore` rules) hashed over each file's path, size and mtime. Any change reindexes once; paths the indexer skips (including cbm's own `.codebase-memory/` output) never trigger. The first poll after the daemon starts reindexes each such project once, since nothing records which tree state the index holds. The scan walks the whole tree every poll, so it costs more than git polling on very large trees. Read once when the daemon starts, like `watcher_enabled`. |
9595
| `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |
9696
| `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
97+
| `index_max_rss_mb` | `off` | Optional maximum charged memory in MiB for the complete contained index-worker process tree (`64..1048576`). On macOS this is phys_footprint, not resident size. |
98+
| `index_max_duration_seconds` | `off` | Optional maximum wall-clock duration in seconds for the whole index request, including crash/hang recovery (`1..86400`). Like `index_max_rss_mb`, it stops a worker only before it starts to publish the new index; see [INDEX_RESOURCE_LIMITS.md](INDEX_RESOURCE_LIMITS.md). |
9799

98100
For a watched project, `index_status` names the polling method in `watch.strategy`:
99101
`git` (HEAD and dirty-state polling), `tree` (a non-git root polled by the
@@ -135,11 +137,13 @@ has not run yet).
135137
> `auto_index` still runs, and `index_repository` stays available for manual
136138
> reindexing.
137139
138-
The two `index_max_*` settings are independent and disabled by default. They
140+
The four `index_max_*` settings are independent and disabled by default. They
139141
apply to explicit indexing, automatic indexing, and watcher re-indexing, but not
140142
to `cross-repo-intelligence`, which does not scan repository source files.
141-
Equality is allowed; exceeding either setting fails the complete index request
142-
and preserves any previously serving database. See
143+
Equality is allowed; exceeding any setting fails the complete index request and
144+
preserves any previously serving database. Worker RSS covers descendants and is
145+
not the same as the internal `CBM_MEM_BUDGET_MB` allocation budget. Total
146+
duration is independent of the existing 15-minute no-log-progress timeout. See
143147
[Index resource limits](INDEX_RESOURCE_LIMITS.md) for counting, validation, and
144148
error-response details.
145149

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 69 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Index resource limits
22

33
Index resource limits are optional operator controls for repositories whose
4-
discovery breadth is not known in advance. They are disabled by default so
5-
existing large-repository workloads retain their current behavior.
4+
discovery breadth or worker runtime is not known in advance. They are disabled
5+
by default so existing large-repository workloads retain their current behavior.
66

77
## Discovery settings
88

@@ -23,6 +23,61 @@ Values use base-10 integers. MiB means 1,048,576 bytes. Empty values, zero,
2323
negative values, suffixes, trailing characters, and values outside the stated
2424
ranges are rejected without changing the stored value.
2525

26+
## Worker settings
27+
28+
| Key | Default | Accepted value | Protects |
29+
|---|---:|---:|---|
30+
| `index_max_rss_mb` | `off` | `off` or `64..1048576` | Charged memory of the complete worker process tree |
31+
| `index_max_duration_seconds` | `off` | `off` or `1..86400` | Wall-clock duration of the whole index request |
32+
33+
```bash
34+
codebase-memory-mcp config set index_max_rss_mb 8192
35+
codebase-memory-mcp config set index_max_duration_seconds 3600
36+
```
37+
38+
RSS is the charged memory of the contained worker and every descendant, not
39+
the worker's allocation budget and not peak memory. On macOS that quantity is
40+
`phys_footprint` (the same number `cbm_mem_charged()` enforces), not
41+
`resident_size`, which still counts pages the allocator has already handed
42+
back. Linux and Windows use RSS / working set. This hard watchdog is
43+
separate from the internal `CBM_MEM_BUDGET_MB` soft budget. The supervisor
44+
samples the tree at most once every 250 milliseconds so the watchdog does not
45+
turn full process-table enumeration into a busy loop.
46+
47+
Duration is per request: the clock starts at the first worker spawn and is
48+
not reset when crash/hang recovery starts a later attempt. Continuous log
49+
progress does not reset it. It is independent of the existing 15-minute quiet
50+
timeout, which still identifies a worker that stops making progress. A
51+
duration limit shorter than that quiet timeout kills a hung worker before hang
52+
quarantine can name the in-flight file, so the next attempt may hang on the
53+
same file.
54+
55+
Equality is allowed. The first RSS or elapsed-duration observation above its
56+
limit starts the existing graceful-to-force process-tree shutdown. CBM reports
57+
terminal only after the tree is quiescent or a bounded containment failure is
58+
explicitly surfaced. Resource termination is not retried and does not
59+
quarantine a source file.
60+
61+
A worker limit never stops a worker after it has started to publish. The
62+
supervisor enforces the limits, but only the worker replaces the serving
63+
database, so the two coordinate through a per-attempt publication claim that
64+
exactly one of them can win. The worker claims it immediately before it
65+
replaces the serving database; the supervisor claims it before it stops the
66+
worker for a limit. If the limit is reached first, the worker can no longer
67+
publish and the previous index stays untouched. If the worker claimed first,
68+
the limits no longer apply to that attempt: the new index is published and the
69+
rest of the attempt, including the persistence artifact export, runs to
70+
completion. The supervisor then logs `index.supervisor.resource_limit_after_publication`
71+
instead of stopping it. With both worker limits `off` there is no claim.
72+
73+
If RSS is enabled and three consecutive probes cannot obtain any trustworthy
74+
tree measurement while the root worker is still running, CBM fails closed with
75+
`code=resource_probe_failed`. A probe is untrustworthy as soon as one member of
76+
the tree cannot be measured: only a process that exits between the listing and
77+
its measurement is skipped, because a total without a live member would
78+
under-report. On Linux the probe reads every `/proc/<pid>/stat` to find the
79+
tree's members, so a `/proc` mounted with `hidepid=1` makes it fail closed.
80+
2681
## Counting and failure semantics
2782

2883
`index_max_files` counts a file only after it passes directory pruning, ignore
@@ -55,6 +110,15 @@ The previous database remains available because publication occurs only after a
55110
complete discovery and successful staged build. If no previous database exists,
56111
`serving_index_preserved` is false.
57112

113+
Worker limit failures use the same shape with `stage=worker`,
114+
`resource=rss_bytes` and `unit=bytes`, or `resource=duration_ms` and
115+
`unit=milliseconds`. RSS measurement failures use `code=resource_probe_failed`
116+
and omit `observed`, `limit`, and `unit` because no trustworthy observation was
117+
available. For a worker failure `serving_index_preserved` describes the whole
118+
request: it is true only when a servable index existed when the request began,
119+
no attempt of the request (including an earlier one that crashed and was
120+
recovered) claimed publication, and that index still serves.
121+
58122
## Trust and compatibility
59123

60124
Limits are read from the CLI-managed `_config.db`; they are not MCP request
@@ -64,5 +128,6 @@ parent policy.
64128

65129
These settings do not replace or increase `auto_index_limit`, change the 512 MiB
66130
single-file cap, alter workspace-root authorization, or affect
67-
`cross-repo-intelligence`. With both settings `off`, discovery follows the
68-
existing unbounded path.
131+
`cross-repo-intelligence`. With all settings `off`, discovery follows the
132+
existing path and the supervisor performs no periodic RSS probe or total-duration
133+
termination.

‎scripts/test-runtime.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,8 @@ _cbm_test_runtime_daemon() {
112112
# cbm_mcp_index_policy_add_to_args: every key in cbm_index_policy_key_at, and
113113
# nothing else.
114114
cbm_test_index_worker_policy_json() {
115-
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"}'
115+
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"'
116+
printf '%s' ',"index_max_rss_mb":"off","index_max_duration_seconds":"off"}'
116117
}
117118

118119
cbm_test_runtime_cleanup() {

‎src/cli/cli.c‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8101,6 +8101,9 @@ static const config_key_def_t CONFIG_KEYS[] = {
81018101
{CBM_CONFIG_UI_PORT, "9749", "Port for the graph UI listener when enabled"},
81028102
{CBM_INDEX_CONFIG_MAX_FILES, "off", "Max accepted source files per index, or off"},
81038103
{CBM_INDEX_CONFIG_MAX_SOURCE_MB, "off", "Max accepted source MiB per index, or off"},
8104+
{CBM_INDEX_CONFIG_MAX_RSS_MB, "off", "Max worker process-tree charged memory MiB, or off"},
8105+
{CBM_INDEX_CONFIG_MAX_DURATION_SECONDS, "off",
8106+
"Max wall-clock seconds for the whole index request, or off"},
81048107
};
81058108

81068109
/* #1558: ui_enabled and ui_port were reachable ONLY by hand-editing
@@ -8127,8 +8130,12 @@ static bool config_key_is_ui(const char *key) {
81278130
}
81288131

81298132
static bool config_key_is_index_policy(const char *key) {
8130-
return key && (strcmp(key, CBM_INDEX_CONFIG_MAX_FILES) == 0 ||
8131-
strcmp(key, CBM_INDEX_CONFIG_MAX_SOURCE_MB) == 0);
8133+
for (size_t index = 0; key && index < cbm_index_policy_key_count(); index++) {
8134+
if (strcmp(key, cbm_index_policy_key_at(index)) == 0) {
8135+
return true;
8136+
}
8137+
}
8138+
return false;
81328139
}
81338140

81348141
static int config_index_policy_write(cbm_config_t *config, const char *key, const char *value) {

‎src/daemon/application.c‎

Lines changed: 45 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,11 @@ struct cbm_daemon_application_job {
176176
bool async_owned;
177177
bool worker_started;
178178
application_index_watch_owner_t *watch_owners;
179+
uint64_t request_started_ms;
180+
/* What a worker resource failure of this job is reported against; set
181+
* before the first attempt, folded over every attempt (mcp_internal.h). */
182+
bool request_facts_begun;
183+
cbm_mcp_index_request_facts_t request_facts;
179184
cbm_daemon_application_job_t *next;
180185
};
181186

@@ -378,8 +383,17 @@ static void application_worker_diagnostic(const char *line, void *context) {
378383
static int application_worker_start_default(void *context, const char *args_json,
379384
size_t memory_budget_bytes, const char *marker_file,
380385
const char *quarantine_file,
386+
uint64_t duration_origin_ms,
381387
cbm_daemon_application_worker_t *worker_out) {
382388
(void)context;
389+
cbm_index_resource_policy_t resource_policy;
390+
char error[CBM_SZ_256] = {0};
391+
if (!cbm_mcp_index_policy_from_internal_args(args_json, &resource_policy, error,
392+
sizeof(error))) {
393+
cbm_log_error("daemon.index.policy", "error", error);
394+
*worker_out = NULL;
395+
return -1;
396+
}
383397
application_default_worker_t *worker = cbm_calloc(CBM_MEM_CLASS_OTHER, sizeof(*worker));
384398
if (!worker) {
385399
return -1;
@@ -389,9 +403,9 @@ static int application_worker_start_default(void *context, const char *args_json
389403
const char *root = yyjson_get_str(yyjson_obj_get(yyjson_doc_get_root(args_doc), "repo_path"));
390404
worker->project = application_index_project_key(root, args_json);
391405
yyjson_doc_free(args_doc);
392-
int result = cbm_index_worker_start_with_log(args_json, memory_budget_bytes, false, marker_file,
393-
quarantine_file, application_worker_diagnostic,
394-
worker, &worker->handle);
406+
int result = cbm_index_worker_start_with_policy_log(
407+
args_json, memory_budget_bytes, &resource_policy, false, marker_file, quarantine_file,
408+
application_worker_diagnostic, worker, duration_origin_ms, &worker->handle);
395409
if (result != 0) {
396410
application_worker_destroy_default(NULL, worker);
397411
*worker_out = NULL;
@@ -1539,6 +1553,17 @@ static void application_cancelled_result(cbm_index_worker_result_t *result) {
15391553
result->tree_quiesced = true;
15401554
}
15411555

1556+
/* Only an enabled worker limit can end a job in a worker resource failure, the
1557+
* one response the request facts feed; without one nothing is read. */
1558+
static void application_job_request_facts_begin(cbm_daemon_application_job_t *job) {
1559+
cbm_index_resource_policy_t policy;
1560+
char error[CBM_SZ_256] = {0};
1561+
if (cbm_mcp_index_policy_from_internal_args(job->args_json, &policy, error, sizeof(error)) &&
1562+
cbm_index_policy_worker_enabled(&policy)) {
1563+
cbm_mcp_index_request_facts_begin(job->args_json, &job->request_facts);
1564+
}
1565+
}
1566+
15421567
static application_attempt_status_t application_job_run_attempt(cbm_daemon_application_job_t *job,
15431568
const char *marker_path,
15441569
const char *quarantine_path,
@@ -1572,10 +1597,17 @@ static application_attempt_status_t application_job_run_attempt(cbm_daemon_appli
15721597
}
15731598

15741599
cbm_daemon_application_worker_t worker = NULL;
1600+
if (job->request_started_ms == 0) {
1601+
job->request_started_ms = cbm_index_worker_now_ms();
1602+
}
1603+
if (!job->request_facts_begun) {
1604+
job->request_facts_begun = true;
1605+
application_job_request_facts_begin(job);
1606+
}
15751607
application_tmp_lock();
1576-
int start_result =
1577-
application->worker_ops.start(application->worker_ops.context, job->args_json,
1578-
memory_budget_bytes, marker_path, quarantine_path, &worker);
1608+
int start_result = application->worker_ops.start(
1609+
application->worker_ops.context, job->args_json, memory_budget_bytes, marker_path,
1610+
quarantine_path, job->request_started_ms, &worker);
15791611
application_tmp_unlock();
15801612
if (start_result != 0 || !worker) {
15811613
return application_job_cancel_requested(job) ? APPLICATION_ATTEMPT_CANCELLED
@@ -1763,13 +1795,20 @@ static application_attempt_decision_t application_consume_attempt(
17631795
application_attempt_disposition(job, attempt);
17641796
application_record_attempt(attempt, &execution->last_result, &execution->have_last_result,
17651797
execution->last_log);
1798+
cbm_mcp_index_request_facts_record(&job->request_facts, &attempt->result);
17661799
if (disposition == CBM_MCP_SUPERVISED_RESULT_SUCCESS) {
17671800
execution->response = attempt->result.response;
17681801
attempt->result.response = NULL;
17691802
execution->successful = application_index_response_succeeded(execution->response);
17701803
application_attempt_free(attempt);
17711804
return APPLICATION_ATTEMPT_DECISION_SUCCESS;
17721805
}
1806+
if (disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_FAILURE) {
1807+
execution->response = cbm_mcp_index_worker_resource_response(
1808+
job->args_json, &attempt->result, &job->request_facts);
1809+
application_attempt_free(attempt);
1810+
return APPLICATION_ATTEMPT_DECISION_STOP;
1811+
}
17731812
if (disposition == CBM_MCP_SUPERVISED_RESULT_UNSAFE_TERMINAL) {
17741813
execution->unsafe_terminal = true;
17751814
execution->supervision_failed =

‎src/daemon/application.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ typedef void *cbm_daemon_application_update_worker_t;
3131
typedef struct {
3232
void *context;
3333
int (*start)(void *context, const char *args_json, size_t memory_budget_bytes,
34-
const char *marker_file, const char *quarantine_file,
34+
const char *marker_file, const char *quarantine_file, uint64_t duration_origin_ms,
3535
cbm_daemon_application_worker_t *worker_out);
3636
cbm_index_worker_poll_t (*poll)(void *context, cbm_daemon_application_worker_t worker,
3737
const cbm_index_worker_result_t **result_out);

‎src/daemon/bootstrap.c‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,12 @@ static bool bootstrap_worker_argv_exact(int argc, char *const argv[]) {
141141
}
142142
next += 2;
143143
}
144+
if (next < argc && bootstrap_arg_is(argv[next], "--index-worker-publish-fence")) {
145+
if (next + 1 >= argc || !argv[next + 1] || !argv[next + 1][0]) {
146+
return false;
147+
}
148+
next += 2;
149+
}
144150
return next == argc;
145151
}
146152

0 commit comments

Comments
 (0)