Skip to content

Commit 3c7427e

Browse files
authored
Merge pull request #1597 from DeusData/feat/lean-output-contract
feat: make CLI and MCP output lean by default
2 parents ceba427 + 9427f43 commit 3c7427e

47 files changed

Lines changed: 17019 additions & 4170 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -629,7 +629,13 @@ Every MCP tool can be invoked as a local, one-shot command. CLI tools neither st
629629

630630
Commands that mutate graph data use shared OS-backed, per-project locks. This serializes conflicting work from CLI and MCP sessions on the same project while allowing unrelated projects to proceed independently.
631631

632-
When stderr is an interactive terminal, the CLI automatically shows lifecycle and indexing progress. Pass `--progress` to force the same feedback when stderr is redirected or the command is run non-interactively. Progress is written only to stderr; stdout remains reserved for the command result, so pipes and scripts stay machine-safe. Pass `--json` when the full MCP result envelope is needed.
632+
When stderr is an interactive terminal, the CLI automatically shows lifecycle and indexing progress. Pass `--progress` to force the same feedback when stderr is redirected or the command is run non-interactively. Pass `--quiet` to disable automatic terminal progress and ordinary diagnostics while retaining errors; it cannot be combined with `--progress` or outer `cli --verbose`. Routine informational logs are quiet by default; pass outer `cli --verbose` to include them. Progress and logs use stderr while stdout remains reserved for the command result. Read tools return a compact tree by default; pass a tool's `--format json` for machine-readable payload JSON, or outer `--json` for the full MCP envelope.
633+
634+
Large compact-tree tables may start with a response-local `<section>_refs` directory and an explicit `<section>_ref_rule`. A cell such as `@0+handler.go` reconstructs to ref `0`'s prefix plus `handler.go`. References are local to that sibling `<section>` table and expansion is non-recursive: entries inside `<section>_refs` are always literal prefixes. This is limited to declared path and qualified-name columns and activates only when the exact rendered table is at least 15% and 64 bytes smaller and a conservative model-neutral token-shape proxy also improves by at least 1%. Search and trace likewise render direct and prefix-grouped tree shapes and keep the smaller complete representation, so singleton or scattered answers do not pay directory overhead. Keys are declared once per table but never cryptically abbreviated, and `--format json` keeps stable literal strings for machine consumers. Both gates are deterministic; exact token counts still depend on the caller's tokenizer.
635+
636+
Lean responses truncate semantically, never by cutting arbitrary bytes from code or identifiers. Ranked graph rows are retained ahead of raw grep rows and diagnostic summaries; omitted rows/sections report totals, `has_more`, and a strictly advancing continuation offset or cursor. If even the first whole row cannot fit, CBM asks for a higher budget and emits no self-looping cursor. `max_output_tokens` is model-neutral sizing guidance: CBM enforces a deterministic ceiling of four UTF-8 bytes per requested token, so it is not a tokenizer-exact count. Detail flags such as `diagnostics`, `source_mode`, and `detail` opt into heavier fields. `search_code` pages ranked rows with `result_limit`/`result_offset` (`limit` remains a compatibility alias), raw rows with `raw_limit`/`raw_offset`, and directory summaries with `directory_limit`/`directory_offset`. Raw lines default to a UTF-8-safe match-centered preview; each row reports `content_start_byte`, returned/total byte counts, match byte bounds when known, and a content continuation offset. Pass `raw_content_offset` to page the original line without moving the raw-row cursor. `match_limit` and `source_max_lines` bound per-result details, with exact omission metadata. `detect_changes` pages changed files, impacted symbols, and module summaries independently; prefer its snapshot-bound `*_cursor` continuations, which reject changed commits, worktree bytes, graph generation, or semantic arguments instead of silently skipping or duplicating rows.
637+
638+
Every response is standard UTF-8. Identifiers, paths, and raw search previews preserve POSIX byte-string identities: a preserved value containing malformed UTF-8 is emitted reversibly as `@bytes:<lowercase hex of every original byte>`. A valid preserved value that literally begins with the reserved `@bytes:` or `@utf8:` prefix is emitted as `@utf8:<original value>`, so decoding is unambiguous: strip one `@utf8:` prefix for literal UTF-8, or hex-decode one `@bytes:` prefix for original bytes. Ordinary valid UTF-8 is unchanged and pays no output-token overhead. To keep code readable, source bodies replace malformed UTF-8 with U+FFFD; use the pageable raw search preview when byte-exact source inspection is required.
633639

634640
Use `cli <tool> --help` to see the flags generated from that tool's input schema:
635641

@@ -644,7 +650,10 @@ codebase-memory-mcp cli query_graph --project my-project --query 'MATCH (f:Funct
644650

645651
# Force human-readable progress without contaminating stdout.
646652
codebase-memory-mcp cli --progress index_repository --repo-path /path/to/repo
647-
codebase-memory-mcp cli search_graph --project my-project --label Function | jq '.results[].name'
653+
# Suppress automatic terminal progress and non-error diagnostics.
654+
codebase-memory-mcp cli --quiet list_projects --format json
655+
codebase-memory-mcp cli search_graph --project my-project --label Function --format json
656+
codebase-memory-mcp cli list_projects --format json --detail stats | jq '.projects[].name'
648657
```
649658

650659
JSON arguments can also be piped on stdin, for tools that take arguments. A tool whose input schema declares none — `list_projects` — never reads stdin, so it stays responsive when it inherits a pipe the caller never closes (the default for `child_process.spawn` and similar wrappers). Inline JSON remains accepted for backward compatibility but is deprecated in favor of flags, `--args-file`, or stdin.
@@ -664,7 +673,7 @@ JSON arguments can also be piped on stdin, for tools that take arguments. A tool
664673

665674
| Tool | Description |
666675
|------|-------------|
667-
| `search_graph` | Structured search by label, name pattern, file pattern, degree filters. Pagination via limit/offset. |
676+
| `search_graph` | Structural, BM25, and semantic search. Page structural rows with `offset`/`limit` and ranked semantic rows independently with `semantic_offset`/`semantic_limit`. |
668677
| `trace_path` | BFS traversal — who calls a function and what it calls (alias: `trace_call_path`). Depth 1-5. |
669678
| `detect_changes` | Map git diff to affected symbols + blast radius with risk classification. |
670679
| `query_graph` | Execute Cypher-like graph queries (read-only). |
@@ -730,7 +739,7 @@ codebase-memory-mcp config reset auto_index # reset to default
730739
| `CBM_CACHE_DIR` | `~/.cache/codebase-memory-mcp` | Override the database storage directory. All project indexes and config are stored here. One account can use only one canonical cache root at a time; close active CBM sessions/commands before switching it. |
731740
| `CBM_DIAGNOSTICS` | `false` | Set to `1` or `true` to enable the shared daemon's periodic `snapshot.json` and retained `trajectory.ndjson` below a fresh owner-private directory in the system temp directory. Exact paths are logged by `diagnostics.start`. |
732741
| `CBM_DOWNLOAD_URL` | *(GitHub releases)* | Override the download URL for updates. Used for testing or self-hosted deployments. |
733-
| `CBM_LOG_LEVEL` | `info` | Set the minimum log level. Accepted values (case-insensitive): `debug`, `info`, `warn`, `error`, `none` — or their numeric equivalents `0`–`4` matching the internal enum. Thin-frontend messages go to that session's stderr; detached daemon events go to `${CBM_CACHE_DIR}/logs/cbm-daemon.log`. Stdout is reserved for MCP JSON-RPC. |
742+
| `CBM_LOG_LEVEL` | role-aware | Set the minimum log level. Thin MCP/CLI/hook frontends default to `warn`; the detached daemon and its supervised index workers default to `info` so lifecycle and liveness records remain available. Accepted values (case-insensitive): `debug`, `info`, `warn`, `error`, `none` — or their numeric equivalents `0`–`4`. A physical worker retains INFO liveness records even under a stricter override because its private log drives the supervisor's no-progress timeout. Frontend messages go to that session's stderr; detached daemon events go to `${CBM_CACHE_DIR}/logs/cbm-daemon.log`. Stdout is reserved for MCP JSON-RPC. |
734743
| `CBM_WORKERS` | *(detected)* | Override the parallel-indexing worker count returned by `cbm_default_worker_count`. Useful inside containers where `sysconf(_SC_NPROCESSORS_ONLN)` reports host CPUs rather than the cgroup's effective quota. Range 1–256; invalid values are ignored with a warning. |
735744
| `CBM_MEM_BUDGET_MB` | *(detected)* | Override the in-memory graph budget with an explicit cap in MiB, taking precedence over the `ram_fraction × total_RAM` default. Useful on bare-metal hosts without a cgroup limit, or to pin a budget *below* the cgroup limit so headroom is left for sibling processes. Must be a positive integer; it is clamped to detected total RAM (logged as `mem.budget.clamped`), and non-numeric or non-positive values are ignored with a warning (`mem.budget.env.invalid`). |
736745
| `CBM_DUMP_VERIFY_MIN_RATIO` | `0.5` | After indexing, compare persisted SQLite node count to the in-memory dump count. When persisted nodes fall below this fraction of committed nodes (and committed > 50), `index_repository` returns `status:"degraded"` instead of silent `indexed`. Range 0–1; set `0` to disable. Invalid values are ignored with a warning. |

‎docs/CONFIGURATION.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,7 @@ These environment variables affect runtime behavior:
149149
| `CBM_CACHE_DIR` | `~/.cache/codebase-memory-mcp` | Override the cache directory used for indexes, `_config.db`, and UI `config.json`. |
150150
| `CBM_DIAGNOSTICS` | `false` | Enable periodic `snapshot.json` and retained `trajectory.ndjson` below a fresh owner-private directory in the system temp directory. The daemon records the randomized paths in the `diagnostics.start` discovery record (a single JSON line) in `${CBM_CACHE_DIR}/logs/cbm-daemon.log`; that one record is emitted even when `CBM_LOG_LEVEL` suppresses ordinary logging, so the paths always remain discoverable. |
151151
| `CBM_DOWNLOAD_URL` | GitHub releases | Override the update download URL. |
152-
| `CBM_LOG_LEVEL` | `info` | Set the log level to `debug`, `info`, `warn`, `error`, or `none` (or `0`-`4`). Thin-frontend messages use that session's stderr; detached daemon events use `${CBM_CACHE_DIR}/logs/cbm-daemon.log`. |
152+
| `CBM_LOG_LEVEL` | role-aware | Set the log level to `debug`, `info`, `warn`, `error`, or `none` (or `0`-`4`). Thin MCP/CLI/hook frontends default to `warn`; the detached daemon and supervised index workers default to `info`. Physical workers retain INFO liveness records because their private logs drive the supervisor's no-progress timeout. Frontend messages use that session's stderr; detached daemon events use `${CBM_CACHE_DIR}/logs/cbm-daemon.log`. |
153153
| `CBM_RUNTIME_DIR` | `%LOCALAPPDATA%` (Windows), `/private/tmp` (macOS), `/tmp` (other) | Parent directory for the daemon/CLI rendezvous directory, which CBM creates inside it as `cbm-daemon-<uid>` (`cbm-daemon-<key>` on Windows). Set it when the default ancestry cannot pass the private-directory check — see below. `CBM_CACHE_DIR` does **not** move the rendezvous. |
154154
| `CBM_WORKERS` | auto-detected | Override the indexing worker count. |
155155

‎graph-ui/src/components/NodeDetailPanel.test.tsx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,12 @@ describe("NodeDetailPanel code preview + deep-link", () => {
6868
/* …but was NOT parsed into a real <script> element, and did not execute. */
6969
expect(container.querySelector("script")).toBeNull();
7070
expect((window as unknown as { __pwned?: boolean }).__pwned).toBeUndefined();
71+
expect(callToolMock).toHaveBeenCalledWith("get_code_snippet", {
72+
qualified_name: "app::render",
73+
project: "demo",
74+
format: "json",
75+
source_mode: "full",
76+
});
7177
});
7278

7379
it("builds an https GitHub deep-link with URL-encoded path segments", () => {

‎graph-ui/src/components/NodeDetailPanel.tsx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,8 @@ export function NodeDetailPanel({
7676
const res = await callTool<SnippetResult>("get_code_snippet", {
7777
qualified_name: node.qualified_name,
7878
project,
79+
format: "json",
80+
source_mode: "full",
7981
});
8082
setCode(res.source ?? "(source not available)");
8183
} catch (e) {

‎graph-ui/src/components/StatsTab.test.tsx‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ function mockProjectsFetch(extra?: (url: string, init?: RequestInit) => Response
1212
if (overridden) return overridden;
1313
if (url === "/rpc") {
1414
return new Response(JSON.stringify({
15-
result: { content: [{ text: JSON.stringify({ projects: [] }) }] },
15+
result: { content: [{ text: JSON.stringify({ projects: [], has_more: false }) }] },
1616
}), { status: 200, headers: { "Content-Type": "application/json" } });
1717
}
1818
if (url.startsWith("/api/ui-config")) {
@@ -221,8 +221,11 @@ describe("StatsTab index modal", () => {
221221
root_path: "/repo",
222222
indexed_at: "2026-01-01T00:00:00Z",
223223
}],
224+
has_more: false,
224225
}
225-
: { node_labels: [], edge_types: [], total_nodes: 0, total_edges: 0 };
226+
: {
227+
node_labels: [], edge_types: [], total_nodes: 0, total_edges: 0, has_more: false,
228+
};
226229
return new Response(JSON.stringify({
227230
result: { content: [{ text: JSON.stringify(result) }] },
228231
}), { status: 200, headers: { "Content-Type": "application/json" } });
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
/* @vitest-environment jsdom */
2+
import { renderHook, waitFor } from "@testing-library/react";
3+
import { beforeEach, describe, expect, it, vi } from "vitest";
4+
import { useProjects } from "./useProjects";
5+
6+
const callToolMock = vi.fn();
7+
vi.mock("../api/rpc", () => ({
8+
callTool: (...args: unknown[]) => callToolMock(...args),
9+
}));
10+
11+
describe("useProjects machine-readable pagination", () => {
12+
beforeEach(() => callToolMock.mockReset());
13+
14+
it("requests JSON and merges every project and schema page", async () => {
15+
callToolMock.mockImplementation(async (name: string, args: Record<string, unknown>) => {
16+
if (name === "list_projects") {
17+
if (args.offset === 0) {
18+
return {
19+
projects: [{ name: "alpha", root_path: "/alpha", indexed_at: "now" }],
20+
has_more: true,
21+
next_offset: 1,
22+
};
23+
}
24+
return {
25+
projects: [{ name: "beta", root_path: "/beta", indexed_at: "now" }],
26+
has_more: false,
27+
};
28+
}
29+
if (name === "get_graph_schema" && args.project === "alpha") {
30+
if (args.offset === 0) {
31+
return {
32+
node_labels: [{ label: "Function", count: 3 }],
33+
edge_types: [],
34+
total_nodes: 3,
35+
total_edges: 2,
36+
has_more: true,
37+
next_offset: 1,
38+
};
39+
}
40+
return {
41+
node_labels: [],
42+
edge_types: [{ type: "CALLS", count: 2 }],
43+
total_nodes: 3,
44+
total_edges: 2,
45+
has_more: false,
46+
};
47+
}
48+
return {
49+
node_labels: [{ label: "Class", count: 1 }],
50+
edge_types: [],
51+
total_nodes: 1,
52+
total_edges: 0,
53+
has_more: false,
54+
};
55+
});
56+
57+
const { result } = renderHook(() => useProjects());
58+
await waitFor(() => expect(result.current.loading).toBe(false));
59+
60+
expect(result.current.projects).toHaveLength(2);
61+
expect(result.current.projects[0].schema?.node_labels).toEqual([
62+
{ label: "Function", count: 3 },
63+
]);
64+
expect(result.current.projects[0].schema?.edge_types).toEqual([
65+
{ type: "CALLS", count: 2 },
66+
]);
67+
expect(callToolMock).toHaveBeenCalledWith("list_projects", {
68+
format: "json",
69+
detail: "stats",
70+
limit: 500,
71+
offset: 0,
72+
});
73+
expect(callToolMock).toHaveBeenCalledWith("list_projects", {
74+
format: "json",
75+
detail: "stats",
76+
limit: 500,
77+
offset: 1,
78+
});
79+
expect(callToolMock).toHaveBeenCalledWith("get_graph_schema", {
80+
project: "alpha",
81+
format: "json",
82+
limit: 500,
83+
offset: 1,
84+
});
85+
});
86+
});

‎graph-ui/src/hooks/useProjects.ts‎

Lines changed: 66 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,70 @@ interface ProjectInfo {
77
schema: SchemaInfo | null;
88
}
99

10+
interface ProjectPage {
11+
projects?: Project[];
12+
has_more?: boolean;
13+
next_offset?: number;
14+
}
15+
16+
interface SchemaPage extends SchemaInfo {
17+
has_more?: boolean;
18+
next_offset?: number;
19+
}
20+
21+
const PAGE_LIMIT = 500;
22+
23+
function nextPageOffset(page: { has_more?: boolean; next_offset?: number }, offset: number) {
24+
if (typeof page.has_more !== "boolean") {
25+
throw new Error("Invalid pagination response");
26+
}
27+
if (!page.has_more) return null;
28+
if (!Number.isInteger(page.next_offset) || page.next_offset! <= offset) {
29+
throw new Error("Invalid pagination response");
30+
}
31+
return page.next_offset!;
32+
}
33+
34+
async function fetchAllProjects(): Promise<Project[]> {
35+
const projects: Project[] = [];
36+
let offset = 0;
37+
for (;;) {
38+
const page = await callTool<ProjectPage>("list_projects", {
39+
format: "json",
40+
detail: "stats",
41+
limit: PAGE_LIMIT,
42+
offset,
43+
});
44+
projects.push(...(page.projects ?? []));
45+
const next = nextPageOffset(page, offset);
46+
if (next === null) return projects;
47+
offset = next;
48+
}
49+
}
50+
51+
async function fetchFullSchema(project: string): Promise<SchemaInfo> {
52+
const nodeLabels: SchemaInfo["node_labels"] = [];
53+
const edgeTypes: SchemaInfo["edge_types"] = [];
54+
let firstPage: SchemaPage | null = null;
55+
let offset = 0;
56+
for (;;) {
57+
const page = await callTool<SchemaPage>("get_graph_schema", {
58+
project,
59+
format: "json",
60+
limit: PAGE_LIMIT,
61+
offset,
62+
});
63+
firstPage ??= page;
64+
nodeLabels.push(...(page.node_labels ?? []));
65+
edgeTypes.push(...(page.edge_types ?? []));
66+
const next = nextPageOffset(page, offset);
67+
if (next === null) {
68+
return { ...firstPage, node_labels: nodeLabels, edge_types: edgeTypes };
69+
}
70+
offset = next;
71+
}
72+
}
73+
1074
interface UseProjectsResult {
1175
projects: ProjectInfo[];
1276
loading: boolean;
@@ -23,16 +87,13 @@ export function useProjects(): UseProjectsResult {
2387
setLoading(true);
2488
setError(null);
2589
try {
26-
const result = await callTool<{ projects: Project[] }>("list_projects");
27-
const list = result.projects ?? [];
90+
const list = await fetchAllProjects();
2891

2992
/* Fetch schema for each project */
3093
const infos: ProjectInfo[] = await Promise.all(
3194
list.map(async (p) => {
3295
try {
33-
const schema = await callTool<SchemaInfo>("get_graph_schema", {
34-
project: p.name,
35-
});
96+
const schema = await fetchFullSchema(p.name);
3697
return { project: p, schema };
3798
} catch {
3899
return { project: p, schema: null };

0 commit comments

Comments
 (0)