Skip to content

feat(mcp): per-message caching for prune_tools, integration tests#2333

Merged
bug-ops merged 1 commit intomainfrom
per-message-caching-prune-tool
Mar 28, 2026
Merged

feat(mcp): per-message caching for prune_tools, integration tests#2333
bug-ops merged 1 commit intomainfrom
per-message-caching-prune-tool

Conversation

@bug-ops
Copy link
Copy Markdown
Owner

@bug-ops bug-ops commented Mar 28, 2026

Summary

  • Add PruningCache to agent loop state in zeph-core, keyed on (message_content_hash, tool_list_hash), eliminating redundant LLM calls when tool list and user message are unchanged between agent loop iterations
  • Cache is single-slot (~48 bytes stack + ~3 KB heap on hit), with CachedResult::Failed negative caching to prevent LLM retry storms on persistent pruning failures
  • Wire caching into rebuild_system_prompt (once per turn); apply_pruned_mcp_tools updates shared_tools RwLock independently of sync_mcp_executor_tools
  • Err branch of pruning match calls sync_mcp_executor_tools to self-heal shared_tools on failure
  • Add 13 integration tests covering always_include partition, max_tools cap, min_tools_to_prune early-return (replaces tautology), LLM failure, parse error, and all cache invalidation paths

Test plan

  • cargo nextest run -p zeph-mcp -p zeph-core --lib — 1455/1455 passed
  • cargo +nightly fmt --check — clean
  • cargo clippy -p zeph-mcp -p zeph-core -- -D warnings — clean (workspace-level pre-existing errors are not introduced by this PR, verified by checking main without changes)

Closes #2298, #2300

@github-actions github-actions bot added documentation Improvements or additions to documentation rust Rust code changes enhancement New feature or request core zeph-core crate size/XL Extra large PR (500+ lines) labels Mar 28, 2026
@bug-ops bug-ops enabled auto-merge (squash) March 28, 2026 09:09
@bug-ops bug-ops force-pushed the per-message-caching-prune-tool branch from 28448ca to 988b6ee Compare March 28, 2026 09:15
, #2300)

Add PruningCache to agent loop state, keyed on (message_content_hash,
tool_list_hash), to avoid redundant LLM calls when the tool list and
user message have not changed between agent loop iterations.

- PruningCache with CachedResult::Ok/Failed enum; single-slot, ~48 bytes
  on stack plus at most max_tools McpTool clones (~3 KB) on hit
- prune_tools_cached() wrapper; negative caching prevents LLM retry
  storms on persistent pruning failures
- tool_list_hash hashes full metadata (name, description, input_schema)
  sorted by qualified name using BTreeMap-backed serde_json::to_vec;
  sentinel byte on schema serialization failure
- Cache wired into rebuild_system_prompt (once per turn); apply_pruned_
  mcp_tools updates shared_tools RwLock without calling sync_mcp_executor_
  tools; both write sites carry cross-referencing ordering contract comments
- Cache reset at: process_user_message_inner start, check_tool_refresh,
  handle_mcp_add, handle_mcp_remove
- Err branch of pruning match in rebuild_system_prompt calls
  sync_mcp_executor_tools to self-heal shared_tools on pruning failure

Add 13 integration tests covering: always_include partition (including
cross-server bare-name matching), max_tools cap, min_tools_to_prune
early-return (replaces tautology), LLM failure propagation, parse error,
positive cache hit, cache miss on message/tool-list change, negative
cache hit (skips LLM on second call), and reset-clears-negative.

Closes #2298, #2300
@bug-ops bug-ops force-pushed the per-message-caching-prune-tool branch from 988b6ee to 65c7364 Compare March 28, 2026 09:22
@bug-ops bug-ops merged commit 3f10e88 into main Mar 28, 2026
25 checks passed
@bug-ops bug-ops deleted the per-message-caching-prune-tool branch March 28, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core zeph-core crate documentation Improvements or additions to documentation enhancement New feature or request rust Rust code changes size/XL Extra large PR (500+ lines)

Projects

None yet

1 participant