Skip to content

feat: usage-aware tool ranking for LLM agent discovery - #43

Merged
knowsuchagency merged 4 commits into
knowsuchagency:mainfrom
Christian-Sidak:feat/usage-aware-tool-ranking
Apr 6, 2026
Merged

knowsuchagency merged 4 commits into
knowsuchagency:mainfrom
Christian-Sidak:feat/usage-aware-tool-ranking

Conversation

@Christian-Sidak

Copy link
Copy Markdown
Contributor

Summary

Implements the feature described in #31: track tool invocations locally and use that data to rank --list output, reducing token costs for LLM agents working with large MCP servers.

  • --sort usage|recent|alpha|default -- control list ordering by call frequency, recency, alphabetical, or insertion order
  • --top N -- show only the top N tools (the key feature for agents: --list --top 10 --compact yields ~20 tokens instead of ~1,400)
  • --compact -- space-separated tool names only, no descriptions

Usage data is stored in ~/.cache/mcp2cli/usage.json with the format from the issue. Last-write-wins for file locking per maintainer approval. When usage data exists for a source, the default sort is usage; otherwise insertion order is preserved.

Design decisions (per discussion in #31)

  1. Single module -- all code stays in __init__.py as approved
  2. Last-write-wins -- no fcntl or sqlite, simple JSON read/write
  3. Default to usage sort when usage data exists for the source

Hooks

Usage is recorded after successful execution in:

  • handle_mcp() -- MCP HTTP and stdio modes
  • _handle_openapi_mode() -- OpenAPI mode
  • handle_graphql() -- GraphQL mode

LLM agent workflow

# Before: ~1,400 tokens for 96 tools
mcp2cli @myapi --list

# After: ~20 tokens for the 10 most-used tools
mcp2cli @myapi --list --top 10 --compact

Test plan

  • 24 new tests in tests/test_usage.py covering:
    • Usage file load/save/corruption handling
    • Recording usage (increment, multiple tools, multiple sources)
    • Source hash determinism
    • All four sort modes (default, alpha, usage, recent)
    • --compact output for MCP, OpenAPI, and GraphQL listing
    • --top N truncation
    • Combined --sort + --top + --compact
  • All 112 existing tests pass (2 pre-existing failures in test_helpers.py::TestCaching unrelated to this PR)

Closes #31

Christian-Sidak and others added 4 commits April 5, 2026 18:17
Track tool invocations locally in ~/.cache/mcp2cli/usage.json and use
that data to rank --list output. Adds three new CLI flags:

- --sort usage|recent|alpha|default: control list ordering
- --top N: show only the top N tools (key for LLM agent token savings)
- --compact: space-separated names only, no descriptions

When usage data exists for a source, the default sort becomes "usage"
(most-called first). Otherwise insertion order is preserved.

Usage is recorded after successful execution in handle_mcp(),
_handle_openapi_mode(), and handle_graphql().

Closes knowsuchagency#31
The sort_commands lookup and record_usage in _handle_openapi_mode used
different key derivations, which could cause recorded usage to not match
during sorting. Unify on `tool_name or graphql_field_name or name`.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@knowsuchagency

Copy link
Copy Markdown
Owner

Thanks for the contribution!

@knowsuchagency
knowsuchagency merged commit 42c324e into knowsuchagency:main Apr 6, 2026
knowsuchagency added a commit that referenced this pull request Apr 6, 2026
Document the usage-aware tool ranking feature added in #43: new CLI
reference entries and a usage examples section showing the LLM agent
workflow.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
knowsuchagency added a commit that referenced this pull request Apr 6, 2026
…44)

Document the usage-aware tool ranking feature added in #43: new CLI
reference entries and a usage examples section showing the LLM agent
workflow.

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: usage-aware tool ranking for LLM agent discovery (--top, --sort)

2 participants