Skip to content

[Story] Telemetry for dynamic tool loading #582

Open

Description

@edelauna

Context

We need data to know whether dynamic loading is actually helping: how often the threshold is hit, how large the attached set ends up after the rolling window, how often the model calls `mcp_load`, and which ranker tier fired. Without telemetry we can't tune the defaults.

Depends on #578, #579, #580.

Developer Notes

  • Events to emit (via the existing telemetry pipeline — check `src/services/telemetry/`):
    • `dynamic_tools.task_start`: `{ scopedCount, attachedCount, ranker: "embeddings" | "bm25" }`
    • `dynamic_tools.mcp_load_invoked`: `{ taskTurn, queryLength, returnedCount, alreadyAttachedCount }`
    • `dynamic_tools.rolling_window_step`: `{ turn, addedCount, totalAttached }` (only when something changes)
    • `dynamic_tools.ranker_fallback`: `{ from: "embeddings", to: "bm25", reason }`
  • Respect existing telemetry opt-out.
  • Emit from `src/services/tools/ToolRouter.ts` (most events) and `src/core/tools/mcpLoadTool.ts` (`mcp_load_invoked`).

Acceptance Criteria

  • All four event types fire in integration tests
  • Opt-out correctly suppresses all events
  • No telemetry emitted when dynamic loading is inactive (under threshold)

Activity

  1. changed the title [-][Story] W2.2 — Telemetry for dynamic tool loading[/-] [+][Story] Telemetry for dynamic tool loading[/+] on Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions