Skip to content

Zero metrics for queue lanes, MCP reconnect, LLM 429 — only HTTP is instrumented #3495

Description

@houko

Summary

The only metrics::counter! / metrics::histogram! call sites in the entire repo are in librefang-telemetry/src/metrics.rs:104,112 (HTTP request count + latency). Queue-lane saturation, MCP reconnect rate, LLM provider 429s, and tool-call failures all have zero metrics, despite the codebase recording 429s into SQLite for cross-restart durability.

Location

  • crates/librefang-telemetry/src/metrics.rs:104,112 — only metrics call sites.
  • crates/librefang-runtime/src/command_lane.rsCommandQueue lane semaphore acquire has no instrumentation.
  • crates/librefang-kernel/src/kernel/mod.rs:13402 — MCP reconnect logged at debug! (below default INFO filter), uncounted.
  • crates/librefang-llm-drivers/src/shared_rate_guard.rs:358record_429_from_headers writes SQLite but no counter.

Failure mode

  • Trigger lane saturating (default trigger_lane: 8) is invisible in Grafana/Prometheus — operators can't see "queue building up".
  • MCP server flapping every 30s shows nothing on dashboards; only RUST_LOG=debug reveals it.
  • Anthropic/OpenAI 429 climb is durable in SQLite but never surfaces as a counter — capacity-planning blind.

Fix direction

Concrete additions:

  • librefang_queue_wait_seconds{lane} histogram around Lane semaphore acquire in command_lane.rs.
  • librefang_mcp_reconnect_total{server,outcome} counter at kernel/mod.rs:13401-13403.
  • librefang_llm_provider_errors_total{provider,status} counter at shared_rate_guard.rs:358.
  • librefang_tool_call_total{tool,outcome} counter where agent_loop.rs records tool results.

Each addition is one line via the metrics macros that are already in the workspace.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiREST/WS endpoints and dashboardarea/runtimeAgent loop, LLM drivers, WASM sandboxhas-prA pull request has been linked to this issueseverity/highSignificant functional, security, or performance impact

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions