[FEATURE] Close three observability gaps: GenAI token metrics, controller reconcile Events, and a controller log→OTLP bridge (all gated default-OFF) #2148

Affected Service(s): Multiple services / System-wide (App Service + Controller Service)

📋 Prerequisites

• I have searched the existing issues to avoid creating a duplicate
• By submitting this issue, you agree to follow the Code of Conduct

📝 Feature Summary

Add three small, independent, default-OFF observability improvements that build directly on the OTel pipeline landed in #1433 / #1292:

1. GenAI token-usage metrics in the ADK (go/adk) — the first MeterProvider in kagent, emitting the OpenTelemetry GenAI-semconv instrument gen_ai.client.token.usage.
2. Kubernetes Events on reconcile for the Agent / ModelConfig / MCPServerTool / RemoteMCPServer controllers (go/core), mirroring the pattern the harness/substrate controller already uses.
3. Controller log → OTLP bridge (go/core) — an otelzap tee on the controller zap core so logs can be exported over OTLP additively, without changing stdout.

Each is additive and byte-identical to current behavior when its gate is unset. Happy to split this into three separate PRs (they're fully independent) — filing one issue first to align on approach per CONTRIBUTING.

❓ Problem Statement / Motivation

kagent already has solid tracing (controller distributed tracing + A2A propagation, #1433) and standardized OTel env-var naming (#1292), but three observability primitives are still missing on main:

• No metrics pipeline at all. There is no MeterProvider anywhere in the repo (go/adk/pkg/telemetry/ contains only attributes.go + tracing.go). Every model adapter already computes token usage — go/adk/pkg/models/{openai,anthropic,bedrock,ollama,sapaicore}_adk.go all populate genai.GenerateContentResponseUsageMetadata (PromptTokenCount / CandidatesTokenCount / ThoughtsTokenCount) — but that data is never recorded to a metric, so operators can't graph or alert on token spend / cost per model or provider without scraping traces. This is the still-unchecked "Metrics: LLM calls/token-count" item from the (now closed) observability EPIC [EPIC] Observability improvements #130, and the abandoned Prometheus attempt in Enable /metrics endpoint with prometheus metrics #1412 (closed unmerged) shows the demand is real; this proposes a standards-aligned OTLP/semconv take instead of a scraped /metrics endpoint.
• Reconcile outcomes are invisible via kubectl. The Agent, ModelConfig, MCPServerTool, and RemoteMCPServer controllers emit no k8s Events (none reference EventRecorder/.Event(). Users debugging a bad Agent/ModelConfig get nothing from kubectl describe, even though the agentharness-substrate-controller already demonstrates the Event pattern in this codebase.
• Controller logs can't be shipped over OTLP. go/core/internal/telemetry/ has only tracing.go. The ADK side already honors OTEL_LOGGING_ENABLED, but the controller's zap logs remain stdout-only, so log/trace correlation isn't possible in the controller.

Who benefits: anyone running kagent in a cluster with an OTel collector / ClickHouse / Langfuse / Datadog backend who wants token-cost dashboards, kubectl-visible reconcile status, and correlated controller logs.

💡 Proposed Solution

Three independent changes, each gated and additive. All reuse existing conventions.

1. GenAI token metrics (go/adk)

• New pkg/telemetry/metrics.go: a MeterProvider + OTLP metric exporter that reuses the existing resolveEndpoint(...) / resolveOTLPProtocol(...) helpers already in tracing.go, so protocol/endpoint resolution is identical to traces.
• Emit gen_ai.client.token.usage — Int64 histogram, unit {token} — with attributes gen_ai.token.type (input/output), gen_ai.request.model, gen_ai.provider.name, per GenAI semantic conventions.
• Record from the A2A executor loop where per-event usage metadata is already surfaced (pkg/a2a/executor.go): input = PromptTokenCount, output = CandidatesTokenCount + ThoughtsTokenCount. Model/provider labels resolved at startup (cmd/main.go).
• Gate: OTEL_METRICS_ENABLED=true (+ the standard OTEL_EXPORTER_OTLP[_METRICS]_ENDPOINT/_PROTOCOL), matching the existing OTEL_TRACING_ENABLED / OTEL_LOGGING_ENABLED gates. Unit test included.

2. Kubernetes Events on reconcile (go/core)

• Add an EventRecorder to the Agent / ModelConfig / MCPServerTool / RemoteMCPServer controllers: Normal Accepted / ToolsDiscovered, Warning ReconcileFailed / ValidationFailed — mirroring the existing agentharness-substrate-controller pattern. Nil-guarded so it's a no-op if no recorder is wired.
• RBAC is already covered by the leader-election role (events: create,patch).
• Gate: none needed — it's a purely additive status surface.

3. Controller log → OTLP bridge (go/core)

• New internal/telemetry/logging.go: an otelzap tee installed on the controller zap core via RawZapOpts(zap.WrapCore(NewTee(...))). Stdout logging is preserved; OTLP is additive.
• Gate: OTEL_LOGGING_ENABLED=true (same gate name already used ADK-side).

Acceptance bar: everything default-OFF, additive, unit-tested, make lint + make test green, DCO sign-off on every commit.

🔄 Alternatives Considered

• Prometheus /metrics endpoint (Enable /metrics endpoint with prometheus metrics #1412). A prior attempt exposed agent LLM/token metrics via a scraped Prometheus endpoint; it was closed unmerged. OTLP histograms fit kagent's existing OTLP-first pipeline (same exporter/env-var machinery as traces), avoid a second telemetry transport, and align with GenAI semconv — so downstream backends (ClickHouse, Langfuse, Datadog, Honeycomb) get token metrics through the collector they already run.
• Status conditions instead of Events (gap 2). Conditions and Events are complementary; Events give the kubectl describe timeline that's missing today and are cheap/additive. Conditions could be a follow-up.
• Structured stdout log scraping instead of an OTLP bridge (gap 3). Works, but loses trace/log correlation that the OTLP path (already used for traces via feat: instrument controller with distributed tracing and A2A trace propagation #1433) provides for free.

📚 Additional Context

• Builds on feat: instrument controller with distributed tracing and A2A trace propagation #1433 (controller tracing + A2A propagation, merged) and Standardize OpenTelemetry environment variable naming #1292 (OTel env-var standardization, merged); related to open Helm OTLP-config work in feat(helm): expose OTEL headers, service name, and resource attributes #2128 / discussion helm/kagent: declarative OTLP headers + service name + resource attributes (unblocks Langfuse Cloud / Honeycomb / Datadog / ...) #2127 (headers / service name / resource attributes) — these three changes slot alongside that config surface.
• Revisits the metrics items from the closed EPIC [EPIC] Observability improvements #130 and the abandoned Enable /metrics endpoint with prometheus metrics #1412.

🙌 Contribution

• I am willing to submit a PR for this feature (as up to 3 independent PRs, or one, per maintainer preference).