feat(agent): add OTLP observability plugin (observability/otel)

Summary

Adds a third bundled observability plugin, observability/otel, that
exports Hermes's existing observability events over OpenTelemetry (OTLP).
It slots in beside the two shipped plugins — observability/langfuse and
observability/nemo_relay — and makes no changes to Hermes core. Hermes
already produces exactly the right events (a turn, per-LLM-API-call
usage/cost/finish-reason, per-tool-call results); this plugin adds an OTLP sink
for them and exports three complementary signals:

1. GenAI-convention spans (gen_ai.*) — the vendor-neutral trace model, so
   telemetry flows to any OTel backend (an OpenTelemetry Collector, Jaeger,
   Grafana Tempo, Honeycomb, …) with no lock-in.
2. Dashboard-shaped OTLP log records (user_prompt / session_start /
   api_response / tool_result) — the event model agent-coding dashboards
   aggregate on, so a Hermes session shows up in Activity / Sessions /
   Leaderboards views — including its prompts — the same way a Claude Code /
   Codex / Copilot session does.
3. Local-model cost — when the backend returns no price (Ollama / HF / vLLM
   local models), cost is estimated from the model's parameter size, so spans
   and log records carry a real cost_usd instead of $0.

Why

observability/langfuse and observability/nemo_relay cover Langfuse and the
NVIDIA NeMo relay, but there is no OpenTelemetry path. OTLP is the de-facto open
standard; an OTLP plugin lets Hermes telemetry flow to any OTel backend with
no vendor lock-in. Emitting both the standard gen_ai.* spans and the
log-record event model means Hermes interoperates with conventions-aware tracing
backends and populates agent-coding dashboards — and the model-size cost
fallback means local-model runs report real cost rather than $0.

What genai_otel adds (beyond a vanilla OTel SDK or other GenAI instrumentors)

The plugin runs on a plain OTel SDK by default (spans, tokens, latency,
finish reasons, log records — portable to any OTLP backend). Routing through the
optional genai-otel-instrument emitter additionally unlocks four signals that
are genuinely unique — neither a raw OTel SDK nor the common GenAI
instrumentors (OpenLLMetry / Traceloop, OpenInference / Arize, OpenLIT, Langfuse)
emit them:

• On-prem GPU metrics — per-GPU utilization, power, memory, temperature, clocks, throttle state (nvidia-ml-py / amdsmi).
• Energy + CO2 — energy_kwh, co2_emissions_gco2e, region-aware (codecarbon).
• Local-model cost — Ollama / HF / vLLM cost estimated from parameter size when the backend returns no price.
• Inline eval / guardrails — PII, toxicity, bias, prompt-injection, restricted-topics, hallucination scored as span attributes + metrics at instrumentation time (no separate eval pipeline).

All of it runs in-process and on-prem — no data leaves the host, no SaaS
observability backend, no separate eval service. The token/latency/cost fields
stay standard OTel GenAI conventions; the four signals above are extra
attributes/metrics a conventions-aware backend can use or ignore. See the plugin
README for the full comparison table + setup.

What changed

New plugin only — one directory plus its test, no core edits:

plugins/observability/otel/__init__.py     Hermes plugin: register(ctx) + lifecycle hooks
plugins/observability/otel/emitter.py      Hermes-agnostic OTel GenAI span builder
plugins/observability/otel/log_emitter.py  Hermes-agnostic OTel log-record builder (dashboard events)
plugins/observability/otel/cost.py         local-model cost estimation (model size → price tier)
plugins/observability/otel/provider.py     tracer + logger bootstrap (genai_otel preferred, OTel SDK fallback)
plugins/observability/otel/plugin.yaml     manifest
plugins/observability/otel/README.md       enable / configure / verify / disable
tests/plugins/test_otel_plugin.py          manifest + in-memory-exporter tests (39 tests)

Span mapping

invoke_agent          per Hermes turn   (gen_ai.conversation.id = session id)
  ├── chat            per LLM API call  (gen_ai.usage.*_tokens, cost_usd,
  │                                      gen_ai.response.finish_reasons, TTFT)
  └── execute_tool    per tool call     (gen_ai.tool.name, gen_ai.tool.type, errors)

gen_ai.system = hermes on every span; service.name defaults to
agent.coding.hermes (configurable). Prompt/response content is captured only
when HERMES_OTEL_CAPTURE_CONTENT=true (off by default — privacy-gated).

Log-record mapping (emitted in parallel with the spans)

user_prompt mirrors Claude Code's claude_code.user_prompt, so the dashboard's
prompt count and prompt-text drill-down populate for Hermes sessions. It
is emitted from pre_llm_call (Hermes delivers the prompt there, not at
on_session_start), once per turn. Without it a session would show tool calls
and API responses but no prompts — the dashboard reads log records, and the
prompt otherwise lives only on the invoke_agent span as gen_ai.prompt, which
log-record-based dashboards never see.

Identity (service.name, user.id, and any team.id/organization.id from
OTEL_RESOURCE_ATTRIBUTES) is placed on the OTel resource so per-user /
per-org leaderboard aggregations work. Same privacy gate as spans. Log emission
is best-effort — a logs-pipeline failure never disables spans.

Cost estimation

cost.py mirrors the genai-otel-instrument methodology: parse the parameter
size from the model name (llama3.1:8b → 8 B, qwen3:0.6b → 0.6 B,
smollm2:360m → 0.36 B), map it to a per-1K-token price tier, and compute
cost = in_tok/1000 * promptPrice + out_tok/1000 * completionPrice. Used only
as a fallback when the agent reported no cost (an agent-provided cost_usd
always wins); cost is omitted entirely when the size can't be inferred (no fake
$0).

Hook wiring (matches the bundled observability plugins)

Confirmed assumptions

The scaffold for this plugin was written against the observable behaviour of
the two shipped observability plugins, with five integration points flagged for
confirmation against the real Hermes plugin contract. All five were confirmed
against plugins/observability/langfuse, plugins/observability/nemo_relay,
and website/docs/guides/build-a-hermes-plugin.md, and the plugin was adapted
to the real contract:

The Hermes-agnostic span-mapping core (emitter.py), log-record-mapping core
(log_emitter.py), cost estimator (cost.py), and the tracer/logger bootstrap
(provider.py) are all Hermes-free and unit-tested in isolation — only
__init__.py (the plugin glue) is written against the real Hermes contract.

Design notes

• Minimal hard dependencies. Only opentelemetry-{api,sdk} + the OTLP HTTP
  exporter (which provides both the span and log exporters). No new
  dependency for the logs/cost scope. genai-otel-instrument is an optional
  extra (preferred, auto-detected at runtime) — imported, never vendored, so the
  MIT license stays clean.
• Fails open like the siblings. If the OTel SDK isn't importable, the hooks
  no-op (cached _INIT_FAILED sentinel). A malformed event or a dead OTLP
  endpoint degrades to a no-op tracer — it never raises out of a hook. The logs
  pipeline and cost estimator are independently best-effort: if build_logger()
  or cost estimation fails, spans still emit.
• Logs flushed at exit. The plugin owns its LoggerProvider, so it registers
  an atexit force_flush + shutdown — without it the batch processor can
  drop late-turn records (the api_response / tool_result emitted just before
  the CLI exits).
• Thread-safe singleton. The emitter (and the log emitter built alongside it)
  is built once under a double-checked lock (concurrent agent sessions don't race
  to build two providers), per the Build-a-Plugin guide's lazy-singleton guidance.
• Hermes coupling isolated to one file (__init__.py). emitter.py,
  log_emitter.py, and cost.py are pure and unit-tested with in-memory
  exporters — no Hermes import, no network.

Testing instructions

# From the hermes-agent checkout, in the project venv (`uv sync --extra all`,
# plus the `dev` group for pytest/ruff):
scripts/run_tests.sh tests/plugins/test_otel_plugin.py    # 39 tests
python scripts/check-windows-footguns.py plugins/observability/otel tests/plugins/test_otel_plugin.py
ruff check plugins/observability/otel tests/plugins/test_otel_plugin.py

The test suite uses OTel's in-memory span and log exporters (no network) and
asserts: the produced spans carry the correct gen_ai.* attributes with correct
parent/child nesting (chat and execute_tool under invoke_agent); the log
records carry the dashboard event.name / token / cost / tool fields; the
user_prompt record carries the prompt text (content-gated) + prompt_length
and is emitted exactly once per turn even across a tool loop; local-model
cost is estimated from model size across the price tiers and boundaries (and is
None when unknown); content-capture is gated; every path fails open; and
register(ctx) wires exactly the hooks declared in plugin.yaml. It drives the
full hook surface end-to-end from **kwargs-style Hermes events, asserting spans,
logs, and cost together.

End-to-end (optional, requires a collector)

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
hermes plugins enable observability/otel
HERMES_OTEL_ENDPOINT=http://localhost:4318 hermes chat -q "hello"
# → an "invoke_agent" trace with chat/execute_tool children appears in your OTel
#   backend, AND session_start/api_response/tool_result log records (with cost)
#   appear on the logs endpoint.

Platforms tested

Run in the project venv (CPython 3.11.15) on Windows 11:

scripts/run_tests.sh itself probes a POSIX-layout venv (<venv>/bin/activate),
which a native-Windows venv (<venv>/Scripts/) doesn't have, so on Windows the
runner it execs — scripts/run_tests_parallel.py — was invoked directly with the
same deterministic env the wrapper sets (TZ=UTC, LANG=C.UTF-8,
PYTHONHASHSEED=0). On Linux/macOS/WSL2 scripts/run_tests.sh works unchanged.

• Linux / macOS / WSL2: no OS-specific code paths; the plugin performs no
  path I/O (network export only — provider.py deliberately avoids
  get_hermes_home() since there is no filesystem work), uses no POSIX-only
  calls, and the footgun checker is clean. CI runs all three gates on Ubuntu.

Scope (deliberately narrow)

• ✅ One new bundled plugin, beside the existing two observability plugins.
• ✅ Exports OTel spans (gen_ai.*) and dashboard-shaped OTLP log
  records and local-model cost (model-size pricing) — all three driven
  by the same HERMES_OTEL_* config, no new config surface.
• ✅ No changes to Hermes core, config schema, or the other plugins.
• ✅ Standard OTel GenAI conventions for spans — no bespoke schema; the log
  records use the established agent-coding event model.
• ✅ Opt-in (hermes plugins enable observability/otel); fails open when the
  OTel SDK is absent. The logs/cost paths are best-effort — they never disable
  spans.
• ❌ No new core dependencies forced on users who don't enable the plugin (the
  logs + cost scope adds zero dependencies beyond the OTLP HTTP exporter the
  spans already need).

Related issues

No tracking issue — this is a self-initiated contribution that fills the
OpenTelemetry gap alongside the two shipped observability plugins. Context:

• Hermes built-in observability plugins (observability/langfuse,
  observability/nemo_relay) — this is the OTel sibling.
• OpenTelemetry GenAI semantic conventions (the attribute schema emitted).