TraceVerde

The most comprehensive OpenTelemetry auto-instrumentation library for LLM/GenAI applications

Trace from OpenTelemetry traces. Verde meaning green - for sustainable, transparent AI observability.

Documentation | Examples | Discord | PyPI

---

Get Started in 30 Seconds

pip install genai-otel-instrument

import genai_otel
genai_otel.instrument()

# Your existing code works unchanged - traces, metrics, and costs are captured automatically
import openai
client = openai.OpenAI()
response = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello!"}])

That's it. No wrappers, no decorators, no config files. Every LLM call, database query, and agent interaction is automatically traced with full cost breakdown.

Why TraceVerde?

Feature	TraceVerde	OpenLIT	Traceloop/OpenLLMetry	Langfuse	Galileo	Arize (Phoenix)	Opik (Comet)
Zero-code setup	Yes	Yes	Yes	SDK required	SDK required	SDK / auto (OpenInference)	SDK required
LLM providers	20+	25+	15+	Via integrations	Via integrations	Via integrations	Via integrations
Multi-agent frameworks	8 (CrewAI, LangGraph, ADK, AutoGen, OpenAI Agents, Pydantic AI, etc.)	Limited	Limited	Limited	Limited	Limited	Limited
Cost tracking	Automatic (1,050+ models)	Manual config	Manual config	Manual config	Yes	Yes	Yes
GPU metrics (NVIDIA + AMD)	Yes	No	No	No	No	No	No
MCP tool instrumentation	Yes (databases, caches, vector DBs, queues)	Limited	Limited	No	No	No	No
Evaluation (PII, toxicity, bias, hallucination, prompt injection)	Built-in (6 detectors)	No	No	Separate service	Extensive (core focus)	Built-in (Phoenix evals)	Built-in (core focus)
OpenTelemetry native	Yes	Yes	Yes	Partial	Partial	Yes (OpenInference)	Partial
Self-hosted / on-prem	Yes (fully local)	Yes	Yes	Yes	Enterprise tier	Yes (Phoenix)	Yes
License	Apache-2.0	Apache-2.0	Apache-2.0	MIT	Proprietary	Open-source + Commercial	Apache-2.0

What Gets Instrumented?

LLM Providers (20+)

OpenAI, OpenRouter, CometAPI, Anthropic, Google AI, Google GenAI, AWS Bedrock, Azure OpenAI, Cohere, Mistral AI, Together AI, Groq, Ollama, Vertex AI, Replicate, HuggingFace, SambaNova, Sarvam AI, Hyperbolic, LiteLLM

See all providers with examples >>

Multi-Agent Frameworks (8)

CrewAI, LangGraph, Google ADK, AutoGen, AutoGen AgentChat, OpenAI Agents SDK, Pydantic AI, AWS Bedrock Agents

See all frameworks with examples >>

MCP Tools (20+)

Databases: PostgreSQL, MySQL, MongoDB, SQLAlchemy, TimescaleDB, OpenSearch, Elasticsearch, FalkorDB Caching: Redis | Queues: Kafka, RabbitMQ | Storage: MinIO Vector DBs: Pinecone, Weaviate, Qdrant, ChromaDB, Milvus, FAISS, LanceDB

See all MCP tools >>

Built-in Evaluation (6 Detectors)

PII Detection (GDPR/HIPAA/PCI-DSS), Toxicity Detection, Bias Detection, Prompt Injection Detection, Restricted Topics, Hallucination Detection

See all evaluation features with examples >>

Screenshots

OpenAI traces with token usage, costs, and latency

More screenshots

Ollama (Local LLM)

SmolAgents with Tool Calls

GPU Metrics

OpenSearch Dashboard

Key Features

Automatic Cost Tracking

1,050+ models across 30+ providers with per-request cost breakdown. Supports differential pricing (prompt vs completion), reasoning tokens, cache pricing, and custom model pricing.

# Cost tracking is enabled by default - just instrument and go
genai_otel.instrument()

# Or add custom pricing for proprietary models
export GENAI_CUSTOM_PRICING_JSON='{"chat":{"my-model":{"promptPrice":0.001,"completionPrice":0.002}}}'

Cost tracking guide >>

GPU Metrics (NVIDIA + AMD)

Real-time monitoring of utilization, memory, temperature, power, PCIe throughput, throttling, and ECC errors. Multi-GPU aggregate metrics included.

pip install genai-otel-instrument[gpu]      # NVIDIA
pip install genai-otel-instrument[amd-gpu]  # AMD

GPU metrics guide >>

Multi-Agent Tracing

Complete span hierarchy for agent frameworks with automatic context propagation:

Crew Execution
  +-- Agent: Senior Researcher (gpt-4)
  |     +-- Task: Research OpenTelemetry
  |           +-- openai.chat.completions (tokens: 1250, cost: $0.03)
  +-- Agent: Technical Writer (ollama:llama2)
        +-- Task: Write blog post
              +-- ollama.chat (tokens: 890, cost: $0.00)

Multimodal Observability (v1.1.0)

First-class capture of image, audio, video, and document content parts on OpenAI, Anthropic, Google Gemini, and Groq spans. Bytes are offloaded to your configured object store (MinIO / S3 / filesystem / HTTP) and referenced from spans by URI — they never appear inline in span attributes.

# Opt in (default is off — text-only behaviour is byte-identical to 1.0.x)
export GENAI_OTEL_MEDIA_CAPTURE_MODE=full
export GENAI_OTEL_MEDIA_STORE=minio
export GENAI_OTEL_MEDIA_STORE_ENDPOINT=http://localhost:9000
export GENAI_OTEL_MEDIA_STORE_ACCESS_KEY=...
export GENAI_OTEL_MEDIA_STORE_SECRET_KEY=...
# Optional: plug in a redactor before upload
export GENAI_OTEL_MEDIA_REDACTOR=genai_otel.media.redactors.face_blur

Spans get two co-emitted representations of the same multimodal content:

• A flat, queryable attribute namespace — gen_ai.prompt.{n}.content.{m}.{type, media_uri, media_mime_type, media_byte_size, media_source} plus a gen_ai.completion.* mirror — for backends that index on flat attributes.
• The upstream-canonical gen_ai.input.messages / gen_ai.output.messages JSON conforming to the gen-ai message schemas in the dedicated semantic-conventions-genai repo, including the document modality, optional byte_size, and stripped_reason shape standardised by our upstream PRs #142 / #143 / #144 (see Standards Contributions below).

Multimodal guide >>

Safety & Evaluation

genai_otel.instrument(
    enable_pii_detection=True,       # GDPR/HIPAA/PCI-DSS compliance
    enable_toxicity_detection=True,  # Perspective API + Detoxify
    enable_bias_detection=True,      # 8 bias categories
    enable_prompt_injection_detection=True,
    enable_hallucination_detection=True,
    enable_restricted_topics=True,
)

Evaluation guide with 50+ examples >>

Configuration

# Required
export OTEL_SERVICE_NAME=my-llm-app
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Optional
export GENAI_ENABLE_GPU_METRICS=true
export GENAI_ENABLE_COST_TRACKING=true
export GENAI_SAMPLING_RATE=0.5                    # Reduce volume in production
export GENAI_ENABLED_INSTRUMENTORS=openai,crewai  # Select specific instrumentors

Full configuration reference >>

Backend Integration

Works with any OpenTelemetry-compatible backend:

Jaeger, Zipkin, Prometheus, Grafana, Datadog, New Relic, Honeycomb, AWS X-Ray, Google Cloud Trace, Elastic APM, Splunk, SigNoz, self-hosted OTel Collector

Pre-built Grafana dashboard templates included.

Examples

90+ ready-to-run examples covering every provider, framework, and evaluation feature:

examples/
+-- openai/              # OpenAI chat, embeddings
+-- anthropic/           # Anthropic + PII/toxicity detection
+-- ollama/              # Local models + all evaluation features
+-- crewai_example.py    # Multi-agent crew orchestration
+-- langgraph_example.py # Stateful graph workflows
+-- google_adk_example.py # Google Agent Development Kit
+-- autogen_example.py   # Microsoft AutoGen agents
+-- pii_detection/       # 10 PII examples (GDPR, HIPAA, PCI-DSS)
+-- toxicity_detection/  # 8 toxicity examples
+-- bias_detection/      # 8 bias examples (hiring compliance, etc.)
+-- prompt_injection/    # 6 injection defense examples
+-- hallucination/       # 4 hallucination detection examples
+-- ...                  # And many more

Browse all examples >>

OpenTelemetry Standards Contributions

TraceVerde isn't only a consumer of OpenTelemetry GenAI semantic conventions — production gaps surfaced by the library are being upstreamed back into the spec. Active proposals on open-telemetry/semantic-conventions-genai:

PR	Proposal	Status
#142	Add document to the Modality enum on BlobPart / FilePart / UriPart — PDFs, DOCX, and other non-image/video/audio payloads currently fall through to the free-form string branch. BFSI KYC extraction is a high-volume real example.	Approved (@MikeGoldsmith)
#143	Add optional byte_size on the three media-part types so consumers get a uniform handle on payload size whether the content was carried inline, by URI, or by provider file id — useful for cost-of-capture telemetry and storage planning. Pydantic ge=0 → JSON schema "minimum": 0.	Under review
#144	Make content / file_id / uri optional and add a free-form stripped_reason (size_exceeded, modality_not_allowed, redactor_error, upload_error, no_store_configured) so an instrumentation can fail-closed — record that it observed a media part but intentionally did not capture its bytes — while preserving the original part type and modality. Enforced via a top-level anyOf so structurally-empty parts cannot validate.	Under review

All three were migrated from the closed open-telemetry/semantic-conventions#3673 after the GenAI conventions split into the dedicated repo on 2026-05-05. Each PR ships under the new repo's V2 Weaver schema with the corresponding models.ipynb updates and make check-policies / make generate-all validation.

TraceVerde v1.1.1 already emits the proposed shape on the wire via dual-emission (OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai), providing the reference implementation for these conventions.

Ecosystem & Framework Contributions

Beyond the spec, genai_otel is shipping inside agent frameworks as their OpenTelemetry observability layer — in-tree where the project accepts it, or as a standalone plugin where the project's policy keeps integrations out of the core tree — demonstrating the library powering real third-party agents, not just first-party services:

Integration	Framework	Contribution	Status
hermes-otel-plugin	NousResearch Hermes	A standalone otel plugin (hermes plugins install Mandark-droid/hermes-otel-plugin) exporting Hermes turns / LLM calls / tool calls as OTel GenAI spans and dashboard log records, with no changes to Hermes core. When genai-otel-instrument is installed it additionally unlocks on-prem GPU / energy / CO2 metrics, local-model cost (parameter-size pricing for Ollama / HF / vLLM), and an inline eval / guardrail suite (PII, toxicity, bias, prompt-injection, restricted-topics, hallucination) scored on prompt and response — signals no vanilla OTel SDK or other GenAI instrumentor emits. Originally upstream PR #48184 — approved on code review, then republished standalone per Hermes's policy that observability backends ship as standalone plugin repos.	Shipped (standalone plugin, MIT)

Who Uses TraceVerde?

TraceVerde is used by developers and teams building production GenAI applications. If you're using TraceVerde, we'd love to hear from you!

Add your company | Join Discord

Community

• Documentation - Full guides, API reference, and tutorials
• Discord - Chat with the community
• GitHub Issues - Bug reports and feature requests
• Contributing - How to contribute

License

Copyright 2025 Kshitij Thakkar. Licensed under the Apache License 2.0.