[FEA]: Add Langsmith and Langfuse telemetry exporters #233
Description
Is this a new feature, an improvement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
Medium
Please provide a clear description of problem this feature solves
I want to add langsmith and langfuse telemetry exporters to AIQ Toolkit to make it easier to configure these popular LLM observability tools. There are some difference in how langfuse and langsmith are set up:
- project: langfuse seems to use credentials that are scoped to a project. langsmith requires project to be set in headers
- credentials: langfuse use a secret key and public key, and these need to be base64 encoded (
base64(public_key:secret_key)), langsmith uses an API key - endpoint: langsmith accepts otel traces on
/otel/v1/tracesand langfuse accpets otel traces on/api/public/otel/v1/traces
It would be easier to use langsmith and langfuse as registered telemetry exporters rather than using the otelcollector. The otelcollector would be possible, but it would require setting OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS instead of passing in values from the config_file.
Describe your ideal solution
Include the following telemetry exporters in src/aiq/observability/register.py:
class LangfuseTelemetryExporter(TelemetryExporterBaseConfig, name="langfuse"):
"""A telemetry exporter to transmit traces to externally hosted langfuse service."""
endpoint: str = Field(
description="The langfuse OTEL endpoint (/api/public/otel/v1/traces)"
)
public_key: str = Field(description="The Langfuse public key", default="")
secret_key: str = Field(description="The Langfuse secret key", default="")
@register_telemetry_exporter(config_type=LangfuseTelemetryExporter)
async def langfuse_telemetry_exporter(
config: LangfuseTelemetryExporter, builder: Builder
):
"""Create a Langfuse telemetry exporter."""
import base64
trace_exporter = telemetry_optional_import(
"opentelemetry.exporter.otlp.proto.http.trace_exporter"
)
secret_key = config.secret_key or os.environ.get("LANGFUSE_SECRET_KEY")
public_key = config.public_key or os.environ.get("LANGFUSE_PUBLIC_KEY")
if not secret_key and public_key:
raise ValueError("secret and public keys are required for langfuse")
credentials = f"{public_key}:{secret_key}".encode("utf-8")
auth_header = base64.b64encode(credentials).decode("utf-8")
headers = {"Authorization": f"Basic {auth_header}"}
yield trace_exporter.OTLPSpanExporter(endpoint=config.endpoint, headers=headers)
class LangsmithTelemetryExporter(TelemetryExporterBaseConfig, name="langsmith"):
"""A telemetry exporter to transmit traces to externally hosted langsmith service."""
endpoint: str = Field(
description="The langfuse OTEL endpoint",
default="https://api.smith.langchain.com/otel/v1/traces",
)
api_key: str = Field(description="The Langsmith API key", default="")
project: str = Field(description="The project name to group the telemetry traces.")
@register_telemetry_exporter(config_type=LangsmithTelemetryExporter)
async def langsmith_telemetry_exporter(
config: LangsmithTelemetryExporter, builder: Builder
):
"""Create a Langsmith telemetry exporter."""
trace_exporter = telemetry_optional_import(
"opentelemetry.exporter.otlp.proto.http.trace_exporter"
)
api_key = config.api_key or os.environ.get("LANGSMITH_API_KEY")
if not api_key:
raise ValueError("API key is required for langsmith")
headers = {"x-api-key": api_key, "LANGSMITH_PROJECT": config.project}
yield trace_exporter.OTLPSpanExporter(endpoint=config.endpoint, headers=headers)Include two additional examples in examples/simple_calculator/configs demonstrating these tracing tools:
For langsmith:
general:
use_uvloop: true
telemetry:
tracing:
langsmith:
_type: langsmith
project: default
For langfuse:
general:
use_uvloop: true
telemetry:
tracing:
langfuse:
_type: langfuse
endpoint: http://localhost:3000/api/public/otel/v1/traces
Additional context
I have tested the above langfuse and langsmith exporters and I'm able to get traces on a locally hosted langfuse instance and on the langsmith platform hosted on smith.langchain.com:
langfuse:
langsmith
These two would be in addition to the telemetry exporters the phoenix and weave that are also shown in the example.
AIQ Toolkit Search Results
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ package ┃ version ┃ component_type ┃ component_name ┃ description ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ aiqtoolkit │ 0.1.dev121+g6542799.d20250509 │ tracing │ weave │ A telemetry exporter to transmit traces to Weights & Biases Weave using OpenTelemetry. │
│ │ │ │ │ │
│ │ │ │ │ Args: │
│ │ │ │ │ _type (str): The type of the object. │
│ │ │ │ │ project (str): The W&B project name. │
│ │ │ │ │ entity (typing.Optional[str]): The W&B username or team name. Defaults to None. │
├────────────┼───────────────────────────────┼────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ aiqtoolkit │ 0.1.dev121+g6542799.d20250509 │ tracing │ phoenix │ A telemetry exporter to transmit traces to externally hosted phoenix service. │
│ │ │ │ │ │
│ │ │ │ │ Args: │
│ │ │ │ │ _type (str): The type of the object. │
│ │ │ │ │ endpoint (str): The phoenix endpoint to export telemetry traces. │
│ │ │ │ │ project (str): The project name to group the telemetry traces. │
├────────────┼───────────────────────────────┼────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ aiqtoolkit │ 0.1.dev121+g6542799.d20250509 │ tracing │ langfuse │ A telemetry exporter to transmit traces to externally hosted langfuse service. │
│ │ │ │ │ │
│ │ │ │ │ Args: │
│ │ │ │ │ _type (str): The type of the object. │
│ │ │ │ │ endpoint (str): The langfuse OTEL endpoint (/api/public/otel/v1/traces). │
│ │ │ │ │ public_key (str): The Langfuse public key. Defaults to "". │
│ │ │ │ │ secret_key (str): The Langfuse secret key. Defaults to "". │
├────────────┼───────────────────────────────┼────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ aiqtoolkit │ 0.1.dev121+g6542799.d20250509 │ tracing │ langsmith │ A telemetry exporter to transmit traces to externally hosted langsmith service. │
│ │ │ │ │ │
│ │ │ │ │ Args: │
│ │ │ │ │ _type (str): The type of the object. │
│ │ │ │ │ endpoint (str): The langfuse OTEL endpoint. Defaults to "https://api.smith.langchain.com/otel/v1/traces". │
│ │ │ │ │ api_key (str): The Langsmith API key. Defaults to "". │
│ │ │ │ │ project (str): The project name to group the telemetry traces. │
├────────────┼───────────────────────────────┼────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ aiqtoolkit │ 0.1.dev121+g6542799.d20250509 │ tracing │ otelcollector │ A telemetry exporter to transmit traces to externally hosted otel collector service. │
│ │ │ │ │ │
│ │ │ │ │ Args: │
│ │ │ │ │ _type (str): The type of the object. │
│ │ │ │ │ endpoint (str): The otel endpoint to export telemetry traces. │
│ │ │ │ │ project (str): The project name to group the telemetry traces. │
└────────────┴───────────────────────────────┴────────────────┴────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Code of Conduct
- I agree to follow this project's Code of Conduct
- I have searched the open feature requests and have found no duplicates for this feature request