I was wondering maybe even before the formats/semantics across instrumentation libraries become mature, you can probably still able to build some OTel collectors to integrate with langfuse, as the formats/semantics across instrumentation libraries did not impact much for the implementation

I am unsure about this one as

• we will not be able to support all Langfuse features while they are not yet covered by the conventions
• depending on the speed of the development of the conventions, once we add an initial otel collector (with basic features), there will probably be great interest to support vendor-specific implementation aspects while the standards still develop. This leads to potential maintenance overhead and breaking backwards compatibility once conventions are in place for everyone who relied on the 3rd party vendor instrumentation.

Yes, but the OTEL community is trying to unify this via the LLM working group, there is already a initial draft spec for LLM semantic convention at https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai

I'll try to join the working group to keep in the loop with these changes, very interested in this. I am especially excited about this proposal: open-telemetry/community#2326

Fair, we might ship something like this as a quick first POC at one point to get feedback and collaborate with everyone, labeling it as beta. However, getting to GA quickly was something we focused on with all of our other integrations as it was valued by many teams using Langfuse. In this case, we need to make clear that things can still break while the conventions change.

btw, awesome to see you being interested to contribute to the new otel instrumentation libraries

Fair, we might ship something like this as a quick first POC at one point to get feedback and collaborate with everyone

I'm excited for this, look forward to see this :)

thanks!

logfire is superb, would be awesome to get the integration (also given that pydantic is ubiquitous).

Maybe merge this issue too?
#199

makes sense, let's merge all threads on this here. thank you!

I'm a member of the community, but not yet a spec approver or lead. Here's my 2p for what it's worth:

• for observing basic chat completions and embeddings traces, metrics and logs, there's some value from system POV (think dashboards that count tokens).
• there's a cavaet in logs as many eval products do not accept otel's log format, yet.
• specs have focused so far on LLM access, while agentic use or evaluations are a work in progress
• things could be driven more closely to end requirements with more participation not just in issues, but also in zoom meetings, so please help if you can.
• the spec will call itself experimental for the foreseeable future, but that doesn't mean people won't use it.

I think LLMOps is tricky as many people think of observability just like distributed systems with certain types of metrics etc, and think the problem is solved, while folks on the MLOps side may see lack of quality metrics, or specs around training pipelines, etc and feel it isn't.

Hope I was fair and balanced in above. In summary, folks working on the spec anchor what ends up there. With more folks saying "what is practical", it is possible to have specs achieve that.

thanks for sharing!

thanks for sharing!

Does this change the outlook for Langfuse wrt OTel? ;-)

Not really, this is something that is very actively discussed already and this post adds another upvote that it might be the right time to finally add an initial Otel collector to Langfuse.

fyi, @mstrathman and @nguyenth-sieuviet as you were interested in Pydantic AI support, this will be available soon

@mjrusso Yes that should work. You can probably create a custom "context" and use that to create a new span with the correct traceId.

Here's what I have so far:

with logfire.span("Root operation") as root_span:
    parent_trace_id = root_span.get_span_context().trace_id

    # Open Telemetry reports the trace ID as a 32-byte hexadecimal string!
    parent_trace_id_hex = format(parent_trace_id, "032x")

    compute_response(
        foo,
        bar,
        langfuse_parent_trace_id=parent_trace_id_hex,
    )

Where compute_response is defined as:

    @observe  # Trace function execution with Langfuse.
    def compute_response(
        foo: Foo,
        bar: Bar,
        *,
        **kwargs,
    ):
        langfuse_context.update_current_trace(
            name="Graph Invocation",
            session_id=str(Foo.id),
        )

        ...snip...

(Burnt too much time on Open Telemetry reporting the ID in hex 🤦)

On the Langfuse end, all spans show up together in the same trace. They are nested in a different parent span (vs. the Langfuse-native spans), so the overall trace isn't really "in order".

Ah, let me try setting langfuse_parent_observation_id, too.

Ah, let me try setting langfuse_parent_observation_id, too.

The OTel span doesn't know about the span created by Langfuse (via @observe, in this case), so the Langfuse spans and OTel spans won't share a parent, and thus won't appear in their natural order. There might be a way to give Logfire the context it needs but I'm not 100% sure; the API is confusing. (And not the end of the world; at least everything is grouped together now in the same trace! That's a huge improvement. Thanks @Steffen911 for your help!)

@mjrusso Thank you for the feedback and the investigation. We'll take a look into the Langfuse SDK - OTel connection as one of the next projects and will keep that feedback in mind. We'll keep you posted here if we make improvements in that direction.

Glad to hear that you found a setup that's already good for you 🙌

for the self-hosted version, what's the OTLP endpoint? http://localhost:3000/api/public/oteldoesn't seem to work

@iamhatesz Which version of the self-hosted version are you running? OTel should work starting from v3.24.0.

@iamhatesz When using logfire, configure like below

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:3000/api/public/otel/v1/traces
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic {AUTH_HEADER}"

The AUTH_HEADER can be generated as below:

echo -n "pk-lf-1234:sk-lf-7890" | base64

Thanks. I wanted to integrate smolagents with Langfuse, but it didn't work. However, I think there's something wrong on their side.

Thanks. I wanted to integrate smolagents with Langfuse, but it didn't work. However, I think there's something wrong on their side.

Hi @iamhatesz, happy to help, could you you open a new GitHub post for this?

Yes, we plan to extend the coverage for vendor-specific semantics over time. Please create a github issue with an example (code and link to public trace) which helps to improve the parsing of the different semantics.

@marcklingen Thanks.

In pydantic 0.0.23(spec PR pydantic/pydantic-ai@35a097d), we can easily the Model for observation. I created a simple wrapper for this: https://github.com/Wh1isper/langfuse-pydantic-ai, which solves my needs nicely.

Maybe we should add this usage to the documentation, also continue to discuss OTEL Trace with logfire support.

Hello guys, after some investigation I found out that pydantic-ai uses a customized set of trace specs. And they are working on switching to OTEL specs: pydantic/pydantic-ai#875 (comment)

I think users can use wrapper for now and switch to OTEL after that.

Great to hear this, thanks for sharing!

The OpenTelemetry Collector version is 0.118.0. I'm sharing the config setup.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
exporters:
  debug:
    verbosity: detailed
  otlphttp/langfuse:
    endpoint: "http://localhost:3000/api/public/otel"
    headers:
      Authorization: "Basic <API-KEY>"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlphttp/langfuse]
  telemetry:
    logs:
      level: "debug"

Hey @Su-hub9 ,
Thank you for providing the configuration. I was able to reproduce it using your settings. It turns out that the otel collector uses gzip compression by default and our endpoint did not handle that correctly. This will be fixed with #5710 which should go live on our cloud version later today.

In the meantime, you can work around this by setting compression: none as documented in https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter.

Hi,

I have been reading your structured output information and watched the recent tool calling and structured outputs in the Langfuse playground YouTube video, and I like the feature but cannot find a way to submit the response_format structured outputs using OpenTelemetry on the OpenTelemetry property mapping page.

Does anyone know whether this is currently working?

If so, a code snippet showing the OpenTelemetry attribute keys would be much appreciated.

They are included within the input/output in langfuse tracing.

Hi @marcklingen, thanks for your reply. It’s working now after setting the attribute on a span to the following key:

langfuse.metadata.response_format

NOTE: Note, I am using the Java OpenTelemetry SDK.

.setAttribute("langfuse.metadata.response_format", jsonString)

@xiaoping378 As of now, we recommend to still use span attributes instead of events as this makes the correlation to other properties significantly easier.
Could you share more details about your instrumentation and how your spans are generated?

used zero-code instrumentation , now i need choose a sdk instrumentation

any update on this? i'm sending gen_ai.output.messages and not seeing it be respected. How do i send you the response with otel?

@krrishdholakia Could you share the events you're sending or a small reproduction? Currently standalone events are not supported, but if you send them as Span events they should be included.