feat(llmobs): prompt tracking#4608
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files
🚀 New features to boost your workflow:
|
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: f4d6d76 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
BenchmarksBenchmark execution time: 2026-04-08 17:51:36 Comparing candidate commit f4d6d76 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 217 metrics, 7 unstable metrics.
|
PROFeNoM
left a comment
There was a problem hiding this comment.
LGTM - the data model matches Python's, defaults are correct, and tests are good 👍 Left one non-blocking comment ![]()
| if a.Prompt != nil { | ||
| if s.spanKind != SpanKindLLM { | ||
| log.Warn("llmobs: input prompt can only be annotated on llm spans, ignoring") | ||
| } else { | ||
| if a.Prompt.RAGContextVariables == nil { | ||
| a.Prompt.RAGContextVariables = []string{"context"} | ||
| } | ||
| if a.Prompt.RAGQueryVariables == nil { | ||
| a.Prompt.RAGQueryVariables = []string{"question"} | ||
| } | ||
| if a.Prompt.ID == "" { | ||
| a.Prompt.ID = s.mlApp + "_unnamed-prompt" | ||
| } | ||
| a.Prompt.MLApp = s.mlApp | ||
| s.llmCtx.prompt = a.Prompt | ||
| } |
There was a problem hiding this comment.
nit: Python validates that template and chat_template are mutually exclusive. Might be worth a similar guard here since both fields are documented as alternatives, and setting both would produce a payload with both keys. Not blocking.
ddtrace/llmobs/_utils.py#L92-L93
if template and chat_template:
raise ValueError("Only one of 'template' or 'chat_template' can be provided, not both.")There was a problem hiding this comment.
thanks, that makes sense! In the Go SDK we are avoiding returning errors / panics, so we would prefer not to raise an exception like in Python. But we could automatically drop one of the 2 (keep chat_template maybe which looks more specific?) and log a warning. WDYT?
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
Tests failed on this commit b52e94a:
What to do next?
|
8eba8fb
into
main
… merge After merging main (which added the prompt-tracking PR #4608), test cases that used the old testtracer.TestTracer API (Stop/WaitForLLMObsSpans) and mismatched testTracer return-value counts needed updating for the new llmobstest.Collector API (RequireSpan + tracer.Flush). Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
What does this PR do?
Adds prompt tracking support for LLM spans in LLMObs via a new
WithAnnotatedPromptannotation option.Example usage:
Additional changes:
WithAnnotatedToolDefinitions(toolDefinitions []ToolDefinition)annotation option for LLM spans.WithIntent→WithAnnotatedIntent(old function deprecated, kept as alias)Motivation
Prompt tracking LLMObs feature
Reviewer's Checklist
make lintlocally.make testlocally.make generatelocally.make fix-moduleslocally.Unsure? Have a question? Request a review!