Skip to content

feat(llmobs/experiment): add summary evaluators#4040

Merged
darccio merged 2 commits into
mainfrom
rarguelloF/experiment-summary-evaluators
Oct 9, 2025
Merged

feat(llmobs/experiment): add summary evaluators#4040
darccio merged 2 commits into
mainfrom
rarguelloF/experiment-summary-evaluators

Conversation

@rarguelloF

@rarguelloF rarguelloF commented Oct 8, 2025

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds WithSummaryEvaluators to the llmobs/experiment package to provide summary evaluators to the experiment.

Motivation

Feature parity with dd-trace-py

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
  • System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
  • There is a benchmark for any new code, or changes to existing code.
  • If this interacts with the agent in a new way, a system test has been added.
  • New code is free of linting errors. You can check this by running ./scripts/lint.sh locally.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.
  • Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

@rarguelloF
rarguelloF requested a review from a team as a code owner October 8, 2025 15:01
@pr-commenter

pr-commenter Bot commented Oct 8, 2025 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2025-10-09 10:20:32

Comparing candidate commit bbca096 in PR branch rarguelloF/experiment-summary-evaluators with baseline commit 6eb51be in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 metrics, 0 unstable metrics.

@datadog-official

datadog-official Bot commented Oct 8, 2025 •
edited by datadog-datadog-prod-us1 Bot
Loading

Copy link
Copy Markdown
Contributor

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bbca096 | Docs | Was this helpful? Give us feedback!

mtoffl01
mtoffl01 approved these changes Oct 8, 2025
View reviewed changes
Comment thread llmobs/experiment/option.go
// WithSummaryEvaluators sets the summary evaluators for the experiment.
// Summary evaluators run after all tasks and evaluators have completed,
// receiving all experiment results to compute aggregate metrics.
func WithSummaryEvaluators(summaryEvaluators ...SummaryEvaluator) Option {

@mtoffl01 mtoffl01 Oct 8, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be configurable via env var as well?

@rarguelloF rarguelloF Oct 9, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this could be configured as an env var, since this option is used to provided summary evaluators for the experiment, which is basically user code that gets access to the experiment results in order to compute metrics.

@rarguelloF

rarguelloF commented Oct 9, 2025

Copy link
Copy Markdown
Contributor Author

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Oct 9, 2025 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-10-09 08:23:32 UTC ℹ️ Start processing command /merge

2025-10-09 08:23:42 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2025-10-09 10:10:26 UTC ⚠️ MergeQueue: This merge request was unqueued

[email protected] unqueued this merge request

@darccio

darccio commented Oct 9, 2025

Copy link
Copy Markdown
Member

/merge -c

@darccio
darccio requested a review from a team as a code owner October 9, 2025 10:07
@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Oct 9, 2025 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-10-09 10:10:18 UTC ℹ️ Start processing command /merge -c

@darccio
darccio merged commit 01837b8 into main Oct 9, 2025
241 of 242 checks passed
@darccio
darccio deleted the rarguelloF/experiment-summary-evaluators branch October 9, 2025 10:30
hannahkm pushed a commit that referenced this pull request Oct 9, 2025
@rarguelloF @darccio @hannahkm
feat(llmobs/experiment): add summary evaluators (#4040)
338ef7a 
Co-authored-by: Dario Castañé <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@darccio darccio darccio approved these changes
@mtoffl01 mtoffl01 mtoffl01 approved these changes

Assignees

No one assigned

Labels

mergequeue-status: removed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rarguelloF @darccio @mtoffl01