Skip to content

feat(llmobs): report experiment status and error to backend#16713

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 4 commits into
mainfrom
fouad.wahabi/llmobs-experiment-status
Mar 6, 2026
Merged

feat(llmobs): report experiment status and error to backend#16713
gh-worker-dd-mergequeue-cf854d[bot] merged 4 commits into
mainfrom
fouad.wahabi/llmobs-experiment-status

Conversation

@FouadWahabi

@FouadWahabi FouadWahabi commented Mar 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Add experiment_update() to LLMObsExperimentsClient that PATCHes experiment status/error via the backend API.

Update Experiment.run() to report status transitions:

  • "running" when execution starts
  • "completed" on success
  • "interrupted" on KeyboardInterrupt
  • "failed" with error message on other exceptions

Testing

Risks

Additional Notes

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Mar 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/llmobs/_experiment.py                                           @DataDog/ml-observability
ddtrace/llmobs/_writer.py                                               @DataDog/ml-observability
releasenotes/notes/llmobs-experiment-status-reporting-16a969ad004ea43c.yaml  @DataDog/apm-python

@FouadWahabi
FouadWahabi force-pushed the fouad.wahabi/llmobs-experiment-status branch 3 times, most recently from 12a98d1 to f7a7857 Compare March 4, 2026 10:54
@FouadWahabi FouadWahabi mentioned this pull request Mar 4, 2026
Closed
2 tasks
@FouadWahabi
FouadWahabi force-pushed the fouad.wahabi/llmobs-experiment-status branch 3 times, most recently from 30618d4 to 3b45312 Compare March 4, 2026 12:36
@FouadWahabi
FouadWahabi marked this pull request as ready for review March 4, 2026 13:00
@FouadWahabi
FouadWahabi requested review from a team as code owners March 4, 2026 13:00
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Mar 4, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 609a6a3074

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread ddtrace/llmobs/_experiment.py Outdated
cdfox
cdfox reviewed Mar 4, 2026
View reviewed changes
Comment thread releasenotes/notes/llmobs-experiment-status-reporting-16a969ad004ea43c.yaml
cdfox
cdfox reviewed Mar 4, 2026
View reviewed changes
Comment thread ddtrace/llmobs/_experiment.py
gsvigruha
gsvigruha approved these changes Mar 4, 2026
View reviewed changes

@gsvigruha gsvigruha left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending Chris's two comments

Comment thread ddtrace/llmobs/_experiment.py
Comment thread releasenotes/notes/llmobs-experiment-status-reporting-16a969ad004ea43c.yaml
emmettbutler
emmettbutler approved these changes Mar 4, 2026
View reviewed changes

@emmettbutler emmettbutler left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved the release note

Comment thread releasenotes/notes/llmobs-experiment-status-reporting-16a969ad004ea43c.yaml Outdated
@FouadWahabi
FouadWahabi force-pushed the fouad.wahabi/llmobs-experiment-status branch from 041a9b3 to b931e44 Compare March 5, 2026 11:12
@FouadWahabi
feat(llmobs): report experiment status to backend
3c1e66f
@FouadWahabi
feat(llmobs): include error summary when reporting failed experiment …
8d2a943 
…status

When an experiment completes with errors (raise_errors=False), the status
update now includes a descriptive error message (e.g. "3/10 tasks failed,
2 evaluator error(s)") so the backend can display why the experiment failed.
@FouadWahabi
FouadWahabi force-pushed the fouad.wahabi/llmobs-experiment-status branch 2 times, most recently from 9b23d66 to af4bdea Compare March 5, 2026 14:09
@FouadWahabi
FouadWahabi force-pushed the fouad.wahabi/llmobs-experiment-status branch from af4bdea to 8060c90 Compare March 5, 2026 14:14
@FouadWahabi

FouadWahabi commented Mar 6, 2026

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Mar 6, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-03-06 16:05:18 UTC ℹ️ Start processing command /merge

2026-03-06 16:05:22 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 3h (p90).

2026-03-06 16:59:40 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit db32c90 into main Mar 6, 2026
425 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the fouad.wahabi/llmobs-experiment-status branch March 6, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cdfox cdfox cdfox left review comments
@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@emmettbutler emmettbutler emmettbutler approved these changes
@gsvigruha gsvigruha gsvigruha approved these changes
@Kyle-Verhoog Kyle-Verhoog Awaiting requested review from Kyle-Verhoog Kyle-Verhoog is a code owner automatically assigned from DataDog/apm-python
@taegyunkim taegyunkim Awaiting requested review from taegyunkim taegyunkim is a code owner automatically assigned from DataDog/apm-python

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@FouadWahabi @emmettbutler @cdfox @gsvigruha