This product is not supported for your selected
Datadog site. (
).
このページは日本語には対応しておりません。随時翻訳に取り組んでいます。
翻訳に関してご質問やご意見ございましたら、
お気軽にご連絡ください。
Overview
Agent Observability offers several ways to support evaluations. They can be configured by navigating to AI Observability > Evaluations.
Custom LLM-as-a-judge evaluations
Custom LLM-as-a-judge evaluations allow you to define your own evaluation logic using natural language prompts. You can create custom evaluations to assess subjective or objective criteria (like tone, helpfulness, or factuality) and run them at scale across your traces and spans.
Managed evaluations
Datadog builds and supports managed evaluations to support common use cases. You can enable and configure them within the Agent Observability application.
Submit end-user feedback
End-user feedback lets you submit thumbs-up or thumbs-down ratings, accepted changes, free-text comments, and other user or agent feedback to Datadog. Feedback can be connected to spans, traces, sessions, or customer-defined entities with a feedback join key.
Submit external evaluations
You can also submit external evaluations using Datadog’s API. Use this approach when you have your own evaluation system but want to centralize evaluation results within Datadog.
Building custom evaluators
For developers building custom evaluators, see the Evaluation Developer Guide.
Evaluation integrations
Datadog also supports integrations with some 3rd party evaluation frameworks, such as NeMo.
Annotation Queues
Annotation Queues provide a structured workflow for systematic human review of LLM traces.
Sensitive Data Scanner integration
In addition to evaluating the input and output of LLM requests, agents, workflows, or the application, Agent Observability integrates with Sensitive Data Scanner, which helps prevent data leakage by identifying and redacting any sensitive information. For a list of the out-of-the-box rules included with Sensitive Data Scanner, see Library Rules.
Security
Get real-time security guardrails for your AI apps and agents
AI Guard helps secure your AI apps and agents in real time against prompt injection, jailbreaking, tool misuse, and sensitive data exfiltration attacks. Try it today!
Permissions
Agent Observability Write permissions are necessary to configure evaluations.
Retrieving spans
Agent Observability offers an Export API that you can use to retrieve spans for running external evaluations. This helps circumvent the need to keep track of evaluation-relevant data at execution time.
Further Reading
