Skip to content

Add multi-modal messages support to AI Guard#5564

Merged
y9v merged 2 commits into
masterfrom
ai-guard-multi-modal-messages
Apr 8, 2026
Merged

Add multi-modal messages support to AI Guard#5564
y9v merged 2 commits into
masterfrom
ai-guard-multi-modal-messages

Conversation

@y9v

@y9v y9v commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?
It adds multi-modal content support to AI Guard messages. SDK users can now pass structured content parts (text and image URLs) instead of only plain strings, using a block builder.

Datadog::AIGuard.message(role: :user) do |message|
  message.text("What's in this image?")
  message.image_url("data:image/jpeg;base64,...")
end

Motivation:
We need to support multi-modal input to evaluate messages that include media.

Change log entry
Yes. AI Guard: Added multi-model messages support, enabling text and image URL content parts in evaluation SDK

Additional Notes:
APPSEC-61514

How to test the change?
CI and manual testing.

@y9v y9v self-assigned this Apr 7, 2026
@y9v
y9v requested review from a team as code owners April 7, 2026 15:41
@github-actions

github-actions Bot commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 1 steep:ignore comment, and clears 1 steep:ignore comment.

steep:ignore comments (+1-1) ❌ Introduced: 
lib/datadog/ai_guard/evaluation.rb:64
✅ Cleared: 
lib/datadog/ai_guard/evaluation.rb:60

Untyped methods

This PR introduces 1 partially typed method, and clears 1 partially typed method. It increases the percentage of typed methods from 61.72% to 61.83% (+0.11%).

Partially typed methods (+1-1) ❌ Introduced: 
sig/datadog/ai_guard/evaluation/request.rbs:32
└── def build_request_body: () -> ::Hash[::Symbol, untyped]
✅ Cleared: 
sig/datadog/ai_guard/evaluation/request.rbs:28
└── def build_request_body: () -> ::Hash[::Symbol, untyped]

If you believe a method or an attribute is rightfully untyped or partially typed, you can add # untyped:accept on the line before the definition to remove it from the stats.

@datadog-official

datadog-official Bot commented Apr 7, 2026 •
edited by datadog-prod-us1-4 Bot
Loading

Copy link
Copy Markdown

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 85.71%
• Overall Coverage: 95.22% (-0.13%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c9328dc | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@pr-commenter

pr-commenter Bot commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-04-08 13:46:07

Comparing candidate commit c9328dc in PR branch ai-guard-multi-modal-messages with baseline commit 03ddf0c in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

vpellan
vpellan approved these changes Apr 8, 2026
View reviewed changes
Comment thread lib/datadog/ai_guard/evaluation.rb Outdated
manuel-alvarez-alvarez
manuel-alvarez-alvarez approved these changes Apr 8, 2026
View reviewed changes

@manuel-alvarez-alvarez manuel-alvarez-alvarez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ergonomics look good to me, we also need to update the documentation covering the content parts (check Python)

@y9v
y9v force-pushed the ai-guard-multi-modal-messages branch from 863daa1 to c9328dc Compare April 8, 2026 13:20
@y9v y9v mentioned this pull request Apr 8, 2026
Closed
1 task
@y9v

y9v commented Apr 8, 2026

Copy link
Copy Markdown
Member Author

Documentation PR: DataDog/documentation#35837

@y9v
y9v merged commit 7d4a027 into master Apr 8, 2026
629 checks passed
@y9v
y9v deleted the ai-guard-multi-modal-messages branch April 8, 2026 13:49
@github-actions github-actions Bot added this to the 2.31.0 milestone Apr 8, 2026
@y9v y9v mentioned this pull request Apr 8, 2026
Merged
5 tasks
@y9v y9v mentioned this pull request Apr 16, 2026
Closed
@sameerank sameerank mentioned this pull request May 13, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@manuel-alvarez-alvarez manuel-alvarez-alvarez manuel-alvarez-alvarez approved these changes
@vpellan vpellan vpellan approved these changes

Assignees

@y9v y9v

Labels

None yet

Projects

None yet

Milestone

2.31.0

Development

Successfully merging this pull request may close these issues.

3 participants

@y9v @manuel-alvarez-alvarez @vpellan