Overview

Datadog Experiments is a composable platform for end-to-end experimentation. An experiment in Datadog consists of two components:

1. A randomized assignment of subjects (typically users) to two or more variations—either from a Datadog Feature Flag, or your randomization system of choice
2. A set of metrics to compare between variants—computed either within Datadog or with warehouse native analytics.

To get started, select a link from the table below. Otherwise, read on to learn more about Datadog Experiments.

Randomization

Every experiment needs a way to assign subjects to a control or treatment variant. Datadog supports two approaches.

Datadog Feature Flags

Datadog Feature Flags is the default way to randomize experiments. Create a flag, implement it with the Feature Flags SDK, and pass a stable subject identifier as the targetingKey so the same user always receives the same variant. Datadog uses deterministic hashing to keep assignments consistent across sessions and devices.

When you plan and launch an experiment, link it to a feature flag to define traffic splits, targeting rules, and rollout behavior. You can also create an experiment directly from a flag’s detail page. To randomize by a unit other than user—for example, an organization—see Subject Types.

Bring your own randomization

If you randomize subjects outside Datadog—for example, with an in-house system—use Exposure SQL Models to tell Datadog who was exposed to each experiment and when. Exposure SQL Models query exposure records from your connected warehouse and map them to Datadog fields such as subject key, timestamp, experiment ID, and variant ID.

Datadog deduplicates exposure data automatically: if a user appears in multiple variants for the same experiment, that user is excluded from the analysis. When exposures come from your warehouse instead of Feature Flags, metrics built on Datadog SDK events are not supported—you need warehouse-native metrics.

Metrics

Experiment metrics define what you measure to decide whether a change succeeded. Create at least one primary metric before launching an experiment, and add secondary metrics as guardrails for unintended effects on performance, engagement, or revenue.

Warehouse native mode

In warehouse native mode, Datadog runs experiment analysis directly in Snowflake, BigQuery, Redshift, or Databricks. After you connect your warehouse, create a Metric SQL Model that maps warehouse tables to Datadog, then define metrics from that model. Map each model to one or more subject types and specify a timestamp column so Datadog can join metric events to experiment exposures.

Warehouse mode is required when you use Exposure SQL Models for randomization. It also suits teams whose source of truth for business metrics already lives in the warehouse.

Product Analytics and RUM

For client-side experiments, build metrics from events collected by the Real User Monitoring (RUM) and Product Analytics SDKs. Define metrics from actions, views, sessions, and other event types, then choose an aggregation method such as count of events, count of unique users, or sum of a property.

This path works when randomization runs through Datadog Feature Flags and you want to measure user behavior, funnel conversion, or application performance without querying a warehouse. Product Analytics and RUM metrics are available in near real time as experiments launch.

Statistics

Datadog applies statistical analysis to compare variants and estimate lift. When you set up an experiment, choose an analysis method—sequential frequentist, fixed-sample frequentist, or Bayesian—and optionally run a sample size calculation to estimate how long the experiment needs to run. After results are in, use Global Lift to understand how a targeted experiment lift translates to impact on your company-wide metric total, and Cumulative Impact to aggregate noise-adjusted effects across many experiments on the same metric.

Further reading

お役に立つドキュメント、リンクや記事:

Measure the business impact of every product change with Datadog ExperimentsBLOG Feature FlagsDOCUMENTATION Product AnalyticsDOCUMENTATION