Overview

Experiment metrics define the outcomes you use to evaluate an experiment, such as revenue, conversion rate, or pages viewed. You can create these metrics manually in Datadog, or you can define warehouse-native metrics in YAML and sync them to Datadog with the Datadog Experiments Metric Sync CLI.

Use Metric Sync when you want to:

• Review metric changes in pull requests.
• Keep metric definitions in version control.
• Apply metric changes from CI after they are approved.
• Mark synced metrics as certified by default for experiment decision-making.

Metric Sync supports a plan-and-apply workflow. The plan command validates your YAML and previews the changes Datadog would make. The execute command validates the same YAML, submits the write operation, waits for completion, and prints a summary of created, updated, deleted, upgraded, blocked, and failed changes.

Prerequisites

Before using Metric Sync, you need:

• A Datadog account with Product Analytics and Experiments enabled.
• A connected warehouse. See Connect a data warehouse.
• The Product Analytics permissions required to create and update experiment metrics. Synced metrics are certified by default, so custom roles also need the Product Analytics Certified Metrics Write permission. See Product Analytics permissions.
• A Datadog API key and application key for CI authentication. See API and application keys.

Install the CLI

Install the latest Metric Sync CLI release:

curl -fsSL https://raw.githubusercontent.com/DataDog/experiments-metric-sync-cli/main/install.sh | sh

The install script detects macOS or Linux, downloads the matching GitHub release archive, verifies the SHA-256 checksum, and installs the metric-sync binary to /usr/local/bin. Depending on your system permissions, installing to /usr/local/bin may require sudo. To install without elevated permissions, set INSTALL_DIR to a directory you own.

To install a specific version or install to a different directory:

curl -fsSL https://raw.githubusercontent.com/DataDog/experiments-metric-sync-cli/main/install.sh | env VERSION=<VERSION> INSTALL_DIR="$HOME/.local/bin" sh

Set Datadog credentials with environment variables:

export DD_API_KEY=<DATADOG_API_KEY>
export DD_APP_KEY=<DATADOG_APPLICATION_KEY>

By default, the CLI sends requests to datadoghq.com. Set DD_SITE if your organization uses a different Datadog site:

export DD_SITE=

Create a metric definition

Create one or more YAML files in your repository. The CLI accepts a file, multiple files, a directory, or multiple directories. When you pass a directory, the CLI recursively discovers YAML files.

The following example defines one warehouse metric source and one simple metric. By default, the CLI uses the warehouse connection configured in Datadog under Settings > Experiments > Warehouse Connections. Datadog supports one warehouse connection per organization. If your organization has more than one warehouse connection, set warehouse_connection_id in the YAML to choose one explicitly.

schema_version: 1
sync_tag: example-checkout
reference_url: https://github.com/example-org/example-repo

warehouse_metric_sources:
  - sync_id: checkout_events
    name: Checkout events
    sql: |
      SELECT user_id, event_timestamp, revenue
      FROM analytics.checkout_events
    timestamp_column: event_timestamp
    subject_types:
      - name: User
        column_name: user_id
    measures:
      - sync_id: revenue
        name: Revenue
        column_name: revenue
        column_type: FLOAT

metrics:
  - sync_id: checkout_revenue
    name: Checkout revenue
    metric_type: simple
    desired_change: METRIC_INCREASES
    simple_metric_aggregation:
      operation: sum
      measure:
        warehouse_metric_source_sync_id: checkout_events
        measure_sync_id: revenue
      timeframe_start_value: 0

Preview and sync changes

To preview the changes before applying them, run plan:

metric-sync plan ./metrics

The plan output shows the operation ID, sync tag, status, and a summary of the changes Datadog would make.

After reviewing the plan, run execute to apply the changes:

metric-sync execute ./metrics

execute performs local validation, submits a Metric Sync write operation to Datadog, polls the operation until it reaches a terminal state, and prints the result summary.

You can also run:

metric-sync validate ./metrics
metric-sync status <metric_sync_id>
metric-sync result <metric_sync_id>
metric-sync version

The commands do the following:

• validate: Checks only the local YAML and does not call Datadog.
• status: Checks an operation by ID.
• result: Fetches the terminal plan or execute result.
• version: Prints build metadata.

GitHub Actions example

The following workflow runs plan on pull requests and execute after changes merge to main:

name: Sync Datadog experiment metrics

on:
  pull_request:
    paths:
      - "metrics/**/*.yaml"
      - "metrics/**/*.yml"
      - ".github/workflows/datadog-metric-sync.yml"
  push:
    branches:
      - main
    paths:
      - "metrics/**/*.yaml"
      - "metrics/**/*.yml"

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Metric Sync CLI
        run: curl -fsSL https://raw.githubusercontent.com/DataDog/experiments-metric-sync-cli/main/install.sh | sh
      - name: Plan metric changes
        env:
          DD_API_KEY: ${{ secrets.DD_API_KEY }}
          DD_APP_KEY: ${{ secrets.DD_APP_KEY }}
        run: metric-sync plan ./metrics

  execute:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install Metric Sync CLI
        run: curl -fsSL https://raw.githubusercontent.com/DataDog/experiments-metric-sync-cli/main/install.sh | sh
      - name: Apply metric changes
        env:
          DD_API_KEY: ${{ secrets.DD_API_KEY }}
          DD_APP_KEY: ${{ secrets.DD_APP_KEY }}
        run: metric-sync execute ./metrics

Store your Datadog API and application keys as GitHub Actions secrets. If your organization uses a Datadog site other than datadoghq.com, add DD_SITE to the workflow environment.

YAML reference

Sync tags and IDs

A sync_tag identifies the set of metric definitions that are managed together. Use a stable sync_tag for each repository, team, or metric domain that you want to manage independently. For example, a growth team and a checkout team can sync metrics from different repositories by using different sync_tag values, such as growth-metrics and checkout-metrics.

Within a sync_tag, each warehouse metric source and metric has a stable sync_id. Metric aggregations use warehouse_metric_source_sync_id to reference warehouse metric sources and measure_sync_id to reference measures. The source ID is required because measure IDs are scoped to their warehouse metric source.

Top-level fields

Options

Warehouse metric sources

Subject types

Measures and properties

Measures and properties share the same field shape:

Metrics

Add the aggregation field that matches the metric type:

• For metric_type: simple, add simple_metric_aggregation.
• For metric_type: ratio, add ratio_metric_aggregation.
• For metric_type: percentile, add percentile_metric_aggregation.

Simple metric aggregation

Ratio metric aggregation

Use ratio_metric_aggregation to define a metric as a numerator divided by a denominator. Both numerator and denominator use the same fields as simple_metric_aggregation.

Example ratio metric aggregation:

ratio_metric_aggregation:
  numerator:
    operation: sum
    measure:
      warehouse_metric_source_sync_id: checkout_events
      measure_sync_id: revenue
  denominator:
    operation: count
    measure:
      warehouse_metric_source_sync_id: checkout_events
      kind: each_record

Percentile metric aggregation

Example percentile metric aggregation:

percentile_metric_aggregation:
  operation: percentile
  percentile: 95
  measure:
    warehouse_metric_source_sync_id: checkout_events
    measure_sync_id: page_load_time

Property filters

Use property_filters to restrict a simple aggregation to rows with matching property values:

property_filters:
  - property:
      warehouse_metric_source_sync_id: checkout_events
      property_sync_id: country
    operator: IS
    values:
      - US

Supported operators are IS and IS_NOT. When referencing a property:

• Include warehouse_metric_source_sync_tag to reference a property from another sync tag.
• Use warehouse_metric_property_id to reference an existing Datadog property directly.

Measure references

To reference a measure defined in the same sync payload, use warehouse_metric_source_sync_id and measure_sync_id:

measure:
  warehouse_metric_source_sync_id: checkout_events
  measure_sync_id: revenue

To reference a measure from another sync tag, include warehouse_metric_source_sync_tag:

measure:
  warehouse_metric_source_sync_tag: shared-sources
  warehouse_metric_source_sync_id: checkout_events
  measure_sync_id: revenue

To reference an existing Datadog measure directly, use warehouse_metric_measure_id.

Troubleshooting