Export Pipelines

Docs > Product Analytics > Export Pipelines

Cette page n'est pas encore disponible en français, sa traduction est en cours.
Si vous avez des questions ou des retours sur notre projet de traduction actuel, n'hésitez pas à nous contacter.

Overview

Export Pipelines stream your ingested Real User Monitoring (RUM) and Product Analytics events to a cloud storage bucket you own — Amazon S3, Azure Blob Storage, or Google Cloud Storage — in JSON or Parquet format.

Export Pipelines list on the RUM application settings page

Use Export Pipelines to:

Load your event data into your own data warehouse (such as Snowflake, BigQuery, or Databricks) for custom analytics and reporting.
Meet compliance requirements or retain raw events for long-term archival and ad-hoc investigations.

Datadog only manages the export from your Datadog account to your cloud storage system.

How it works

Export Pipelines is a shared feature between Real User Monitoring and Product Analytics. Pipelines are configured at two different levels:

Scope	Source	Max pipelines	Available presets
Per application	RUM	1	All RUM event types, or Sessions, Views & Actions only
Per application	Product Analytics	1	All Product Analytics Events (sessions, views, actions & server-side events)
Per organization	Product Analytics	1	User and Account Profiles

The User and Account Profiles preset is limited to one pipeline per organization. Because user and profile data is shared across all your applications, creating one pipeline per application would result in duplicate records in your storage.

Each pipeline exports continuously and independently of the others.

Prerequisites

Product Analytics is enabled on the application (or RUM, or both).
The Datadog integration for your cloud provider is set up: Amazon Web Services, Azure, or Google Cloud.
Your Datadog user has the rum_write_archives permission. See Role Based Access Control.

Set up an export pipeline

1. Set up your cloud integration

If not already configured, set up the AWS integration for the account that holds your S3 bucket. This usually means creating a role that Datadog can assume.

For AWS China accounts, use access keys instead of role delegation.

Set up the Azure integration in the subscription that holds your storage account, if you haven’t already. This involves creating an app registration that Datadog can use.

Note: Exporting to Azure ChinaCloud and Azure GermanyCloud is not supported. Exporting to Azure GovCloud is supported in Preview — contact Datadog support to request access.

Set up the Google Cloud integration for the project that holds your GCS bucket, if you haven’t already. This involves creating a Google Cloud service account that Datadog can use.

2. Create a storage bucket

In the AWS console, create an S3 bucket for your exports.

Notes:

Do not make the bucket publicly readable.
For US1, US3, and US5 sites, see AWS pricing for inter-region data transfer fees. Consider creating the bucket in us-east-1 to minimize transfer costs.

In the Azure Portal, create a storage account. Choose Standard performance or Block blobs premium, and select either the hot or cool access tier.
Create a container inside that storage account. Note the container name — you reference it when configuring the pipeline.

Note: Do not set immutability policies. Some events occasionally need to be rewritten (typically on retry after a timeout).

In the Google Cloud console, create a GCS bucket for your exports. Under Choose how to control access to objects, select Set object-level and bucket-level permissions.

Note: Do not add a retention policy. Some events occasionally need to be rewritten (typically on retry after a timeout).

3. Grant Datadog access to the bucket

Create a policy with the following statements:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DatadogExportPipelineFiles",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": [
        "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*",
        "arn:aws:s3:::<MY_BUCKET_NAME_2_/_MY_OPTIONAL_BUCKET_PATH_2>/*"
      ]
    }
  ]
}

PutObject is required to upload export files.
GetObject is required to run Test Configuration, which writes a test file and reads it back to verify access.
The resource value must end with /* — these permissions apply to objects inside the buckets.

Edit the bucket names.
Optionally, restrict the policy to specific paths.
Attach the policy to the Datadog integration role:
- In the AWS IAM console, go to Roles and open the role used by the Datadog integration. By default it is named DatadogIntegrationRole, but the name may differ if your organization renamed it.
- Click Add permissions, then Attach policies.
- Enter the name of the policy you just created.
- Click Attach policies.

Grant the Datadog app permission to write to your storage account.
On the Storage accounts page, select your storage account, open Access Control (IAM), and choose Add → Add Role Assignment.
Assign the Storage Blob Data Contributor role to the Datadog app you created when integrating with Azure, then save.

Grant your Datadog Google Cloud service account permission to write to your bucket.
On the Google Cloud IAM Admin page, select your Datadog service account principal and click Edit principal.
Click ADD ANOTHER ROLE, select Storage Object Admin, and save.

Note: If your bucket restricts network access by IP, add the webhook IP ranges from the IP ranges list to the allowlist.

4. Configure the pipeline in Datadog

From Product Analytics, navigate to App Management in the bottom left of the navigation.
Select your application, then go to Routing > Export Pipelines.
Click New Export Pipeline.

Fill in each section of the side panel:

Define Data to Export

Choose the data you want to export. The source and preset options are clearly labeled in the UI.

File format

Parquet: best for loading directly into a data warehouse such as BigQuery, Snowflake, or Databricks.
JSON: best when you want to process the data with your own pipeline.

Select Storage Type and Configure Bucket

Provider	Fields
Amazon S3	AWS account and role (from your AWS integration); Bucket (required); Path (optional prefix)
Azure Blob Storage	Azure tenant and client (from your Azure integration); Storage Account (required); Container (required); Path (optional prefix)
Google Cloud Storage	GCP service account (from your Google Cloud integration); Bucket (required); Path (optional prefix)

Optionally, click Test Configuration. Datadog writes a small test file to your bucket and reads it back to verify access. Fix any reported permissions or naming issues before saving.
Click Add Export Pipeline to start the pipeline.

Pipeline statuses

Once created, a pipeline shows one of these states on the Export Pipelines page:

Status	Meaning
Active	Datadog is exporting events successfully.
Pending	The pipeline was just created or updated. Allow a few minutes before the first upload.
Error	Datadog could not write to the bucket — typically a permissions or naming issue. Open the pipeline and run Test Configuration to see details.
Paused	The pipeline is disabled. No events are exported.

If a pipeline stays in Pending for more than 15 minutes, run Test Configuration to surface the underlying issue.

File format

Files are organized in a directory structure that makes it easy to query archives by date:

/<path>/dt=<YYYYMMDD>/hour=<HH>/archive_<HHmmss.SSSS>.<UUID>.<ext>

Format	Extension	Notes
JSON	`.json.gz`	Gzip-compressed newline-delimited JSON.
Parquet	`.parquet`	Native Parquet encoding (no gzip wrapper). One row per event with typed columns. Directly loadable by Snowflake, BigQuery, and Databricks without preprocessing.