Overview

Data Observability CI/CD checks run automatically when you open a pull request (PR) that modifies dbt models. The checks give you the information you need to decide whether a change is safe to merge.

Datadog posts the results as a comment on your PR, and the comment updates each time you push new changes. A full report is also available in Datadog, and you get a link to it in the PR comment.

Check types

Impact Lineage

Impact Lineage builds a graph of everything downstream of your changed dbt models. Use it to assess the blast radius of a change before merging. See which tables, dashboards, and other consumers depend on the models you modified, and route review to the right owners.

See Lineage for more detail on how Datadog builds and navigates lineage graphs.

Drift Detection

Drift Detection compares the data produced by your models before and after your changes using a series of statistical checks. Use it to confirm that a model change produces the expected output, or to catch unintended side effects such as significant row-count changes, null-rate shifts, or cardinality changes in a column’s values.

Setup

1. Connect your source control provider and dbt project

1. Connect your source-control provider. CI/CD checks support GitHub and GitLab.
2. Connect the supported data source account where your dbt models run.
3. Connect your dbt Cloud or dbt Core project to Datadog. You can also connect your dbt project while configuring CI/CD checks.

2. Select your dbt project and repository

1. From the CI/CD settings, click Add CI/CD Checks.
2. Select the dbt project you want to add checks for.
3. Select the main job for the project. This is the job with the most knowledge of your dbt schema.
4. If Datadog doesn’t automatically infer the repository from your source-control provider, select it manually.

Advanced settings

If your dbt project doesn’t live at the root of your repository, you can specify the path to your dbt project in the advanced settings.

3. Configure checks

You can enable each check independently. Enabling all checks yields the richest reports.

Impact Lineage

Impact lineage generates a graph of the downstream assets that may be affected by your model changes.

General settings

Drift Detection

Drift detection compares the current state of your data on the branch to a baseline and flags any deviations. For Datadog to understand which models ran as part of a CI pipeline, you must send OpenLineage events from your CI job. Datadog uses these events as triggers for drift detection checks. See the OpenLineage setup documentation for instructions on how to set up OpenLineage.

General settings

dbt Cloud

dbt Core

Further reading

お役に立つドキュメント、リンクや記事:

Data Observability OverviewDOCUMENTATION Data CatalogDOCUMENTATION LineageDOCUMENTATION Quality MonitoringDOCUMENTATION Jobs MonitoringDOCUMENTATION