Overview

Use Observability Pipelines’ Datadog Logs destination to send logs to Datadog Log Management. You can also use AWS PrivateLink to send logs from Observability Pipelines to Datadog.

Setup

Configure the Datadog Logs destination when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The steps in this section are configured in the UI.

Optional settings

After you select the Datadog Logs destination in the pipeline UI, you can configure these optional settings.

Route logs to multiple Datadog organizations

You can route logs to multiple Datadog organizations. After routing has been set up, you can view metrics for the component or specific organizations to which you are routing logs.

Note: You can route up to 100 Datadog organizations.

Click Route to Multiple Organizations to set up routing to multiple Datadog organizations.

• If you haven’t added any organizations yet, enter organization details as described in the Add a Datadog organization section.
• If you have already added organizations, you can:
  • Click on an organization in the table to edit or delete it.
  • Use the search bar to find a specific organization by name, filter query, or Datadog site, and then select the organization to edit or delete it.
  • View metrics for an organization.
  • Click Add organization to route to another Datadog organization.

Note: If you don’t set up routing to multiple Datadog organizations, logs are routed to the default Datadog organization. This is the organization tied to the API key when you install the Worker.

Add an organization

1. Enter a name for the organization.
   • Note: The name does not have to correspond to the actual name of the Datadog organization.
2. Define a filter query. Only logs that match the specified filter query are sent to the organization. See Observability Pipelines Search Syntax for more information on writing filter queries.
3. Select the Datadog organization’s site.
4. Enter the identifier for the API key for that Datadog organization.
   • Note: Only enter the identifier for the API key. Do not enter the actual API key.
5. Click Save.

Buffering

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Secret defaults

Note: If you entered identifiers for yours secrets and then choose to use environment variables, the environment variable is the identifier entered prepended with DD_OP. For example, if you entered PASSWORD_1 for the a password identifier, the environment variable for the password is DD_OP_PASSWORD_1.

View metrics for the component or specific organizations

You can view metrics at the component level or organization level.

Component-level metrics

To view metrics for the overall Datadog Logs destination:

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the cog on the Datadog Logs destination and select View details.

Note: The Data dropped (intentional) metric shows logs that didn’t match any of the organizations’ filters.

Organization-level metrics

To view metrics for a specific Datadog organization:

1. Navigate to Observability Pipelines.
2. Select your pipeline.
3. Click the Datadog Logs destination so the organizations show up.

4. Click the organization you want to see metrics for.
5. Click View Health Metrics.

Alternatively, click Review Configured Organizations in the Datadog Logs destination. Then, click the graph icon in the Metrics column for the organization.

Metrics

For component metrics and destination buffer metrics emitted by all destinations, see the Pipelines Usage Metrics documentation.

Datadog Logs metrics

• Use the component_id tag to filter or group by individual components.
• The component_type tag is datadog_logs for Datadog Logs destination metrics.

pipelines.datadog_logs_reserved_attribute_conflicts_total

Description: The number of conflicts encountered when relocating fields with semantic meaning to a Datadog reserved attribute. See the example. Available in Worker version 2.18 and later.

Metric type: count

Example of relocating fields with semantic meaning to a Datadog reserved attribute

The OpenTelemetry source decodes the following event, where severity_text semantically maps to the reserved status attribute:

{
  "message": "GET /api/users returned 404",
  "severity_text": "WARN",
  "attributes": {
    "status": 404,
    "http.method": "GET"
  },
  "timestamp": "..."
}

A processor then flattens the event, so that status and severity_text both exist at the top level:

{
  "message": "GET /api/users returned 404",
  "severity_text": "WARN",
  "status": 404,
  "http.method": "GET",
  "timestamp": "..."
}

Because the reserved status attribute already exists, the destination renames it to _RESERVED_severity to avoid it being overridden by the conflicting field:

{
  "message": "GET /api/users returned 404",
  "status": "WARN",
  "_RESERVED_severity": 404,
  "http.method": "GET",
  "timestamp": "..."
}

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.