Language

日本語 English Français 한국어 Español

Datadog Site

Site help
US1 US3 US5 EU AP1 AP2 UK1 US1-FED US2-FED

Create a pipeline

copy Copy pageCopied

POST https://api.ap1.datadoghq.com/api/v1/logs/config/pipelineshttps://api.ap2.datadoghq.com/api/v1/logs/config/pipelineshttps://api.datadoghq.eu/api/v1/logs/config/pipelineshttps://api.ddog-gov.com/api/v1/logs/config/pipelineshttps://api.us2.ddog-gov.com/api/v1/logs/config/pipelineshttps://api.uk1.datadoghq.com/api/v1/logs/config/pipelineshttps://api.datadoghq.com/api/v1/logs/config/pipelineshttps://api.us3.datadoghq.com/api/v1/logs/config/pipelineshttps://api.us5.datadoghq.com/api/v1/logs/config/pipelines

概要

オーガニゼーションにパイプラインを作成します。 This endpoint requires the logs_write_pipelines permission.

リクエスト

Body Data (required)

新しいパイプラインの定義。

Expand All

フィールド

種類

説明

description

string

A description of the pipeline.

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[ <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

default: message

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

default: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp
  • date
  • _timestamp
  • Timestamp
  • eventTime
  • published_date

If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

default: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

default: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

default: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

default: msg

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

default: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

default: attribute

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

default: attribute

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

default: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

default: http.url

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

default: http.url_details

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

default: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

default: http.useragent

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

default: http.useragent_details

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

default: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding name to assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

default: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

default: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

default: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

description

string

A description of the pipeline.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

tags

[string]

A list of tags associated with the pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

default: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

default: network.client.ip

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

default: network.client.geoip

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

default: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

default: lookup-processor

Option 15

object

Note: Reference Tables are in public beta. Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in a Reference Table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_enrichment_table [required]

string

Name of the Reference Table for the source attribute and their associated target attribute values.

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

default: lookup-processor

Option 16

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

default: dd.trace_id

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

default: trace-id-remapper

Option 17

object

There are two ways to define correlation between application spans and logs:

  1. Follow the documentation on how to inject a span ID in the application logs. Log integrations automatically handle all remaining setup steps by default.

  2. Use the span remapper processor to define a log attribute as its associated span ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

default: dd.span_id

type [required]

enum

Type of logs span remapper. Allowed enum values: span-id-remapper

default: span-id-remapper

Option 18

object

A processor for extracting, aggregating, or transforming values from JSON arrays within your logs. Supported operations are:

  • Select value from matching element
  • Compute array length
  • Append a value to an array

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

operation [required]

 <oneOf>

Configuration of the array processor operation to perform.

Option 1

object

Operation that appends a value to a target array attribute.

preserve_source

boolean

Remove or preserve the remapped source element.

default: true

source [required]

string

Attribute path containing the value to append.

target [required]

string

Attribute path of the array to append to.

type [required]

enum

Operation type. Allowed enum values: append

Option 2

object

Operation that computes the length of a source array and stores the result in the target attribute.

source [required]

string

Attribute path of the array to measure.

target [required]

string

Attribute that receives the computed length.

type [required]

enum

Operation type. Allowed enum values: length

Option 3

object

Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.

filter [required]

string

Filter condition expressed as key:value used to find the matching element.

source [required]

string

Attribute path of the array to search into.

target [required]

string

Attribute that receives the extracted value.

type [required]

enum

Operation type. Allowed enum values: select

value_to_extract [required]

string

Key of the value to extract from the matching element.

type [required]

enum

Type of logs array processor. Allowed enum values: array-processor

default: array-processor

Option 19

object

The decoder processor decodes any source attribute containing a base64/base16-encoded UTF-8/ASCII string back to its original value, storing the result in a target attribute.

binary_to_text_encoding [required]

enum

The encoding used to represent the binary data. Allowed enum values: base64,base16

input_representation [required]

enum

The original representation of input string. Allowed enum values: utf_8,integer

is_enabled

boolean

Whether the processor is enabled.

name

string

Name of the processor.

source [required]

string

Name of the log attribute with the encoded data.

target [required]

string

Name of the log attribute that contains the decoded data.

type [required]

enum

Type of logs decoder processor. Allowed enum values: decoder-processor

default: decoder-processor

Option 20

object

A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.

is_enabled

boolean

Whether or not the processor is enabled.

mappers [required]

[ <oneOf>]

The LogsSchemaProcessor mappers.

Option 1

object

The schema remapper maps source log fields to their correct fields.

name [required]

string

Name of the logs schema remapper.

override_on_conflict

boolean

Override or not the target element if already set.

preserve_source

boolean

Remove or preserve the remapped source element.

sources [required]

[string]

Array of source attributes.

target [required]

string

Target field to map log source field to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

type [required]

enum

Type of logs schema remapper. Allowed enum values: schema-remapper

Option 2

object

Use the Schema Category Mapper to categorize log event into enum fields. In the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Categories are executed in order and processing stops at the first match. Make sure categories are properly ordered in case a log could match multiple queries.
  • Sibling fields always have a numerical ID field and a human-readable string name.
  • A fallback section handles cases where the name or ID value matches a specific value. If the name matches "Other" or the ID matches 99, the value of the sibling name field will be pulled from a source field from the original log.

categories [required]

[object]

Array of filters to match or not a log and their corresponding name to assign a custom value to the log.

filter [required]

object

Filter for logs.

query

string

The filter query.

id [required]

int64

ID to inject into the category.

name [required]

string

Value to assign to target schema field.

fallback

object

Used to override hardcoded category values with a value pulled from a source attribute on the log.

sources

object

Fallback sources used to populate value of field.

<any-key>

[string]

values

object

Values that define when the fallback is used.

<any-key>

string

name [required]

string

Name of the logs schema category mapper.

targets [required]

object

Name of the target attributes which value is defined by the matching category.

id

string

ID of the field to map log attributes to.

name

string

Name of the field to map log attributes to.

type [required]

enum

Type of logs schema category mapper. Allowed enum values: schema-category-mapper

name [required]

string

Name of the processor.

schema [required]

object

Configuration of the schema data to use.

class_name [required]

string

Class name of the schema to use.

class_uid [required]

int64

Class UID of the schema to use.

profiles

[string]

Optional list of profiles to modify the schema.

schema_type [required]

string

Type of schema to use.

version [required]

string

Version of the schema to use.

type [required]

enum

Type of logs schema processor. Allowed enum values: schema-processor

default: schema-processor

Option 21

object

Use this processor to remove an attribute from a log during processing. The processor strips the specified attribute from the log event, which is useful when the attribute contains sensitive data or is no longer needed downstream.

attribute_to_exclude [required]

string

Name of the log attribute to remove from the log event.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

type [required]

enum

Type of logs exclude attribute processor. Allowed enum values: exclude-attribute

default: exclude-attribute

Option 22

object

The array-map processor transforms each element of a source array by applying sub-processors in order and collecting the results into a target array. Results can be written to a new array, to the source array (in-place), or to an existing target array. Sub-processors can read from $sourceElem.<field> (object element field), bare $sourceElem (primitive element), or any parent log attribute path. Sub-processors write to $targetElem.<field> (object output field) or bare $targetElem (primitive output).

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

preserve_source

boolean

When false and source != target, the source attribute is removed after processing. Cannot be false when source == target.

default: true

processors [required]

[ <oneOf>]

Sub-processors applied to each element. Allowed types: attribute-remapper, string-builder-processor, arithmetic-processor, category-processor.

Option 1

object

An attribute remapper sub-processor for use inside an array-map processor. Unlike the top-level attribute remapper, is_enabled, source_type, and target_type are not supported.

name

string

Name of the sub-processor.

override_on_conflict

boolean

Override the target element if already set.

preserve_source

boolean

Remove or preserve the remapped source element.

sources [required]

[string]

Array of source attribute paths.

target [required]

string

Target attribute path.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

default: attribute-remapper

Option 2

object

An arithmetic sub-processor for use inside an array-map processor. Unlike the top-level arithmetic processor, is_enabled is not supported.

expression [required]

string

Arithmetic operation to perform.

is_replace_missing

boolean

Replace missing attribute values with 0.

name

string

Name of the sub-processor.

target [required]

string

Target attribute path for the result.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

default: arithmetic-processor

Option 3

object

A string builder sub-processor for use inside an array-map processor. Unlike the top-level string builder processor, is_enabled is not supported.

is_replace_missing

boolean

Replace missing attribute values with an empty string.

name

string

Name of the sub-processor.

target [required]

string

Target attribute path for the result.

template [required]

string

Formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

default: string-builder-processor

Option 4

object

A category sub-processor for use inside an array-map processor. Unlike the top-level category processor, is_enabled is not supported.

categories [required]

[object]

Array of filters to match against a log and the corresponding value to assign.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

name

string

Name of the sub-processor.

target [required]

string

Target attribute path for the category value.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

default: category-processor

source [required]

string

Attribute path of the source array. Elements are read-only via $sourceElem inside sub-processors.

target [required]

string

Attribute path of the output array. Sub-processors write to $targetElem (or $targetElem.<field>) to build each output element.

type [required]

enum

Type of logs array-map processor. Allowed enum values: array-map-processor

default: array-map-processor

tags

[string]

A list of tags associated with the pipeline.

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "name": "testPipelineArrayMap",
  "processors": [
    {
      "type": "array-map-processor",
      "is_enabled": true,
      "name": "map items",
      "source": "items",
      "target": "out",
      "preserve_source": true,
      "processors": [
        {
          "type": "attribute-remapper",
          "sources": [
            "$sourceElem.id"
          ],
          "target": "$targetElem.uid",
          "preserve_source": true
        },
        {
          "type": "string-builder-processor",
          "template": "item-%{$sourceElem.id}",
          "target": "$targetElem.label"
        }
      ]
    }
  ],
  "tags": []
}
{
  "filter": {
    "query": "source:python"
  },
  "name": "testPipelineArrayMapArithmetic",
  "processors": [
    {
      "type": "array-map-processor",
      "is_enabled": true,
      "name": "double counts",
      "source": "items",
      "target": "out",
      "processors": [
        {
          "type": "arithmetic-processor",
          "expression": "$sourceElem.count * 2",
          "target": "$targetElem.doubled"
        }
      ]
    }
  ],
  "tags": []
}
{
  "filter": {
    "query": "source:python"
  },
  "name": "testPipelineArrayMapCategory",
  "processors": [
    {
      "type": "array-map-processor",
      "is_enabled": true,
      "name": "categorize items",
      "source": "items",
      "target": "out",
      "processors": [
        {
          "type": "category-processor",
          "target": "$targetElem.level",
          "categories": [
            {
              "filter": {
                "query": "@$sourceElem.status:error"
              },
              "name": "error"
            },
            {
              "filter": {
                "query": "*"
              },
              "name": "info"
            }
          ]
        }
      ]
    }
  ],
  "tags": []
}

応答

OK

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Expand All

フィールド

種類

説明

description

string

A description of the pipeline.

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[ <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

default: message

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

default: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp
  • date
  • _timestamp
  • Timestamp
  • eventTime
  • published_date

If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

default: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

default: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

default: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

default: msg

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

default: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

default: attribute

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

default: attribute

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

default: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

default: http.url

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

default: http.url_details

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

default: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

default: http.useragent

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

default: http.useragent_details

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

default: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding name to assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

default: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

default: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

default: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

description

string

A description of the pipeline.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

tags

[string]

A list of tags associated with the pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

default: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

default: network.client.ip

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

default: network.client.geoip

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

default: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

default: lookup-processor

Option 15

object

Note: Reference Tables are in public beta. Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in a Reference Table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_enrichment_table [required]

string

Name of the Reference Table for the source attribute and their associated target attribute values.

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

default: lookup-processor

Option 16

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

default: dd.trace_id

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

default: trace-id-remapper

Option 17

object

There are two ways to define correlation between application spans and logs:

  1. Follow the documentation on how to inject a span ID in the application logs. Log integrations automatically handle all remaining setup steps by default.

  2. Use the span remapper processor to define a log attribute as its associated span ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

default: dd.span_id

type [required]

enum

Type of logs span remapper. Allowed enum values: span-id-remapper

default: span-id-remapper

Option 18

object

A processor for extracting, aggregating, or transforming values from JSON arrays within your logs. Supported operations are:

  • Select value from matching element
  • Compute array length
  • Append a value to an array

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

operation [required]

 <oneOf>

Configuration of the array processor operation to perform.

Option 1

object

Operation that appends a value to a target array attribute.

preserve_source

boolean

Remove or preserve the remapped source element.

default: true

source [required]

string

Attribute path containing the value to append.

target [required]

string

Attribute path of the array to append to.

type [required]

enum

Operation type. Allowed enum values: append

Option 2

object

Operation that computes the length of a source array and stores the result in the target attribute.

source [required]

string

Attribute path of the array to measure.

target [required]

string

Attribute that receives the computed length.

type [required]

enum

Operation type. Allowed enum values: length

Option 3

object

Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.

filter [required]

string

Filter condition expressed as key:value used to find the matching element.

source [required]

string

Attribute path of the array to search into.

target [required]

string

Attribute that receives the extracted value.

type [required]

enum

Operation type. Allowed enum values: select

value_to_extract [required]

string

Key of the value to extract from the matching element.

type [required]

enum

Type of logs array processor. Allowed enum values: array-processor

default: array-processor

Option 19

object

The decoder processor decodes any source attribute containing a base64/base16-encoded UTF-8/ASCII string back to its original value, storing the result in a target attribute.

binary_to_text_encoding [required]

enum

The encoding used to represent the binary data. Allowed enum values: base64,base16

input_representation [required]

enum

The original representation of input string. Allowed enum values: utf_8,integer

is_enabled

boolean

Whether the processor is enabled.

name

string

Name of the processor.

source [required]

string

Name of the log attribute with the encoded data.

target [required]

string

Name of the log attribute that contains the decoded data.

type [required]

enum

Type of logs decoder processor. Allowed enum values: decoder-processor

default: decoder-processor

Option 20

object

A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.

is_enabled

boolean

Whether or not the processor is enabled.

mappers [required]

[ <oneOf>]

The LogsSchemaProcessor mappers.

Option 1

object

The schema remapper maps source log fields to their correct fields.

name [required]

string

Name of the logs schema remapper.

override_on_conflict

boolean

Override or not the target element if already set.

preserve_source

boolean

Remove or preserve the remapped source element.

sources [required]

[string]

Array of source attributes.

target [required]

string

Target field to map log source field to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

type [required]

enum

Type of logs schema remapper. Allowed enum values: schema-remapper

Option 2

object

Use the Schema Category Mapper to categorize log event into enum fields. In the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Categories are executed in order and processing stops at the first match. Make sure categories are properly ordered in case a log could match multiple queries.
  • Sibling fields always have a numerical ID field and a human-readable string name.
  • A fallback section handles cases where the name or ID value matches a specific value. If the name matches "Other" or the ID matches 99, the value of the sibling name field will be pulled from a source field from the original log.

categories [required]

[object]

Array of filters to match or not a log and their corresponding name to assign a custom value to the log.

filter [required]

object

Filter for logs.

query

string

The filter query.

id [required]

int64

ID to inject into the category.

name [required]

string

Value to assign to target schema field.

fallback

object

Used to override hardcoded category values with a value pulled from a source attribute on the log.

sources

object

Fallback sources used to populate value of field.

<any-key>

[string]

values

object

Values that define when the fallback is used.

<any-key>

string

name [required]

string

Name of the logs schema category mapper.

targets [required]

object

Name of the target attributes which value is defined by the matching category.

id

string

ID of the field to map log attributes to.

name

string

Name of the field to map log attributes to.

type [required]

enum

Type of logs schema category mapper. Allowed enum values: schema-category-mapper

name [required]

string

Name of the processor.

schema [required]

object

Configuration of the schema data to use.

class_name [required]

string

Class name of the schema to use.

class_uid [required]

int64

Class UID of the schema to use.

profiles

[string]

Optional list of profiles to modify the schema.

schema_type [required]

string

Type of schema to use.

version [required]

string

Version of the schema to use.

type [required]

enum

Type of logs schema processor. Allowed enum values: schema-processor

default: schema-processor

Option 21

object

Use this processor to remove an attribute from a log during processing. The processor strips the specified attribute from the log event, which is useful when the attribute contains sensitive data or is no longer needed downstream.

attribute_to_exclude [required]

string

Name of the log attribute to remove from the log event.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

type [required]

enum

Type of logs exclude attribute processor. Allowed enum values: exclude-attribute

default: exclude-attribute

Option 22

object

The array-map processor transforms each element of a source array by applying sub-processors in order and collecting the results into a target array. Results can be written to a new array, to the source array (in-place), or to an existing target array. Sub-processors can read from $sourceElem.<field> (object element field), bare $sourceElem (primitive element), or any parent log attribute path. Sub-processors write to $targetElem.<field> (object output field) or bare $targetElem (primitive output).

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

preserve_source

boolean

When false and source != target, the source attribute is removed after processing. Cannot be false when source == target.

default: true

processors [required]

[ <oneOf>]

Sub-processors applied to each element. Allowed types: attribute-remapper, string-builder-processor, arithmetic-processor, category-processor.

Option 1

object

An attribute remapper sub-processor for use inside an array-map processor. Unlike the top-level attribute remapper, is_enabled, source_type, and target_type are not supported.

name

string

Name of the sub-processor.

override_on_conflict

boolean

Override the target element if already set.

preserve_source

boolean

Remove or preserve the remapped source element.

sources [required]

[string]

Array of source attribute paths.

target [required]

string

Target attribute path.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

default: attribute-remapper

Option 2

object

An arithmetic sub-processor for use inside an array-map processor. Unlike the top-level arithmetic processor, is_enabled is not supported.

expression [required]

string

Arithmetic operation to perform.

is_replace_missing

boolean

Replace missing attribute values with 0.

name

string

Name of the sub-processor.

target [required]

string

Target attribute path for the result.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

default: arithmetic-processor

Option 3

object

A string builder sub-processor for use inside an array-map processor. Unlike the top-level string builder processor, is_enabled is not supported.

is_replace_missing

boolean

Replace missing attribute values with an empty string.

name

string

Name of the sub-processor.

target [required]

string

Target attribute path for the result.

template [required]

string

Formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

default: string-builder-processor

Option 4

object

A category sub-processor for use inside an array-map processor. Unlike the top-level category processor, is_enabled is not supported.

categories [required]

[object]

Array of filters to match against a log and the corresponding value to assign.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

name

string

Name of the sub-processor.

target [required]

string

Target attribute path for the category value.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

default: category-processor

source [required]

string

Attribute path of the source array. Elements are read-only via $sourceElem inside sub-processors.

target [required]

string

Attribute path of the output array. Sub-processors write to $targetElem (or $targetElem.<field>) to build each output element.

type [required]

enum

Type of logs array-map processor. Allowed enum values: array-map-processor

default: array-map-processor

tags

[string]

A list of tags associated with the pipeline.

type

string

Type of pipeline.

{
  "description": "string",
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar",
        "support_rules": "rule_name_1 foo\nrule_name_2 bar"
      },
      "is_enabled": false,
      "name": "string",
      "samples": [],
      "source": "message",
      "type": "grok-parser"
    }
  ],
  "tags": [],
  "type": "pipeline"
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

フィールド

種類

説明

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

フィールド

種類

説明

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Too many requests

Error response object.

Expand All

フィールド

種類

説明

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

コード例

                          ## default
# 

# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.uk1.datadoghq.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d @- << EOF
{
  "filter": {
    "query": "source:python"
  },
  "is_enabled": true,
  "name": "My Pipeline",
  "processors": []
}
EOF

                        
                          ## default
# 

# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.uk1.datadoghq.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d @- << EOF
{
  "filter": {
    "query": "source:python"
  },
  "is_enabled": true,
  "name": "My Pipeline",
  "processors": []
}
EOF

                        
                          ## default
# 

# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.uk1.datadoghq.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v1/logs/config/pipelines" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d @- << EOF
{
  "filter": {
    "query": "source:python"
  },
  "is_enabled": true,
  "name": "My Pipeline",
  "processors": []
}
EOF

                        
// Create a pipeline with Array Map Processor returns "OK" response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV1"
)

func main() {
	body := datadogV1.LogsPipeline{
		Filter: &datadogV1.LogsFilter{
			Query: datadog.PtrString("source:python"),
		},
		Name: "testPipelineArrayMap",
		Processors: []datadogV1.LogsProcessor{
			datadogV1.LogsProcessor{
				LogsArrayMapProcessor: &datadogV1.LogsArrayMapProcessor{
					Type:           datadogV1.LOGSARRAYMAPPROCESSORTYPE_ARRAY_MAP_PROCESSOR,
					IsEnabled:      datadog.PtrBool(true),
					Name:           datadog.PtrString("map items"),
					Source:         "items",
					Target:         "out",
					PreserveSource: datadog.PtrBool(true),
					Processors: []datadogV1.LogsArrayMapSubProcessor{
						datadogV1.LogsArrayMapSubProcessor{
							LogsArrayMapAttributeRemapper: &datadogV1.LogsArrayMapAttributeRemapper{
								Type: datadogV1.LOGSATTRIBUTEREMAPPERTYPE_ATTRIBUTE_REMAPPER,
								Sources: []string{
									"$sourceElem.id",
								},
								Target:         "$targetElem.uid",
								PreserveSource: datadog.PtrBool(true),
							}},
						datadogV1.LogsArrayMapSubProcessor{
							LogsArrayMapStringBuilderSubProcessor: &datadogV1.LogsArrayMapStringBuilderSubProcessor{
								Type:     datadogV1.LOGSSTRINGBUILDERPROCESSORTYPE_STRING_BUILDER_PROCESSOR,
								Template: "item-%{$sourceElem.id}",
								Target:   "$targetElem.label",
							}},
					},
				}},
		},
		Tags: []string{},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV1.NewLogsPipelinesApi(apiClient)
	resp, r, err := api.CreateLogsPipeline(ctx, body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.CreateLogsPipeline`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LogsPipelinesApi.CreateLogsPipeline`:\n%s\n", responseContent)
}

// Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV1"
)

func main() {
	body := datadogV1.LogsPipeline{
		Filter: &datadogV1.LogsFilter{
			Query: datadog.PtrString("source:python"),
		},
		Name: "testPipelineArrayMapArithmetic",
		Processors: []datadogV1.LogsProcessor{
			datadogV1.LogsProcessor{
				LogsArrayMapProcessor: &datadogV1.LogsArrayMapProcessor{
					Type:      datadogV1.LOGSARRAYMAPPROCESSORTYPE_ARRAY_MAP_PROCESSOR,
					IsEnabled: datadog.PtrBool(true),
					Name:      datadog.PtrString("double counts"),
					Source:    "items",
					Target:    "out",
					Processors: []datadogV1.LogsArrayMapSubProcessor{
						datadogV1.LogsArrayMapSubProcessor{
							LogsArrayMapArithmeticSubProcessor: &datadogV1.LogsArrayMapArithmeticSubProcessor{
								Type:       datadogV1.LOGSARITHMETICPROCESSORTYPE_ARITHMETIC_PROCESSOR,
								Expression: "$sourceElem.count * 2",
								Target:     "$targetElem.doubled",
							}},
					},
				}},
		},
		Tags: []string{},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV1.NewLogsPipelinesApi(apiClient)
	resp, r, err := api.CreateLogsPipeline(ctx, body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.CreateLogsPipeline`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LogsPipelinesApi.CreateLogsPipeline`:\n%s\n", responseContent)
}

// Create a pipeline with Array Map Processor using category sub-processor returns "OK" response

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
	"github.com/DataDog/datadog-api-client-go/v2/api/datadogV1"
)

func main() {
	body := datadogV1.LogsPipeline{
		Filter: &datadogV1.LogsFilter{
			Query: datadog.PtrString("source:python"),
		},
		Name: "testPipelineArrayMapCategory",
		Processors: []datadogV1.LogsProcessor{
			datadogV1.LogsProcessor{
				LogsArrayMapProcessor: &datadogV1.LogsArrayMapProcessor{
					Type:      datadogV1.LOGSARRAYMAPPROCESSORTYPE_ARRAY_MAP_PROCESSOR,
					IsEnabled: datadog.PtrBool(true),
					Name:      datadog.PtrString("categorize items"),
					Source:    "items",
					Target:    "out",
					Processors: []datadogV1.LogsArrayMapSubProcessor{
						datadogV1.LogsArrayMapSubProcessor{
							LogsArrayMapCategorySubProcessor: &datadogV1.LogsArrayMapCategorySubProcessor{
								Type:   datadogV1.LOGSCATEGORYPROCESSORTYPE_CATEGORY_PROCESSOR,
								Target: "$targetElem.level",
								Categories: []datadogV1.LogsCategoryProcessorCategory{
									{
										Filter: &datadogV1.LogsFilter{
											Query: datadog.PtrString("@$sourceElem.status:error"),
										},
										Name: datadog.PtrString("error"),
									},
									{
										Filter: &datadogV1.LogsFilter{
											Query: datadog.PtrString("*"),
										},
										Name: datadog.PtrString("info"),
									},
								},
							}},
					},
				}},
		},
		Tags: []string{},
	}
	ctx := datadog.NewDefaultContext(context.Background())
	configuration := datadog.NewConfiguration()
	apiClient := datadog.NewAPIClient(configuration)
	api := datadogV1.NewLogsPipelinesApi(apiClient)
	resp, r, err := api.CreateLogsPipeline(ctx, body)

	if err != nil {
		fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.CreateLogsPipeline`: %v\n", err)
		fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
	}

	responseContent, _ := json.MarshalIndent(resp, "", "  ")
	fmt.Fprintf(os.Stdout, "Response from `LogsPipelinesApi.CreateLogsPipeline`:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
// Create a pipeline with Array Map Processor returns "OK" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v1.api.LogsPipelinesApi;
import com.datadog.api.client.v1.model.LogsArrayMapAttributeRemapper;
import com.datadog.api.client.v1.model.LogsArrayMapProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapStringBuilderSubProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapSubProcessor;
import com.datadog.api.client.v1.model.LogsAttributeRemapperType;
import com.datadog.api.client.v1.model.LogsFilter;
import com.datadog.api.client.v1.model.LogsPipeline;
import com.datadog.api.client.v1.model.LogsProcessor;
import com.datadog.api.client.v1.model.LogsStringBuilderProcessorType;
import java.util.Arrays;
import java.util.Collections;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);

    LogsPipeline body =
        new LogsPipeline()
            .filter(new LogsFilter().query("source:python"))
            .name("testPipelineArrayMap")
            .processors(
                Collections.singletonList(
                    new LogsProcessor(
                        new LogsArrayMapProcessor()
                            .type(LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR)
                            .isEnabled(true)
                            .name("map items")
                            .source("items")
                            .target("out")
                            .preserveSource(true)
                            .processors(
                                Arrays.asList(
                                    new LogsArrayMapSubProcessor(
                                        new LogsArrayMapAttributeRemapper()
                                            .type(LogsAttributeRemapperType.ATTRIBUTE_REMAPPER)
                                            .sources(Collections.singletonList("$sourceElem.id"))
                                            .target("$targetElem.uid")
                                            .preserveSource(true)),
                                    new LogsArrayMapSubProcessor(
                                        new LogsArrayMapStringBuilderSubProcessor()
                                            .type(
                                                LogsStringBuilderProcessorType
                                                    .STRING_BUILDER_PROCESSOR)
                                            .template("item-%{$sourceElem.id}")
                                            .target("$targetElem.label")))))));

    try {
      LogsPipeline result = apiInstance.createLogsPipeline(body);
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LogsPipelinesApi#createLogsPipeline");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

// Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v1.api.LogsPipelinesApi;
import com.datadog.api.client.v1.model.LogsArithmeticProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapArithmeticSubProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapSubProcessor;
import com.datadog.api.client.v1.model.LogsFilter;
import com.datadog.api.client.v1.model.LogsPipeline;
import com.datadog.api.client.v1.model.LogsProcessor;
import java.util.Collections;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);

    LogsPipeline body =
        new LogsPipeline()
            .filter(new LogsFilter().query("source:python"))
            .name("testPipelineArrayMapArithmetic")
            .processors(
                Collections.singletonList(
                    new LogsProcessor(
                        new LogsArrayMapProcessor()
                            .type(LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR)
                            .isEnabled(true)
                            .name("double counts")
                            .source("items")
                            .target("out")
                            .processors(
                                Collections.singletonList(
                                    new LogsArrayMapSubProcessor(
                                        new LogsArrayMapArithmeticSubProcessor()
                                            .type(LogsArithmeticProcessorType.ARITHMETIC_PROCESSOR)
                                            .expression("$sourceElem.count * 2")
                                            .target("$targetElem.doubled")))))));

    try {
      LogsPipeline result = apiInstance.createLogsPipeline(body);
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LogsPipelinesApi#createLogsPipeline");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

// Create a pipeline with Array Map Processor using category sub-processor returns "OK" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
import com.datadog.api.client.v1.api.LogsPipelinesApi;
import com.datadog.api.client.v1.model.LogsArrayMapCategorySubProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessor;
import com.datadog.api.client.v1.model.LogsArrayMapProcessorType;
import com.datadog.api.client.v1.model.LogsArrayMapSubProcessor;
import com.datadog.api.client.v1.model.LogsCategoryProcessorCategory;
import com.datadog.api.client.v1.model.LogsCategoryProcessorType;
import com.datadog.api.client.v1.model.LogsFilter;
import com.datadog.api.client.v1.model.LogsPipeline;
import com.datadog.api.client.v1.model.LogsProcessor;
import java.util.Arrays;
import java.util.Collections;

public class Example {
  public static void main(String[] args) {
    ApiClient defaultClient = ApiClient.getDefaultApiClient();
    LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);

    LogsPipeline body =
        new LogsPipeline()
            .filter(new LogsFilter().query("source:python"))
            .name("testPipelineArrayMapCategory")
            .processors(
                Collections.singletonList(
                    new LogsProcessor(
                        new LogsArrayMapProcessor()
                            .type(LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR)
                            .isEnabled(true)
                            .name("categorize items")
                            .source("items")
                            .target("out")
                            .processors(
                                Collections.singletonList(
                                    new LogsArrayMapSubProcessor(
                                        new LogsArrayMapCategorySubProcessor()
                                            .type(LogsCategoryProcessorType.CATEGORY_PROCESSOR)
                                            .target("$targetElem.level")
                                            .categories(
                                                Arrays.asList(
                                                    new LogsCategoryProcessorCategory()
                                                        .filter(
                                                            new LogsFilter()
                                                                .query("@$sourceElem.status:error"))
                                                        .name("error"),
                                                    new LogsCategoryProcessorCategory()
                                                        .filter(new LogsFilter().query("*"))
                                                        .name("info")))))))));

    try {
      LogsPipeline result = apiInstance.createLogsPipeline(body);
      System.out.println(result);
    } catch (ApiException e) {
      System.err.println("Exception when calling LogsPipelinesApi#createLogsPipeline");
      System.err.println("Status code: " + e.getCode());
      System.err.println("Reason: " + e.getResponseBody());
      System.err.println("Response headers: " + e.getResponseHeaders());
      e.printStackTrace();
    }
  }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"
"""
Create a pipeline with Array Map Processor returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.logs_pipelines_api import LogsPipelinesApi
from datadog_api_client.v1.model.logs_array_map_attribute_remapper import LogsArrayMapAttributeRemapper
from datadog_api_client.v1.model.logs_array_map_processor import LogsArrayMapProcessor
from datadog_api_client.v1.model.logs_array_map_processor_type import LogsArrayMapProcessorType
from datadog_api_client.v1.model.logs_array_map_string_builder_sub_processor import (
    LogsArrayMapStringBuilderSubProcessor,
)
from datadog_api_client.v1.model.logs_attribute_remapper_type import LogsAttributeRemapperType
from datadog_api_client.v1.model.logs_filter import LogsFilter
from datadog_api_client.v1.model.logs_pipeline import LogsPipeline
from datadog_api_client.v1.model.logs_string_builder_processor_type import LogsStringBuilderProcessorType

body = LogsPipeline(
    filter=LogsFilter(
        query="source:python",
    ),
    name="testPipelineArrayMap",
    processors=[
        LogsArrayMapProcessor(
            type=LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR,
            is_enabled=True,
            name="map items",
            source="items",
            target="out",
            preserve_source=True,
            processors=[
                LogsArrayMapAttributeRemapper(
                    type=LogsAttributeRemapperType.ATTRIBUTE_REMAPPER,
                    sources=[
                        "$sourceElem.id",
                    ],
                    target="$targetElem.uid",
                    preserve_source=True,
                ),
                LogsArrayMapStringBuilderSubProcessor(
                    type=LogsStringBuilderProcessorType.STRING_BUILDER_PROCESSOR,
                    template="item-%{$sourceElem.id}",
                    target="$targetElem.label",
                ),
            ],
        ),
    ],
    tags=[],
)

configuration = Configuration()
with ApiClient(configuration) as api_client:
    api_instance = LogsPipelinesApi(api_client)
    response = api_instance.create_logs_pipeline(body=body)

    print(response)

"""
Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.logs_pipelines_api import LogsPipelinesApi
from datadog_api_client.v1.model.logs_arithmetic_processor_type import LogsArithmeticProcessorType
from datadog_api_client.v1.model.logs_array_map_arithmetic_sub_processor import LogsArrayMapArithmeticSubProcessor
from datadog_api_client.v1.model.logs_array_map_processor import LogsArrayMapProcessor
from datadog_api_client.v1.model.logs_array_map_processor_type import LogsArrayMapProcessorType
from datadog_api_client.v1.model.logs_filter import LogsFilter
from datadog_api_client.v1.model.logs_pipeline import LogsPipeline

body = LogsPipeline(
    filter=LogsFilter(
        query="source:python",
    ),
    name="testPipelineArrayMapArithmetic",
    processors=[
        LogsArrayMapProcessor(
            type=LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR,
            is_enabled=True,
            name="double counts",
            source="items",
            target="out",
            processors=[
                LogsArrayMapArithmeticSubProcessor(
                    type=LogsArithmeticProcessorType.ARITHMETIC_PROCESSOR,
                    expression="$sourceElem.count * 2",
                    target="$targetElem.doubled",
                ),
            ],
        ),
    ],
    tags=[],
)

configuration = Configuration()
with ApiClient(configuration) as api_client:
    api_instance = LogsPipelinesApi(api_client)
    response = api_instance.create_logs_pipeline(body=body)

    print(response)

"""
Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.logs_pipelines_api import LogsPipelinesApi
from datadog_api_client.v1.model.logs_array_map_category_sub_processor import LogsArrayMapCategorySubProcessor
from datadog_api_client.v1.model.logs_array_map_processor import LogsArrayMapProcessor
from datadog_api_client.v1.model.logs_array_map_processor_type import LogsArrayMapProcessorType
from datadog_api_client.v1.model.logs_category_processor_category import LogsCategoryProcessorCategory
from datadog_api_client.v1.model.logs_category_processor_type import LogsCategoryProcessorType
from datadog_api_client.v1.model.logs_filter import LogsFilter
from datadog_api_client.v1.model.logs_pipeline import LogsPipeline

body = LogsPipeline(
    filter=LogsFilter(
        query="source:python",
    ),
    name="testPipelineArrayMapCategory",
    processors=[
        LogsArrayMapProcessor(
            type=LogsArrayMapProcessorType.ARRAY_MAP_PROCESSOR,
            is_enabled=True,
            name="categorize items",
            source="items",
            target="out",
            processors=[
                LogsArrayMapCategorySubProcessor(
                    type=LogsCategoryProcessorType.CATEGORY_PROCESSOR,
                    target="$targetElem.level",
                    categories=[
                        LogsCategoryProcessorCategory(
                            filter=LogsFilter(
                                query="@$sourceElem.status:error",
                            ),
                            name="error",
                        ),
                        LogsCategoryProcessorCategory(
                            filter=LogsFilter(
                                query="*",
                            ),
                            name="info",
                        ),
                    ],
                ),
            ],
        ),
    ],
    tags=[],
)

configuration = Configuration()
with ApiClient(configuration) as api_client:
    api_instance = LogsPipelinesApi(api_client)
    response = api_instance.create_logs_pipeline(body=body)

    print(response)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
# Create a pipeline with Array Map Processor returns "OK" response

require "datadog_api_client"
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new

body = DatadogAPIClient::V1::LogsPipeline.new({
  filter: DatadogAPIClient::V1::LogsFilter.new({
    query: "source:python",
  }),
  name: "testPipelineArrayMap",
  processors: [
    DatadogAPIClient::V1::LogsArrayMapProcessor.new({
      type: DatadogAPIClient::V1::LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
      is_enabled: true,
      name: "map items",
      source: "items",
      target: "out",
      preserve_source: true,
      processors: [
        DatadogAPIClient::V1::LogsArrayMapAttributeRemapper.new({
          type: DatadogAPIClient::V1::LogsAttributeRemapperType::ATTRIBUTE_REMAPPER,
          sources: [
            "$sourceElem.id",
          ],
          target: "$targetElem.uid",
          preserve_source: true,
        }),
        DatadogAPIClient::V1::LogsArrayMapStringBuilderSubProcessor.new({
          type: DatadogAPIClient::V1::LogsStringBuilderProcessorType::STRING_BUILDER_PROCESSOR,
          template: "item-%{$sourceElem.id}",
          target: "$targetElem.label",
        }),
      ],
    }),
  ],
  tags: [],
})
p api_instance.create_logs_pipeline(body)

# Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response

require "datadog_api_client"
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new

body = DatadogAPIClient::V1::LogsPipeline.new({
  filter: DatadogAPIClient::V1::LogsFilter.new({
    query: "source:python",
  }),
  name: "testPipelineArrayMapArithmetic",
  processors: [
    DatadogAPIClient::V1::LogsArrayMapProcessor.new({
      type: DatadogAPIClient::V1::LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
      is_enabled: true,
      name: "double counts",
      source: "items",
      target: "out",
      processors: [
        DatadogAPIClient::V1::LogsArrayMapArithmeticSubProcessor.new({
          type: DatadogAPIClient::V1::LogsArithmeticProcessorType::ARITHMETIC_PROCESSOR,
          expression: "$sourceElem.count * 2",
          target: "$targetElem.doubled",
        }),
      ],
    }),
  ],
  tags: [],
})
p api_instance.create_logs_pipeline(body)

# Create a pipeline with Array Map Processor using category sub-processor returns "OK" response

require "datadog_api_client"
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new

body = DatadogAPIClient::V1::LogsPipeline.new({
  filter: DatadogAPIClient::V1::LogsFilter.new({
    query: "source:python",
  }),
  name: "testPipelineArrayMapCategory",
  processors: [
    DatadogAPIClient::V1::LogsArrayMapProcessor.new({
      type: DatadogAPIClient::V1::LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
      is_enabled: true,
      name: "categorize items",
      source: "items",
      target: "out",
      processors: [
        DatadogAPIClient::V1::LogsArrayMapCategorySubProcessor.new({
          type: DatadogAPIClient::V1::LogsCategoryProcessorType::CATEGORY_PROCESSOR,
          target: "$targetElem.level",
          categories: [
            DatadogAPIClient::V1::LogsCategoryProcessorCategory.new({
              filter: DatadogAPIClient::V1::LogsFilter.new({
                query: "@$sourceElem.status:error",
              }),
              name: "error",
            }),
            DatadogAPIClient::V1::LogsCategoryProcessorCategory.new({
              filter: DatadogAPIClient::V1::LogsFilter.new({
                query: "*",
              }),
              name: "info",
            }),
          ],
        }),
      ],
    }),
  ],
  tags: [],
})
p api_instance.create_logs_pipeline(body)

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
// Create a pipeline with Array Map Processor returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV1::api_logs_pipelines::LogsPipelinesAPI;
use datadog_api_client::datadogV1::model::LogsArrayMapAttributeRemapper;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapStringBuilderSubProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapSubProcessor;
use datadog_api_client::datadogV1::model::LogsAttributeRemapperType;
use datadog_api_client::datadogV1::model::LogsFilter;
use datadog_api_client::datadogV1::model::LogsPipeline;
use datadog_api_client::datadogV1::model::LogsProcessor;
use datadog_api_client::datadogV1::model::LogsStringBuilderProcessorType;

#[tokio::main]
async fn main() {
    let body = LogsPipeline::new("testPipelineArrayMap".to_string())
        .filter(LogsFilter::new().query("source:python".to_string()))
        .processors(vec![LogsProcessor::LogsArrayMapProcessor(Box::new(
            LogsArrayMapProcessor::new(
                vec![
                    LogsArrayMapSubProcessor::LogsArrayMapAttributeRemapper(Box::new(
                        LogsArrayMapAttributeRemapper::new(
                            vec!["$sourceElem.id".to_string()],
                            "$targetElem.uid".to_string(),
                            LogsAttributeRemapperType::ATTRIBUTE_REMAPPER,
                        )
                        .preserve_source(true),
                    )),
                    LogsArrayMapSubProcessor::LogsArrayMapStringBuilderSubProcessor(Box::new(
                        LogsArrayMapStringBuilderSubProcessor::new(
                            "$targetElem.label".to_string(),
                            "item-%{$sourceElem.id}".to_string(),
                            LogsStringBuilderProcessorType::STRING_BUILDER_PROCESSOR,
                        ),
                    )),
                ],
                "items".to_string(),
                "out".to_string(),
                LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
            )
            .is_enabled(true)
            .name("map items".to_string())
            .preserve_source(true),
        ))])
        .tags(vec![]);
    let configuration = datadog::Configuration::new();
    let api = LogsPipelinesAPI::with_config(configuration);
    let resp = api.create_logs_pipeline(body).await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

// Create a pipeline with Array Map Processor using arithmetic sub-processor
// returns "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV1::api_logs_pipelines::LogsPipelinesAPI;
use datadog_api_client::datadogV1::model::LogsArithmeticProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapArithmeticSubProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapSubProcessor;
use datadog_api_client::datadogV1::model::LogsFilter;
use datadog_api_client::datadogV1::model::LogsPipeline;
use datadog_api_client::datadogV1::model::LogsProcessor;

#[tokio::main]
async fn main() {
    let body = LogsPipeline::new("testPipelineArrayMapArithmetic".to_string())
        .filter(LogsFilter::new().query("source:python".to_string()))
        .processors(vec![LogsProcessor::LogsArrayMapProcessor(Box::new(
            LogsArrayMapProcessor::new(
                vec![
                    LogsArrayMapSubProcessor::LogsArrayMapArithmeticSubProcessor(Box::new(
                        LogsArrayMapArithmeticSubProcessor::new(
                            "$sourceElem.count * 2".to_string(),
                            "$targetElem.doubled".to_string(),
                            LogsArithmeticProcessorType::ARITHMETIC_PROCESSOR,
                        ),
                    )),
                ],
                "items".to_string(),
                "out".to_string(),
                LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
            )
            .is_enabled(true)
            .name("double counts".to_string()),
        ))])
        .tags(vec![]);
    let configuration = datadog::Configuration::new();
    let api = LogsPipelinesAPI::with_config(configuration);
    let resp = api.create_logs_pipeline(body).await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

// Create a pipeline with Array Map Processor using category sub-processor returns
// "OK" response
use datadog_api_client::datadog;
use datadog_api_client::datadogV1::api_logs_pipelines::LogsPipelinesAPI;
use datadog_api_client::datadogV1::model::LogsArrayMapCategorySubProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessor;
use datadog_api_client::datadogV1::model::LogsArrayMapProcessorType;
use datadog_api_client::datadogV1::model::LogsArrayMapSubProcessor;
use datadog_api_client::datadogV1::model::LogsCategoryProcessorCategory;
use datadog_api_client::datadogV1::model::LogsCategoryProcessorType;
use datadog_api_client::datadogV1::model::LogsFilter;
use datadog_api_client::datadogV1::model::LogsPipeline;
use datadog_api_client::datadogV1::model::LogsProcessor;

#[tokio::main]
async fn main() {
    let body = LogsPipeline::new("testPipelineArrayMapCategory".to_string())
        .filter(LogsFilter::new().query("source:python".to_string()))
        .processors(vec![LogsProcessor::LogsArrayMapProcessor(Box::new(
            LogsArrayMapProcessor::new(
                vec![LogsArrayMapSubProcessor::LogsArrayMapCategorySubProcessor(
                    Box::new(LogsArrayMapCategorySubProcessor::new(
                        vec![
                            LogsCategoryProcessorCategory::new()
                                .filter(
                                    LogsFilter::new()
                                        .query("@$sourceElem.status:error".to_string()),
                                )
                                .name("error".to_string()),
                            LogsCategoryProcessorCategory::new()
                                .filter(LogsFilter::new().query("*".to_string()))
                                .name("info".to_string()),
                        ],
                        "$targetElem.level".to_string(),
                        LogsCategoryProcessorType::CATEGORY_PROCESSOR,
                    )),
                )],
                "items".to_string(),
                "out".to_string(),
                LogsArrayMapProcessorType::ARRAY_MAP_PROCESSOR,
            )
            .is_enabled(true)
            .name("categorize items".to_string()),
        ))])
        .tags(vec![]);
    let configuration = datadog::Configuration::new();
    let api = LogsPipelinesAPI::with_config(configuration);
    let resp = api.create_logs_pipeline(body).await;
    if let Ok(value) = resp {
        println!("{:#?}", value);
    } else {
        println!("{:#?}", resp.unwrap_err());
    }
}

Instructions

First install the library and its dependencies and then save the example to src/main.rs and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" cargo run
/**
 * Create a pipeline with Array Map Processor returns "OK" response
 */

import { client, v1 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

const params: v1.LogsPipelinesApiCreateLogsPipelineRequest = {
  body: {
    filter: {
      query: "source:python",
    },
    name: "testPipelineArrayMap",
    processors: [
      {
        type: "array-map-processor",
        isEnabled: true,
        name: "map items",
        source: "items",
        target: "out",
        preserveSource: true,
        processors: [
          {
            type: "attribute-remapper",
            sources: ["$sourceElem.id"],
            target: "$targetElem.uid",
            preserveSource: true,
          },
          {
            type: "string-builder-processor",
            template: "item-%{$sourceElem.id}",
            target: "$targetElem.label",
          },
        ],
      },
    ],
    tags: [],
  },
};

apiInstance
  .createLogsPipeline(params)
  .then((data: v1.LogsPipeline) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

/**
 * Create a pipeline with Array Map Processor using arithmetic sub-processor returns "OK" response
 */

import { client, v1 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

const params: v1.LogsPipelinesApiCreateLogsPipelineRequest = {
  body: {
    filter: {
      query: "source:python",
    },
    name: "testPipelineArrayMapArithmetic",
    processors: [
      {
        type: "array-map-processor",
        isEnabled: true,
        name: "double counts",
        source: "items",
        target: "out",
        processors: [
          {
            type: "arithmetic-processor",
            expression: "$sourceElem.count * 2",
            target: "$targetElem.doubled",
          },
        ],
      },
    ],
    tags: [],
  },
};

apiInstance
  .createLogsPipeline(params)
  .then((data: v1.LogsPipeline) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

/**
 * Create a pipeline with Array Map Processor using category sub-processor returns "OK" response
 */

import { client, v1 } from "@datadog/datadog-api-client";

const configuration = client.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

const params: v1.LogsPipelinesApiCreateLogsPipelineRequest = {
  body: {
    filter: {
      query: "source:python",
    },
    name: "testPipelineArrayMapCategory",
    processors: [
      {
        type: "array-map-processor",
        isEnabled: true,
        name: "categorize items",
        source: "items",
        target: "out",
        processors: [
          {
            type: "category-processor",
            target: "$targetElem.level",
            categories: [
              {
                filter: {
                  query: "@$sourceElem.status:error",
                },
                name: "error",
              },
              {
                filter: {
                  query: "*",
                },
                name: "info",
              },
            ],
          },
        ],
      },
    ],
    tags: [],
  },
};

apiInstance
  .createLogsPipeline(params)
  .then((data: v1.LogsPipeline) => {
    console.log(
      "API called successfully. Returned data: " + JSON.stringify(data)
    );
  })
  .catch((error: any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

    
DD_SITE="datadoghq.comus3.datadoghq.comus5.datadoghq.comdatadoghq.euap1.datadoghq.comap2.datadoghq.comuk1.datadoghq.comddog-gov.comus2.ddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"