Overview

Custom application or non-standard logs can often be hard to parse into structured formats. To address this issue, use the Grok Parser processor to generate parsing rules using AI, apply library rules for vendor-specific formats (such as Apache, Airflow, and MySQL), or create your own parsing rules. Then, test the rules on sample data to validate the syntax and preview the parsed log output.

Notes:

• You must create a separate Grok Parser for each field you want to parse.
• A log is only parsed by the first rule it matches, so the order of your rules matters.
• If you are using Worker versions older than 2.17, your logs must have the source or ddsource field and the message field for the processor to parse them.

Setup

The Grok Parser processor does the following:

1. Uses the processor-level filter query to determine which logs are sent to the parser.
2. Identifies the specified field to parse on the log.
3. (Preview) Uses the rule-level filter query to apply the first parsing rule that matches the log.
4. Overwrites the specified log field with the rule’s output, and then sends the log to the next step in the pipeline.

To set up the Grok Parser processor:

1. Define a processor-level filter query. Only logs that match this filter query are sent to the parser. All logs, regardless of whether they are parsed by the processor, are sent to the next step in the pipeline. See Logs Search Syntax for information on creating queries.
2. Enter the log field to parse on. For example, if you enter logmessage, the content of the logmessage attribute is parsed. If no field is specified, message is the default field used.
3. Toggle Enable Library Rules off to disable all library parsing rules.
   Notes:
   • You must create a custom parsing rule before you can disable library rules.
   • Library rules are applied by default. Disable library rules only if you are relying on custom parsing rules.
4. Click View Library Rules to preview preset rules for integrations. You can test out-of-the-box parsing rules with your log samples. See Library rules for more information.

Create an AI-generated or custom parsing rule

To set up an AI-assisted or custom parsing rule, click Create Parsing Rules on the Grok Parser processor:

1. Enter a name for the parsing rule.
2. (Preview) Enter a filter query to define which logs this rule applies to. The Grok Parser runs a rule only if a log matches the per-rule filter query, which lets you apply different parsing rules on different log formats. See Logs Search Syntax for information on creating queries.
3. Enter a log sample you want to parse. Samples can be copied from Live Capture or pasted in from another source.
4. (Preview) Click Generate New Rule to have AI generate a new parsing rule based on the sample log. Otherwise, see Manually write rules to write your own rules.
   1. Review the parsed log in the Preview Changes panel.
   2. Click Generate New Rule to re-run the AI rule generator or manually update the rule so the log is parsed correctly. See Parsing for information on writing parsing rules.
      Notes:
      • If you re-run the AI rule generator, a new rule is created. You must manually delete previously AI-created rules if you don’t want them.
      • You can run the AI rule generator a maximum of three times per sample.
   3. Repeat step 4 to create rules based on additional sample logs. See Order of custom rules for information on how rule order determines which rule parses a log.
5. After you add a rule, you can add library rules by selecting a library rule from the reference a library rule dropdown menu. You can add multiple library rules. See Library rules for more information.
6. Click Advanced Settings if you want to add helper rules. See Using helper rules to reuse common patterns for more information.
7. Click Create Rule.

If a log is sent to the parser but is not parsed by any rules, the Worker generates a log with the error: The parser failed to apply rule.

Order of custom rules

When you have multiple custom rules for a Grok Parser processor, a log is parsed by the first rule whose query it matches, and then sent to the next step in the pipeline. The processor does not try to match the log to subsequent rules. Therefore, the order of the rules matters if a log might match multiple rules. To reorder rules, drag and drop them into the desired order.

Example

Consider a parser with these parsing rules:

1. Rule Example 1
2. Rule Example 2
3. Rule Example 3

If a log sent to the parser matches all three rule queries, the log is only parsed by Rule Example 1, because it’s listed before rules 2 and 3.

Manually write rules

To write parsing rules manually, in the Create Parsing Rule modal:

1. Click write rules manually.
2. Enter the rules for parsing the logs. See Parsing for information on writing parsing rules with Datadog Grok patterns. Note: The url, useragent, and csv filters are not available.
3. Review the parsed log in the Preview Changes panel and update the rule until the log is parsed as expected.
4. Click Add rule to manually write another rule.

Library rules

When a log is sent to the parser, library rules are automatically applied to the log if there is a source or ddsource field. For example, if a log has source:mysql, the parser applies the MySQL library rules to that log. To browse all available library rules, click View Library Rules in the Grok Parser processor. You can search the table of library rules and click any rule to preview how it is applied to your logs.

You can also add library rules when you create a custom rule. See Create an AI-assisted or custom parsing rule for more information.

Metrics

For component metrics and processor buffer metrics emitted by all processors, see the Pipelines Usage Metrics documentation. To filter or group by Parse processor metrics, use the tag component_type:parse.

Further reading

Additional helpful documentation, links, and articles:

Simplify log collection and aggregation for MSSPs with Datadog Observability PipelinesBLOG