Use Observability Pipelines’ Amazon OpenSearch destination to send logs to Amazon OpenSearch.

Setup

Set up the Amazon OpenSearch destination and its environment variables when you set up a pipeline. The information below is configured in the pipelines UI.

Set up the destination

1. Optionally, enter the name of the Amazon OpenSearch index. See template syntax if you want to route logs to different indexes based on specific fields in your logs.
2. Select an authentication strategy, Basic or AWS. For AWS, enter the AWS region.
3. Optionally, toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing logs to disk, ensuring buffered logs persist through a Worker restart. See Configurable buffers for destinations for more information.
   • If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
   • To configure a buffer on your destination:
     1. Select the buffer type you want to set (Memory or Disk).
     2. Enter the buffer size and select the unit.
        • Maximum memory buffer size is 128 GB.
        • Maximum disk buffer size is 500 GB.

Set the environment variables

• Amazon OpenSearch authentication username:
  • Stored in the environment variable DD_OP_DESTINATION_AMAZON_OPENSEARCH_USERNAME.
• Amazon OpenSearch authentication password:
  • Stored in the environment variable DD_OP_DESTINATION_AMAZON_OPENSEARCH_PASSWORD.
• Amazon OpenSearch endpoint URL:
  • Stored in the environment variable DD_OP_DESTINATION_AMAZON_OPENSEARCH_ENDPOINT_URL.

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.