For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/observability_pipelines/sources/amazon_s3.md. A documentation index is available at /llms.txt.

Amazon S3 Source

This product is not supported for your selected Datadog site. ().
Available for:

Logs

Overview

Use Observability Pipelines’ Amazon S3 source to receive logs from Amazon S3.

Prerequisites

To use Observability Pipelines’ Amazon S3 source, you must configure a SQS queue to receive your S3 bucket notifications.

Setup

Set up this source when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The instructions in this section are for setting up the source in the UI.

For Secrets Management: Only enter the identifiers for the Amazon S3 URL and, if applicable, the TLS key pass. Do not enter the actual values.
If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP_. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

After you select the Amazon S3 source in the pipeline UI:

  1. Enter the identifier for your Amazon S3 URL. If you leave it blank, the default is used.
  2. Enter the AWS region.

Optional settings

AWS authentication

Select an AWS authentication option. If you select Assume role:

  1. Enter the ARN of the IAM role you want to assume.
  2. Optionally, enter the assumed role session name and external ID.

Enable TLS

Toggle the switch to Enable TLS.

  • If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
  • The following certificate and key files are required:
    • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
    • Notes:
      • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
      • The file must be readable by the observability-pipelines-worker group and user.

Secret defaults

These are the defaults used for secret identifiers and environment variables.

    • Amazon S3 URL identifier:
      • References the URL of the SQS queue to which the S3 bucket sends the notification events.
      • The default identifier is SOURCE_AWS_S3_SQS_URL.
    • Amazon S3 TLS passphrase identifier (when TLS is enabled):
      • The default identifier is SOURCE_AWS_S3_KEY_PASS.
    • Amazon S3 SQS URL:
      • The URL of the SQS queue to which the S3 bucket sends the notification events.
      • The default environment variable is DD_OP_SOURCE_AWS_S3_SQS_URL
    • AWS_CONFIG_FILE path:
      • The path to the AWS configuration file local to this node.
      • The default environment variable is AWS_CONFIG_FILE.
    • AWS_PROFILE name:
      • The name of the profile to use within these files.
      • The default environment variable is AWS_PROFILE.
    • AWS S3 TLS passphrase (when enabled):
      • The default environment variable is DD_OP_SOURCE_AWS_S3_KEY_PASS.

    AWS Authentication

    The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

    Permissions

    For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

    • s3:GetObject
    • sqs:ReceiveMessage
    • sqs:DeleteMessage

    Metrics

    For component metrics and source buffer metrics emitted by all sources, see the Pipelines Usage Metrics documentation. To filter or group by Amazon S3 source metrics, use the tag component_type:aws_s3.

    Amazon S3 metrics

    • Use the component_id tag to filter or group by individual components.
    • Use the component_type tag to filter or group by the source type.
    pipelines.sqs_message_received_messages_total
    Description: The number of SQS messages received.
    Metric type: count
    pipelines.sqs_message_processing_succeeded_total
    Description: The number of SQS messages successfully processed.
    Metric type: count
    pipelines.sqs_message_delete_succeeded_total
    Description: The number of successful deletions of SQS messages.
    Metric type: count
    pipelines.sqs_message_defer_succeeded_total
    Description: The number of SQS messages for which visibility-timeout deferral succeeded.
    Metric type: count
    pipelines.sqs_s3_event_record_ignored_total
    Description: The number of S3 event records in an SQS message that were ignored because they were not ObjectCreated event kinds.
    Metric type: count
    pipelines.s3_object_processing_succeeded_duration_seconds
    Description: Time, in seconds, taken to successfully process an S3 object.
    Metric type: distribution
    pipelines.s3_object_processing_failed_duration_seconds
    Description: Time, in seconds, taken to process an S3 object that failed.
    Metric type: distribution