For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/observability_pipelines/sources/google_pubsub.md. A documentation index is available at /llms.txt.

Google Pub/Sub Source

This product is not supported for your selected Datadog site. ().
Available for:

Logs

Overview

Use Observability Pipelines’ Google Pub/Sub source to pull logs from the Google Cloud Pub/Sub messaging system.

Prerequisites

To use Observability Pipelines’ Google Pub/Sub source, you need the following information available:

  • The Google Pub/Sub source requires a Pub/Sub subscription.
  • The Worker uses standard Google authentication methods. See Authentication methods at Google for more information about choosing the authentication method for your use case.
  • Use roles/pubsub.subscriber for the Pub/Sub IAM role. See Available Pub/Sub roles for more information.

Setup

Set up this source when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The instructions in this section are for setting up the source in the UI.

After you select the Google Pub/Sub source in the pipeline UI:

  1. Enter the name of the source project.
  2. If you have a credentials JSON file, enter the path to your credentials JSON file. The credentials file must be placed under DD_OP_DATA_DIR/config. Alternatively, you can use the GOOGLE_APPLICATION_CREDENTIALS environment variable to provide the credential path.
  3. Enter the subscription name.
  4. Select the decoder you want to use (Bytes, GELF, JSON, syslog).

Optional TLS settings

For Secrets Management: Only enter the identifier for the TLS key pass. Do not enter the actual value.
If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP_. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

Toggle the switch to Enable TLS.

  • If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
  • The following certificate and key files are required:
    • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
    • Notes:
      • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
      • The file must be readable by the observability-pipelines-worker group and user.

Secret defaults

These are the defaults used for secret identifiers and environment variables.

    There are no default secret identifiers for this source.

    There are no environment variables for the Google Pub/Sub source.

    Metrics

    For component metrics and source buffer metrics emitted by all sources, see the Pipelines Usage Metrics documentation. To filter or group by Google Pub/Sub source metrics, use the tag component_type:gcp_pubsub.