Overview

Use Observability Pipelines’ ClickHouse destination to send logs to a ClickHouse server over the HTTP interface. The destination supports JSON insert formats for mapping events to columns by name or storing raw payloads, and Apache Arrow IPC streaming for higher-throughput inserts.

Prerequisites

Before you configure the ClickHouse destination, you must have:

• A running ClickHouse server that the Observability Pipelines Worker can access using the HTTP interface.
• A ClickHouse database and target table to which the Worker sends events.
• (Optional) Credentials for a ClickHouse user that has INSERT permission on the target table. The Worker authenticates with HTTP basic authentication.
• (Optional) TLS certificates if your ClickHouse server requires HTTPS with client certificates.

Setup

Configure the ClickHouse destination when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The steps in this section are configured in the UI.

After you select the ClickHouse destination in the pipeline UI:

1. Enter the identifier for your ClickHouse HTTP endpoint URL. If you leave it blank, the default is used.

2. (Optional) Enter the identifier for your ClickHouse username. If you leave it blank, the default is used.

3. (Optional) Enter the identifier for your ClickHouse password. If you leave it blank, the default is used.

4. (Optional) In the Database field, enter the name of the ClickHouse database that contains the table. If you leave it blank, the ClickHouse user’s default database is used.

5. In the Table field, enter the name of the ClickHouse table to insert events into.

6. In the Format dropdown menu, select the insert format for events:

   • json_each_row (default): Inserts each event as a JSON object on its own line. Event fields are mapped to columns of the same name. This maps to ClickHouse’s JSONEachRow format.
   • json_as_object: Inserts each event into a single JSON-typed column. This maps to ClickHouse’s JSONAsObject format.
   • json_as_string: Inserts each event into a single String-typed column, storing the raw JSON. This maps to ClickHouse’s JSONAsString format.
   • arrow_stream: Batches events using the Apache Arrow IPC streaming format. When you select this format, you must also configure Enable batching.

   Note: When you use arrow_stream, your ClickHouse server must be reachable when your worker loads the pipeline configuration from the remote configuration backend, or upon worker restart. The Clickhouse table schema is not persisted between restarts.

Optional settings

Skip unknown fields

In the Skip unknown fields dropdown menu, select either Drop Unknown Fields or Fail On Unknown Fields for fields that are not present in the target table schema instead of returning an insert error. The default is Use ClickHouse Server, which uses the input_format_skip_unknown_fields setting.

Date time best effort

Toggle Date time best effort to enable flexible DateTime parsing on the ClickHouse server. When enabled, the server accepts a wider range of date and time string formats. See ClickHouse’s date_time_input_format setting for more information.

Compression

Toggle Enable Compression to compress outbound HTTP requests with gzip. You can optionally set the Compression level to an integer from 1 (fastest) to 9 (best compression). If left unset, the default is 6.

Enable TLS

Toggle the switch to Enable TLS.

• If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
• The following certificate and key files are required:
  • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  • Notes:
    • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
    • The file must be readable by the observability-pipelines-worker group and user.

Enable batching

Batch encoding is required when Format is set to arrow_stream and is optional for JSON formats.

Toggle Enable batching for batch encoding.

1. Toggle Allow nullable fields to allow null values for non-nullable columns in the target table. When this setting is off (default), missing values for non-nullable columns results in encoding errors.
2. In the Max events field, enter the maximum number of events per batch. Must be 1 or greater.
3. In the Timeout (secs) field, enter the maximum time, in seconds, before a partial batch is flushed. Must be between 1 and 65535. If left unset, the default is 1 second.

See Event batching for more information.

Secret defaults

These are the defaults used for secret identifiers and environment variables.

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. Batching is also configurable for this destination. See event batching for more information.

Further reading

Additional helpful documentation, links, and articles:

Observability PipelinesDOCUMENTATION