---
title: ClickHouse Destination
description: Learn how to configure the ClickHouse destination.
breadcrumbs: Docs > Observability Pipelines > Destinations > ClickHouse Destination
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# ClickHouse Destination

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}
Available for:
{% icon name="icon-logs" /%}
 Logs 
## Overview{% #overview %}

Use Observability Pipelines' ClickHouse destination to send logs to a [ClickHouse](https://clickhouse.com/docs) server over the HTTP interface. The destination supports JSON insert formats for mapping events to columns by name or storing raw payloads, and Apache Arrow IPC streaming for higher-throughput inserts.

## Prerequisites{% #prerequisites %}

Before you configure the ClickHouse destination, you must have:

- A running ClickHouse server that the Observability Pipelines Worker can access using the [HTTP interface](https://clickhouse.com/docs/interfaces/http).
- A ClickHouse database and target table to which the Worker sends events.
- (Optional) Credentials for a ClickHouse user that has `INSERT` permission on the target table. The Worker authenticates with HTTP basic authentication.
- (Optional) TLS certificates if your ClickHouse server requires HTTPS with client certificates.

## Setup{% #setup %}

Configure the ClickHouse destination when you [set up a pipeline](https://docs.datadoghq.com/observability_pipelines/configuration/set_up_pipelines.md). You can set up a pipeline in the [UI](https://app.datadoghq.com/observability-pipelines), using the [API](https://docs.datadoghq.com/api/latest/observability-pipelines.md), or with [Terraform](https://registry.terraform.io/providers/datadog/datadog/latest/docs/resources/observability_pipeline). The steps in this section are configured in the UI.

{% alert level="danger" %}
Only enter the identifiers for the ClickHouse endpoint URL and, if applicable, the username, password, and TLS key pass. Do not enter the actual values.
{% /alert %}

After you select the ClickHouse destination in the pipeline UI:

1. Enter the identifier for your ClickHouse HTTP endpoint URL. If you leave it blank, the default is used.

1. (Optional) Enter the identifier for your ClickHouse username. If you leave it blank, the default is used.

1. (Optional) Enter the identifier for your ClickHouse password. If you leave it blank, the default is used.

1. (Optional) In the **Database** field, enter the name of the ClickHouse database that contains the table. If you leave it blank, the ClickHouse user's default database is used.

1. In the **Table** field, enter the name of the ClickHouse table to insert events into.

1. In the **Format** dropdown menu, select the insert format for events:

   - `json_each_row` (default): Inserts each event as a JSON object on its own line. Event fields are mapped to columns of the same name. This maps to ClickHouse's [`JSONEachRow`](https://clickhouse.com/docs/interfaces/formats/JSONEachRow) format.
   - `json_as_object`: Inserts each event into a single `JSON`-typed column. This maps to ClickHouse's [`JSONAsObject`](https://clickhouse.com/docs/interfaces/formats/JSONAsObject) format.
   - `json_as_string`: Inserts each event into a single `String`-typed column, storing the raw JSON. This maps to ClickHouse's [`JSONAsString`](https://clickhouse.com/docs/interfaces/formats/JSONAsString) format.
   - `arrow_stream`: Batches events using the Apache Arrow IPC streaming format. When you select this format, you must also configure Enable batching.

**Note**: When you use `arrow_stream`, your ClickHouse server must be reachable when your worker loads the pipeline configuration from the remote configuration backend, or upon worker restart. The Clickhouse table schema is not persisted between restarts.

### Optional settings{% #optional-settings %}

#### Skip unknown fields{% #skip-unknown-fields %}

In the **Skip unknown fields** dropdown menu, select either **Drop Unknown Fields** or **Fail On Unknown Fields** for fields that are not present in the target table schema instead of returning an insert error. The default is **Use ClickHouse Server**, which uses the [`input_format_skip_unknown_fields`](https://clickhouse.com/docs/operations/settings/formats#input_format_skip_unknown_fields) setting.

#### Date time best effort{% #date-time-best-effort %}

Toggle **Date time best effort** to enable flexible `DateTime` parsing on the ClickHouse server. When enabled, the server accepts a wider range of date and time string formats. See ClickHouse's [`date_time_input_format`](https://clickhouse.com/docs/operations/settings/formats#date_time_input_format) setting for more information.

#### Compression{% #compression %}

Toggle **Enable Compression** to compress outbound HTTP requests with gzip. You can optionally set the **Compression level** to an integer from `1` (fastest) to `9` (best compression). If left unset, the default is `6`.

#### Enable TLS{% #enable-tls %}

Toggle the switch to **Enable TLS**.

- If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
- The following certificate and key files are required:
  - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  - **Notes**:
    - The configuration data directory `/var/lib/observability-pipelines-worker/config/` is automatically appended to the file paths. See [Advanced Worker Configurations](https://docs.datadoghq.com/observability_pipelines/configuration/install_the_worker/advanced_worker_configurations.md) for more information.
    - The file must be readable by the `observability-pipelines-worker` group and user.

#### Enable batching{% #enable-batching %}

Batch encoding is required when **Format** is set to `arrow_stream` and is optional for JSON formats.

Toggle **Enable batching** for batch encoding.

1. Toggle **Allow nullable fields** to allow `null` values for non-nullable columns in the target table. When this setting is off (default), missing values for non-nullable columns results in encoding errors.
1. In the **Max events** field, enter the maximum number of events per batch. Must be `1` or greater.
1. In the **Timeout (secs)** field, enter the maximum time, in seconds, before a partial batch is flushed. Must be between `1` and `65535`. If left unset, the default is 1 second.

See Event batching for more information.

## Secret defaults{% #secret-defaults %}

These are the defaults used for secret identifiers and environment variables.

{% tab title="Secrets Management" %}

- ClickHouse HTTP endpoint URL identifier:
  - References the HTTP interface endpoint of your ClickHouse server.
  - The default identifier is `DESTINATION_CLICKHOUSE_ENDPOINT_URL`.
- ClickHouse username identifier (when basic auth is enabled):
  - The default identifier is `DESTINATION_CLICKHOUSE_USERNAME`.
- ClickHouse password identifier (when basic auth is enabled):
  - The default identifier is `DESTINATION_CLICKHOUSE_PASSWORD`.
- ClickHouse TLS passphrase identifier (when TLS is enabled with an encrypted private key):
  - The default identifier is `DESTINATION_CLICKHOUSE_KEY_PASS`.

{% /tab %}

{% tab title="Environment Variables" %}

- ClickHouse HTTP endpoint URL:
  - The HTTP interface endpoint your Observability Pipelines Worker sends events to. For example, `https://clickhouse.example.com:8443`.
  - The default environment variable is `DD_OP_DESTINATION_CLICKHOUSE_ENDPOINT_URL`.
- ClickHouse authentication username (when basic auth is enabled):
  - The default environment variable is `DD_OP_DESTINATION_CLICKHOUSE_USERNAME`.
- ClickHouse authentication password (when basic auth is enabled):
  - The default environment variable is `DD_OP_DESTINATION_CLICKHOUSE_PASSWORD`.
- ClickHouse TLS passphrase (when TLS is enabled with an encrypted private key):
  - The default environment variable is `DD_OP_DESTINATION_CLICKHOUSE_KEY_PASS`.

{% /tab %}

## How the destination works{% #how-the-destination-works %}

### Event batching{% #event-batching %}

A batch of events is flushed when one of these parameters is met. Batching is also configurable for this destination. See [event batching](https://docs.datadoghq.com/observability_pipelines/destinations.md#event-batching) for more information.

| Maximum Events | Maximum Bytes | Timeout (seconds) |
| -------------- | ------------- | ----------------- |
| None           | 10 MB         | 1                 |

## Further reading{% #further-reading %}

- [Observability Pipelines](https://docs.datadoghq.com/observability_pipelines.md)