DuckLake Exporter

Status
Stability	alpha: traces, logs
Distributions	contrib
Issues

This exporter store OpenTelemetry logs and traces to DuckLake, a lakehouse storage format uses parquet for storage and a transactional database for metadata. This implementation currently supports local file system and s3 for writing parquet, and Postgres for the metadata catalog.

Configuration

PostgreSQL Settings

Name	Description	Default
postgresql.dsn	Full PostgreSQL DSN. Takes precedence over individual fields if provided.
postgresql.host	PostgreSQL server hostname.	localhost
postgresql.port	PostgreSQL server port.	5432
postgresql.database	PostgreSQL database name.	ducklake
postgresql.username	PostgreSQL username.
postgresql.password	PostgreSQL password.
postgresql.ssl_mode	SSL mode: disable, require, verify-ca, verify-full.	disable
postgresql.schema	PostgreSQL schema for metadata tables.	public
postgresql.max_open_conns	Maximum number of open connections.	25
postgresql.max_idle_conns	Maximum number of idle connections.	5
postgresql.conn_max_lifetime	Maximum connection lifetime.	5m

Parquet Settings

Name	Description	Default
parquet.batch_size	Number of records to accumulate before writing a parquet file.	10000
parquet.batch_timeout	Maximum time to wait before writing a parquet file.	10s
parquet.compression	Compression codec: SNAPPY, GZIP, ZSTD, UNCOMPRESSED.	SNAPPY
parquet.enable_bloom_filters	Enable bloom filters for trace_id and span_id columns.	true

DuckLake Settings

Name	Description	Default
ducklake.output_path	Directory or S3 URI where parquet files are written (e.g., /var/lib/ducklake or s3://bucket/prefix).	/var/lib/ducklake/data
ducklake.data_path	Actual filesystem path to parquet files for metadata (defaults to output_path for local paths, ignored for S3).
ducklake.logs_table_name	Table name for logs in metadata catalog.	otel_logs
ducklake.traces_table_name	Table name for traces in metadata catalog.	otel_traces
ducklake.s3_region	AWS region for S3 (only when output_path is an S3 URI).	us-east-1
ducklake.s3_endpoint	S3 endpoint URL for S3-compatible services (optional).
ducklake.s3_force_path_style	Force path-style addressing for S3-compatible services.	false
ducklake.s3_role_arn	AWS IAM role ARN to assume for S3 access (optional).

Standard Exporter Settings

Name	Description	Default
timeout	Request timeout.	5s
retry_on_failure	Configuration for retrying failed exports. See exporterhelper.
sending_queue	Configuration for the sending queue. See exporterhelper.	disabled

Example Configuration

Local Storage

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  ducklake:
    postgresql:
      host: localhost
      port: 5432
      database: ducklake
      username: otel
      password: ${env:POSTGRES_PASSWORD}
      ssl_mode: disable
    parquet:
      batch_size: 10000
      batch_timeout: 5m
      compression: SNAPPY
      enable_bloom_filters: true
    ducklake:
      output_path: /var/lib/ducklake/data   # the path within the collector container
      data_path: /host/path   # the path to be recorded in the ducklake metastore, used by the ducklake reader. 
      logs_table_name: otel_logs
      traces_table_name: otel_traces
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 5m
    sending_queue:
      num_consumers: 1


service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [ducklake]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [ducklake]

S3 Storage

exporters:
  ducklake:
    postgresql:
      host: postgres.example.com
      database: ducklake
      username: otel
      password: ${env:POSTGRES_PASSWORD}
    parquet:
      batch_size: 50000
      batch_timeout: 1m
      compression: ZSTD
    ducklake:
      output_path: s3://my-telemetry-bucket/otel-data
      s3_region: us-west-2
      logs_table_name: otel_logs
      traces_table_name: otel_traces
    sending_queue:
      num_consumers: 1

Minimal Configuration

exporters:
  ducklake:
    postgresql:
      username: otel
      password: ${env:POSTGRES_PASSWORD}

This uses all defaults: local storage at /var/lib/ducklake/data, PostgreSQL on localhost:5432, batch size of 10000 records.

Parquet Schema

Logs

Column	Type	Description
timestamp	TIMESTAMP	Log record timestamp
trace_id	VARCHAR	Trace ID (with bloom filter)
span_id	VARCHAR	Span ID (with bloom filter)
severity_number	INT32	Numeric severity level
severity_text	VARCHAR	Text severity level
body	VARCHAR	Log message body
scope_name	VARCHAR	Instrumentation scope name
scope_version	VARCHAR	Instrumentation scope version
resource_attributes	JSON	Resource attributes
log_attributes	JSON	Log attributes
flags	UINT32	Trace flags
observed_timestamp	TIMESTAMP	Observed timestamp

Traces

Column	Type	Description
trace_id	VARCHAR	Trace ID (with bloom filter)
span_id	VARCHAR	Span ID (with bloom filter)
parent_span_id	VARCHAR	Parent span ID
name	VARCHAR	Span name
kind	INT32	Span kind
start_time	TIMESTAMP	Span start time
end_time	TIMESTAMP	Span end time
duration_ns	INT64	Duration in nanoseconds
status_code	INT32	Status code
status_message	VARCHAR	Status message
scope_name	VARCHAR	Instrumentation scope name
scope_version	VARCHAR	Instrumentation scope version
resource_attributes	JSON	Resource attributes
span_attributes	JSON	Span attributes
events	JSON	Span events array
links	JSON	Span links array
dropped_attributes_count	UINT32	Dropped attributes count
dropped_events_count	UINT32	Dropped events count
dropped_links_count	UINT32	Dropped links count
trace_state	VARCHAR	W3C trace state
flags	UINT32	Trace flags

Querying Data

Query with DuckDB using DuckLake

First, attach the DuckLake database to DuckDB:

ATTACH 'ducklake:postgres:host=localhost dbname=ducklake user=otel password=yourpassword' AS otel;

Then query logs and traces through the DuckLake metadata catalog:

-- Recent error logs
SELECT timestamp, severity_text, body, trace_id
FROM otel.main.otel_logs
WHERE severity_text IN ('ERROR', 'FATAL')
  AND timestamp > NOW() - INTERVAL '1 hour'
ORDER BY timestamp DESC
LIMIT 100;

-- Logs for a specific trace
SELECT timestamp, severity_text, body
FROM otel.main.otel_logs
WHERE trace_id = '4bf92f3577b34da6a3ce929d0e0e4736'
ORDER BY timestamp;

-- Slow traces (>1 second)
SELECT trace_id, name, duration_ns / 1e9 AS duration_sec, start_time
FROM otel.main.otel_traces
WHERE duration_ns > 1000000000
  AND start_time > NOW() - INTERVAL '1 hour'
ORDER BY duration_ns DESC
LIMIT 100;

-- Find spans by service name
SELECT trace_id, span_id, name, duration_ns
FROM otel.main.otel_traces
WHERE json_extract(resource_attributes, '$.service.name') = 'my-service'
ORDER BY start_time DESC;

Direct Parquet Access (without DuckLake)

You can also query parquet files directly without the DuckLake extension:

-- Query logs directly
SELECT timestamp, severity_text, body
FROM read_parquet('/var/lib/ducklake/data/otel_logs_*.parquet')
WHERE timestamp > NOW() - INTERVAL '1 hour'
LIMIT 100;

-- Query traces directly
SELECT trace_id, name, start_time
FROM read_parquet('/var/lib/ducklake/data/otel_traces_*.parquet')
WHERE start_time > NOW() - INTERVAL '1 hour'
LIMIT 100;

Query Metadata with PostgreSQL

-- List recent snapshots
SELECT s.snapshot_id, s.created_at, t.table_name,
       ts.record_count, ts.total_size
FROM ducklake_snapshot s
JOIN ducklake_table t USING (table_id)
JOIN ducklake_table_stats ts USING (snapshot_id)
ORDER BY s.created_at DESC
LIMIT 10;

-- Get parquet files for a snapshot
SELECT file_path, record_count, file_size_bytes
FROM ducklake_data_file
WHERE snapshot_id = 123
ORDER BY created_at;

Performance Tuning

Batch Size and Timeout

Adjust based on your throughput, for example:

High volume (>1000 records/sec):

parquet:
  batch_size: 50000
  batch_timeout: 1m

Low volume (<100 records/sec):

parquet:
  batch_size: 1000
  batch_timeout: 10m

Compression

• SNAPPY (default): Balanced compression ratio and speed
• ZSTD: Better compression, slower writes
• UNCOMPRESSED: Fastest writes, largest files

PostgreSQL Connection Pooling

For high-throughput scenarios:

postgresql:
  max_open_conns: 10
  max_idle_conns: 5
  conn_max_lifetime: 10m

References

• DuckLake Specification
• OpenTelemetry Semantic Conventions
• Apache Parquet Format