STEF Exporter

Status
Stability	alpha: metrics
Distributions	contrib
Issues
Code coverage
Code Owners	@tigrannajaryan, @dmitryax

Overview

Exports data via gRPC using Otel/STEF format. Otel/STEF is a compact and fast telemetry format. It is currently the fastest benchmarked metric format in the Collector. Here are recent benchmarking results, comparing CPU usage when using a few different formats (lower is better):

Test	CPU Avg%	CPU Max%
Metric10kDPS/Carbon	174.9	179.3
Metric10kDPS/OpenCensus	59.2	61.3
Metric10kDPS/OTLP	44.8	45.7
Metric10kDPS/OTLP-HTTP	33.5	34.0
Metric10kDPS/SignalFx	78.9	88.0
Metric10kDPS/STEF	20.7	21.7

STEF in this benchmark outperforms all other formats, including OTLP, which was previously known as the fastest format in the Collector.

STEF is also very compact on the wire. In uncompressed mode, it typically yields payloads that are more than 10x smaller than OTLP payloads. In compressed mode, STEF is typically 5-7 times more compact than OTLP.

Here are some STEF benchmark results, comparing STEF and OTLP payload sizes for a few sample payloads produced by Collector:

And here is a dataset from Astronomy Shop demo:

(This exporter implementation uses unsorted STEF format, labeled "STEF Unsorted" in charts above).

There are currently no known formats that match STEF's compactness and performance.

STEF exporter can be used to send metrics from the Collector to any STEF-compatible backend. STEF can also be used to send metric data between Collector instances, in which case on the receiving side a STEF receiver should be used.

STEF format, STEF exporter and receiver implementations are currently in alpha stage of development, during which the format may undergo breaking changes. To ensure interoperability between sending and receiving Collectors make sure you are using versions of exporter and receiver that are compiled with the same version of STEF library.

Feedback about STEF format and implementation is welcome in the form of issues in this repository or in STEF repository.

Configuration

The following settings are required:

• endpoint (no default): host:port to which the exporter is going to send STEF metric data, using the STEF/gRPC protocol. The valid syntax is described here. When sending to another Collector you will typically use <hostname>:4320 as the endpoint, since port 4320 is the default port used by STEF receiver. If a scheme of https is used then client transport security is enabled and overrides the insecure setting.
• tls: see TLS Configuration Settings for the full set of available options.

Example:

exporters:
  stef:
    endpoint: otelcol2:4317
    tls:
      cert_file: file.cert
      key_file: file.key
  stef/2:
    endpoint: otelcol2:4317
    tls:
      insecure: true

By default, no compression is enabled. The only supported compression method is zstd. To enable compression, configure as follows:

exporters:
  stef:
    ...
    compression: zstd

Advanced Settings

STEF exporter supports the following advanced settings:

• gRPC settings
• TLS and mTLS settings
• Queuing, timeout and retry settings, particularly:
  • The timeout setting controls how long the exporter waits for ACK of a data sent over STEF/gRPC stream. Increase this value if you see timeouts in the logs.
  • The num_consumers setting defines how many unacknowledged batches can be in-flight. If the destination is slow to acknowledge then increasing this number can help increase the throughput.
  • For the rest of settings see here.