OpenTelemetry

S2 implements the OTLP/HTTP specification, so it can be used as a telemetry backend for trace, metric, and log signals. Signals are routed to the appropriate streams by setting the s2.stream resource attribute. You’d typically set it when initializing your TracerProvider, MeterProvider, or LoggerProvider. For example, in Python, if you want all logs from process-a to go to stream-a, you need to configure it as follows:

import logging
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk.resources import Resource

provider = LoggerProvider(resource=Resource.create({"s2.stream": "stream-a"}))
set_logger_provider(provider)
handler = LoggingHandler(level=logging.INFO, logger_provider=provider)
logging.basicConfig(handlers=[handler], level=logging.INFO)

You can export your signals directly to a telemetry backend from your processes, but it is generally recommended to use a Collector. If you are already using a Collector, you can easily start exporting signals to S2 by using an OTLP/HTTP exporter. S2’s OTLP/HTTP endpoint supports both json and proto encodings. You need to set the Content-Type header to application/json or application/x-protobuf accordingly. Below is our recommended config for the exporter:

exporters:
  otlphttp:
    endpoint: https://<your-basin>.b.s2.dev/otlp
    encoding: <your-preferred-encoding>
    headers:
      Content-Type: <value-based-on-encoding>
      Authorization: Bearer <your-access-token>
    sending_queue:
      num_consumers: 1
      batch:
        flush_timeout: 50ms
        min_size: 262144 # 256KiB
        max_size: 524288 # 512KiB
        sizer: bytes

Note:

If you don’t have an access token already, you can generate one from the dashboard.
You need to create a basin with create-stream-on-append flag on, if you want streams to be created automatically when signals are exported.
sending_queue.num_consumers: 1 ensures batches are sent in order to S2. You can increase it if ordering is not a concern.
If you want to change sending_queue.batch config, please be aware of the limits [1, 2] and how that will impact your changes.

AI SDKs

Observability

Pipelines

Sync

OpenTelemetry