Skip to content

opampsupervisor

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

opampsupervisor

README.md

OpAMP Supervisor for the OpenTelemetry Collector

Status
Stability alpha: metrics
Issues Open issues Closed issues
Code Owners @evan-bradley, @atoulme, @tigrannajaryan, @douglascamata, @dpaasman00

This is an implementation of an OpAMP Supervisor that runs a Collector instance using configuration provided from an OpAMP server. This implementation is following a design specified here. The design is still undergoing changes, and as such this implementation may change as well.

Binary and container images for the Supervisor are available under tags starting with cmd/opampsupervisor here.

More information

If you'd like to learn more about OpAMP, see the OpAMP specification. For production-ready client and server implementations in Go, see the OpAMP Go repository.

If you have questions or would like to participate, we'd love to hear from you. There is a regular SIG meeting, see the meeting notes for more information. You can also get in touch with us in the #otel-opamp channel on the CNCF Slack workspace.

Installing the Supervisor

Binary

Binaries are available as downloadable assets from OpenTelemetry Collector releases with tags using the prefix cmd/opampsupervisor. You can download the file containing the binary for your OS and architecture, and install it on your machine manually.

Linux/macOS

Replace <VERSION> with your desired Supervisor version, and <OS> and <ARCH> with your platform-specific information:

curl --proto '=https' --tlsv1.2 -fL -o opampsupervisor \
"https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fopampsupervisor%2Fv<VERSION>/opampsupervisor_<VERSION>_<OS>_<ARCH>"
chmod +x opampsupervisor

Windows

Replace <VERSION> with your desired Supervisor version:

Invoke-WebRequest -Uri "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fopampsupervisor%2Fv<VERSION>/opampsupervisor_<VERSION>_windows_amd64.exe" -OutFile "opampsupervisor.exe"
Unblock-File -Path "opampsupervisor.exe"

Container image

Container images are available for download from GitHub Container registry:

docker pull ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-opampsupervisor

Using the Supervisor

To use the Supervisor, you will need four things:

  1. A Supervisor binary, which can be obtained through the installation step above.
  2. A Collector binary that you would like to control through the Supervisor.
  3. A Supervisor config file. See examples here.
  4. A running OpAMP server.

To test the Supervisor with an example server, download the opamp-go repository, and run the OpAMP example server in the internal/examples/server directory.

git clone [email protected]:open-telemetry/opamp-go.git
cd opamp-go/internal/examples/server
go run .

Visit localhost:4321 to verify that the server is running.

In a separate directory, create a Supervisor configuration file named supervisor.yaml. Copy one of the existing examples or create a new configuration. The agent::executable should be set to the path of your Collector binary:

agent:
  executable: <COLLECTOR_BINARY>

Start the Supervisor:

./opampsupervisor --config=supervisor.yaml

The supervisor's own log encoding is configured under telemetry.logs.encoding. Use console for plain-text logs with ISO8601 timestamps. Leave the field unset (or set it to json) to keep the default JSON output.

telemetry:
  logs:
    encoding: console

Visit localhost:4321 again to verify that your Collector appears in the Agents list.

Collector crash log snippets

The Supervisor can include the tail of the Collector's logs in health and remote config failure messages when the Collector process exits unexpectedly. This is disabled by default because Collector logs may contain sensitive data. To enable it, set agent::collector_crash_log_snippet_kib to the maximum number of log KiB to include. A value of 0 disables the feature. 4 is a reasonable starting point. The maximum value is 1024 (1 MiB).

agent:
  collector_crash_log_snippet_kib: 4

Persistent data storage

The supervisor persists some data to disk in order to mantain state between restarts. The directory where this data is stored may be specified via the supervisor configuration:

storage:
  directory: "/path/to/storage/dir"

By default, the supervisor will use /var/lib/otelcol/supervisor on posix systems, and %ProgramData%/Otelcol/Supervisor on Windows.

This directory will be created on supervisor startup if it does not exist.

Healthcheck

The Supervisor can be configured to expose a healthcheck endpoint that can be used to determine whether the Supervisor is running and healthy. This can be configured in the Supervisor configuration file:

healthcheck:
  endpoint: localhost:13133

At the moment this healtcheck checks that the following conditions are met:

  • The Supervisor is able to persist state to disk.
  • The Supervisor is able to generate the agent's configuration.

For more details on the healthcheck configuration, see the see the full list of ServerConfig options.

Note that the healthceck endpoint is not enabled by default. To enable it, you must explicitly set at least the endpoint field in the configuration.

Startup Fallback Configuration

The Supervisor supports a startup fallback configuration mechanism that provides resilience when the OpAMP server is unreachable at startup and there's no previous configuration state persisted in disk. This is useful for ensuring the Collector can start with a known-good configuration during network outages or server maintenance. When the Supervisor successfully connects to the OpAMP server, the regular configuration (indicated by agent::config_files) is restored and any potential remote configuration received from the OpAMP server is applied.

If, when the Supervisor starts, there's previous configuration state persisted in disk, the Supervisor will use it to start the Collector.

Configuration Options

The following options can be configured under the agent section:

agent:
  startup_fallback_configs:
  - /path/to/fallback_config.yaml
Option Description
startup_fallback_configs List of paths to startup fallback configuration files to use. If more than one path is specified, they are merged in order. Together, these must form a complete, standalone Collector configuration.

Important Notes

  • The startup fallback configuration is a set of standalone configuration files. It is intentionally not merged with the agent::config_files setting. This ensures predictable behavior without dependencies on other configuration files.

  • The OpAMP extension and "own telemetry" configuration are automatically added to the fallback configuration. This maintains the Collector's own telemetry flowing and the communication between Supervisor and Collector.

  • The Supervisor will continue attempting to reconnect to the OpAMP server even while using the fallback configuration.

Example

See examples/supervisor_fallback.yaml for a complete example configuration with fallback enabled.

Automatic Remote Configuration Rollback

The Supervisor supports automatic rollback when a remote configuration received from the OpAMP backend causes the Collector to fail to start correctly. This feature is disabled by default and can be enabled in the Supervisor configuration file:

agent:
  automatic_config_rollback: true

When enabled, the Supervisor will watch the health of the Collector after applying remote configurations received from the OpAMP backend and it will cache the last working one in disk, besides the last received. If a remote configurations causes the Collector to fail to start correctly, the Supervisor will automatically swap to the last working one and restart the Collector. During this process, a new "APPLIED" status will be reported for the last working configuration.

Status

The OpenTelemetry OpAMP Supervisor is intended to be the reference implementation of an OpAMP Supervisor, and as such will support all OpAMP capabilities. Additionally, it follows a design document for the features it intends to support.

For a list of open issues related to the Supervisor, see these issues.

Key:

✅: Fully implemented

⚠️: Implemented with caveats

OpAMP capabilities

OpAMP capability Status
AcceptsRemoteConfig ✅
ReportsEffectiveConfig ✅
AcceptsPackages #47272
ReportsPackageStatuses #47272
ReportsOwnTraces ✅
ReportsOwnMetrics ✅
ReportsOwnLogs ✅
AcceptsOpAMPConnectionSettings ✅
AcceptsOtherConnectionSettings ✅
AcceptsRestartCommand ✅
ReportsHealth ⚠️
ReportsStatus #38729
ReportsRemoteConfig ✅
ReportsAvailableComponents ✅

Supervisor specification features

Feature Status
Offers Supervisor configuration including configuring capabilities ✅
Starts and stops a Collector using remote configuration ✅
Communicates with OpAMP extension running in the Collector ✅
Updates the Collector binary #33947
Configures the Collector to report it's own metrics over OTLP ✅
Configures the Collector to report it's own logs over OTLP ✅
Sanitization or restriction of Collector config #24310