In deps.go, the sink and exporter are now initialized with a HCP specific "TelemetryConfigProvider" which continuously fetches configuration from HCP and updates it as needed.

Moved to client.go

Moved to client_test.go

The OTELExporter now holds an EndpointProvider from which it obtains the endpoint where it needs to send metrics to.

The endpointProvider can be any struct that conforms to the interface. The OTELExporter uses the GetEndpoint() of an endpointProvider to obtain the url.

The OTELSink now holds an ConfigProvider from which it obtains the filters and labels to process metrics.

The ConfigProvider can be any struct that conforms to the interface. The OTELSink uses the GetFilters() and GetLabels() of an ConfigProvider.

Don't need labels / filter logic in here since the provider will give this information.

We now use a resource without attributes (labels). Once created, a resource can't change its attributes, so this won't work anymore.

To support dynamic configuration, i.e. changing labels, we will now send custom labels at the individual metric level.

CC @jjti : I'm assuming there is no impact on the prom side for this, but just tagging you for some 👀

Okay yeah, understood. Payload size will increase, but there shouldn't be any backend change because of this since we wind up adding the attributes to each sample regardless during the OTLP > Prom conversion

Sounds good, let me cut a ticket to look into this and gather info. If it does become a problem, I think we should fast track the compression plan that we had previously, which would mean:

1. Consul/collector should compress export payloads
2. TGW should be able to handle compression (think it currently doesn't?)

TGW should be able to handle compression (think it currently doesn't?)

Correct that it currently doesn't. There's a trade-off where we'll up our CPU/mem usage in Consul + CTGW but for these repetitive sample labels I imagine it's worth it. When we address this it'd be nice to see how the increased cost of compression helps us w/ payload size.

We use snappy encoding already to compress payloads in the gateway, and I see that's a dependency of Consul already, that may be one route (adding Content-Encoding: snappy)

Change from http to grpc would result in a better way of handling payload size as well. We should probably backlog this as the HIP client already uses grpc over traefik and we should probably do the same to reduce the egress charges for customers and for our own hcp clusters.

We need to handle adding the provider labels at the metric level, in addition to the go metrics label. This function merges both into OTEL attribute key value pairs.

Changes:

• Init client with context to add a logger
• Refactored out the TelemetryConfig, MetricsConfig, new RefreshConfigand the conversions into the telemetry_config.go file

Why this refactor?

1. This file was growing larger and larger
2. Modified the endpoint, label and filters validation and parsing to be done when converting the response. This reduces the code over in deps.go and telemetry_config_provider.go. It also avoids code duplication.

Moved to new file telemetry_config.go

Moved conversions to new file telemetry_config.go

Removed most tests in this file to put them in telemetry_config_test.go
Fixed the actual client test which was not mocking the TGW client properly, and test the FetchTelemetryConfig function including validations and conversion.

1. This new file contains all logic pertaining to TelemetryConfig, an object that contains telemetry config from HCP.

I settled on the following model for the object:

type TelemetryConfig struct {
 	MetricsConfig *MetricsConfig
 	RefreshConfig *RefreshConfig
 }

In the future, we can add fields here like TraceConfig *TraceConfig or LogsConfig *LogsConfig

I removed the upper level label and endpoint, instead we should perform this conversion and pass around the object as it should be used.

This greatly reduces the complexity of code in deps.go and telemetry_config.go, and removes any duplicated code.

2. This file contains all conversions that are needed to get a final TelemetryObject, in particular, the default labels setting, the conversion of a string endpoint to a url, etc.

Changes:

1. Init the TelemetryConfigProvider
2. Remove any parsing that is now unnecessary with the client refactor (url parsing)

Now returns an error to test those properly.

Some of the tests were cleaned up thanks to the refactor of the URL parsing into the client.
New tests were added for the new TelemetryConfigProvider

The OTELExporter now holds an EndpointProvider from which it obtains the filters and labels to process metrics.

The ConfigProvider can be any struct that conforms to the interface. The EndpointProvider uses the GetEndpoint() of an EndpointProvider.

New TelemetryConfigProvider which holds dynamic configuration and provides labels, filters and the endpoint to consumers.

pass in the ctx and refreshInterval to avoid needing a rw lock call here.

All the Get* methods acquire a reader lock and defer the unlock before returning the config field

Only modify config if it has changed. This is verified by comparing the hash of the old vs new config

New version to bring in the RefreshConfig changes