I would love to see alternatives here. We've discussed things like dynamically-reloadable or merge rules for declarative config, and it would help reinforce why we need a new concept to solve the problems you are interested in.

Agreed, I need to write down the treatment of why dynamically reloaded config doesn't solve the problems that motivate the proposal.

My answer to your other comment, hopefully, hints at that, but it'll be a longer write-up.

Could we think of this as new "subtypes" of declarative config that can be used in a standalone way? E.g. if we think of the current declarative config as configuration as type "SDK", we could define sub-types like "sampler", "view", or "log-record-processor"?

If we can, I would love to keep the same yaml structure / definitions for these policies that we currently have in the declarative config so we avoid introducing another structured definition of what a "sampler" is. Or do you think because this is targeted at the collector as well that isn't feasible?

I'd expect the declarative config for a policy-component to be used directly in declarative config:

So something like:

- my_policy_component:
  - default_policies
    - type: xyz
      ... the policy yaml...

The primary difference between the policy for sampling and a "sampler" will actually be in flexibility. A sampler component could be written in any language, allow any code and its configuration must be open. A sampler policy MUST have a well-defined behavior, have the same configuration and behavior in all languages or implementations.

So primarily, a policy is highly limited in a way extension points are not.

Dynamic control of SDKs is something that should be able to be built on top of or as an evolution of declarative config. I / we have been conscious of this eventually while building declarative config and I don't think anything will get in the way. Also, I hope that minimally, the declarative config data model can be used as a way for servers to communicate the desired configuration state of components in a dynamic config scenario.

While I agree to a degree, the type of control and abstraction these proposal seeks to enable is NOT possible without agreement on semantics and use-cases across diverse implementations.

E.g. the declarative config + OpAMP could be used to send any config to any component. What it doesn't do, and what we need to sort out, is how to understand what config can be sent to what component, and how to drive control / policy independent of implementation or pipeline set-up, e.g.

Imagine a world where we can control the reporting of metrics across open telemetry SDKs, custom implementations and Prometheus SDKs because we agreed to the semantics of policy independent of configuration.

• In Declarative config I'd expect things that cannot be shared between different implementations:
  • Queue/Buffer sizes that are specific to my pipeline setup
  • Threading / GC configuration specific to my language
• In a Policy we should be limited to ONLY things that can be shared broadly, across
  implementations and have well-defined semantics for how to enforce them.

So I see Declarative config as encompassing more than just policies, where policies would be a subset of what you'd find. Additionally, Policies can be independent things that you can bundle together. I should be able to "add" a policy at any point without needing to understand how it interacts with other components. AN example of this - If I have a configuration reporting metrics, that configuration would have a MetricReader->MetricExporter right? What If there's multiple. How would I know what to change generically, if I just wanted to say "stop producing metric X". Policies are ignorant of this. They just push a policy down and the SDK would be expected to enforce this via a PolicyMetricReader that's configured to pay attention to a metric filter policy.

Apologies not all of this is fleshed out, as it's a working draft, and one we're working on in the repo. Please continue to ask questions and I'll use that to flesh out the motivation more.

This is an exact case I have, turning off metrics. And turning them back on. I implement this by having a flag in a custom exporter which stops/restarts exports. A generic solution to turning it off would be to change the exporter config to none, then I guess you could re-enable by setting again to otlp, but that implies a much more complex action in the SDK rather than switching a boolean on/off

I added this to the alternatives considered discussion

Imagine a world where we can control the reporting of metrics across open telemetry SDKs, custom implementations and Prometheus SDKs

A bold vision. I think I was definitely misunderstanding the scope. I'll revise my position: If we want dynamic control solutions specifically for otel SDKs, the declarative config data model should play a role, because not using it means introducing yet another config interface (YACI 😛). With a broader scope targeting other tools besides otel SDKs, we would of course need something not loaded with otel SDK vocabulary / baggage.

Should this type of thing even live in otel or in some neutral territory? (reminds me of the relationship between w3c trace content and opentelemetry) Are there other ecosystems that have expressed interest in or that we've reached out to for collaborating?

Great quesitons!

If we want dynamic control solutions specifically for otel SDKs, the declarative config data model should play a role

100%!

Should this type of thing even live in otel or in some neutral territory?

Great question. I personally think this belongs in OTEL and should "feel native" to otel, but allow any component in o11y space to interact with it. This can increase the reach of "effective opentelemetry" as components which support writing OTLP can also participate with policies. However, to your question above, if this wasn't first-class in otel, how would we make sure our declarative config data model plays an important role?

Are there other ecosystems that have expressed interest in or that we've reached out to for collaborating?

The idea is the outcome of discussions with both Envoy (and their xDS control plane folks) and Google's Monarch team (see #4672). I would love to pull in more folks to collaborate for sure. First, I want to make sure we all understand the vision, scope and goals.

This PR was meant to be a place for those of us who started discussing to flesh out the proposal in place (as draft), so this PR is meant to be collecting that interest and refining the message. APologies it was rough when you first reviewed it.

This reads like a subset of declarative configuration capabilities. Wouldn't it be easier to unify on one data model (i.e. declarative config) for expressing the desired configuration, and build tooling to detect / apply diffs when a a change is pushed from a remote server?

I.e. an app starts with:

file_format: 1.0
tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_http:
    sampler:
      parent_based:
        root:
          trace_based:
            ratio: 1.0

Later, a remote server pushes a new configuration state with an updated ratio for the trace id ratio sampler:

file_format: 1.0
tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_http:
    sampler:
      parent_based:
        root:
          trace_based:
            ratio: 0.5 # reduce ratio from 1.0 to 0.5

Some controller is responsible for evaluating the diff between the current state and the desired state, and computing / executing update steps as allowed. In this case, substitute the sampler.

You can read some of my rationale at the bottom of the OTEP.

Effectively:

• I think policies will be used as-in in declarative config. There would be a component in Declarative config that can be configured with a default set of policies.
• I think policies will be highly limited in expected behavior / security profile vs. declarative config.
  • I think SDK configuration will opt-in to allow remote-policy control, with explicit permissions per-policy
  • I do not think policies will alter pipeline setup or shape. Policies should have well defined insertion points already defined via Config where they will be enforced.
• Policies will need to have a mechanism via OpAMP to advertise they can be accepted and handled - we can use "custom capabilities" for this.

So there are a lot of similarities, but the key difference is the limitations.

Imagine you are an enterprise running a large fleet of SDKs. You want or you only can control this fleet with remote configuration. It would be great to not be limited to the mentioned policy types. I want also to be able to control what instrumentations are enabled, maybe I need to pause sending of signals during an incident, control the SDKs internal logging level etc.

@hegerchr I do expect we could provide more policies than the ones listed. My main point is why configuration, as it stands, doesn't solve the goals of this proposal.

Regarding managing a fleet of SDKs / Controllers - That's where this policy proposal originates - controlling Google's collection fleet and lessons we've learned in the process. @menderico has talk at KubeCon this year that will cover more details if you're curious what we're looking for.

You want to filter metrics using a filtering reader (this component doesn't exist in the SDK spec and so would have to be custom) vs. views or meter config?

I'm not sure, I can update this to use views instead as well. I was taking from the proposed OTEP where you can control both the reporting of a metric and the report interval (i.e. periodic metric reader would need configuration for how often to report each set of metrics).

If this is a popular use case we should extend the SDK spec to add an additional built in component. We're too reluctant to do this right now.

That still won't tell me if it's safe to send configuration to an SDK or not. I need to know, at runtime, that the version of the SDK I'm trying to control will support that config or if I'll crash a key component.

Additionally, it doesn't help me ignore the implementation detail. E.g. what If I also want to control Prometheus client library? We don't own their config or their specification. However, we could build something that interacts with remote policies, similar to Jaeger-Remote-Sampler of today for traces.

Can the SDK be self-protecting? When the SDK receives a configuration that it does not fully support it can reject the configuration and send an error other information about the rejection back using OpAMP.

Can the SDK be self-protecting? When the SDK receives a configuration that it does not fully support it can reject the configuration and send an error other information about the rejection back using OpAMP.

That still doesn't answer my second question - Do I need to translate configuration to both OTEL + Prometheus if I want to control a prometheus client library and otel SDK ?

What if all I care about is to stop reporting metric X and I don't care what impelmentation is generating it? Having some minor flake/discrepency in the config I push to an OpAMP server lead to rejection of config in HALF of my SDK clients would be problematic. Having to push multiple config files (and understand if i've broken the entire pipeline by accident, i ways the SDK cannot reject) is risky.

I would forsee 2 SDK components:

• Policy Provider: constructs the identity of the agent and contains a collection of policy detectors. It exposes methods to access the collection of policies provided by the detectors and notify a component of an updated policy/profile.
• Policy Agent: provides a detect method enabling components to report it's policy. Designed to be embedded in components.

Why don't we define a policy as follows:

• Policy Definitions: an array of policy definitions
• Instrumentation scope: identifies the component who this policy is for and mirrors otlp definition.

The policy definition would contain the type & schema property as above.

Having the scope allows for restriction of who the policy applies to. information about the agent/resource is left out as opamp natively provide this info.

Could this be simplified to instead use an applyPolicy method on the PolicyAgent. The policyprovider can use the the data (scope & policy type) from the detect method to only invoke apply on the correct audience

Opamp agent/client should be able to report back supported policies to opamp server.

Opamp server should be able to inform client when a policy is updated including the scope it applies to.

@andrzej-stencel i want to continue the OTTL discussion in a thread here.

Looking at the example policy types mentioned, they all seem to be about the transformation of telemetry. Conceptually, I'd love to see OTTL serving as the universal language for describing telemetry transformations. Caveats:
Likely not all transformations are currently supported aka. implemented in OTTL; some might never (trace sampling?)
Perhaps there are other policy types that are not telemetry transformations, just not mentioned in the examples?

IMO, we shouldn't tie ourselves to OTTL just yet for a few big reasons:

• No formal grammar (not very portable, limits implementations)
• Not versioned, no guarantees about breaking changes

Currently, none of the example policy types have to do with transformation (save the redaction policy), and most are focused on filtration or minor configuration. I have a POC for the policy proto, and one of the biggest things for this project to be successful is that policies can be run efficiently based on matching conditions (i.e. run policy X when Y condition is met). For that matching logic, I think it makes sense to be based on semantic convention which is a well-defined standard at this point. I think for now, given we're proposing the overarching Policy model, I don't want to get too caught up in a single policy type and rather focus on the concept as a whole. You could imagine in the future as we design specific policy types, one would be of type ottl allowing an implementation (such as a collector processor) to run ottl based on its loaded policies. I think that should be okay, but let me know if any part of that doesn't track. Thank you 🙇

You raise very good points about OTTL's lack of grammar and stability guarantees. Also limiting policy types to OTTL is probably too restrictive - this was my suspicion too. Thank you for bringing these arguments.

I would really love to see things are getting more dynamic in the SDKs to be able to have more possibilities for remote configuration support.

Does a policy has a target in the sense of a service name (thinking in the context of SDKs) that denotes the policy applies to only that service?

im still working through the actual message for the policy and it being service specific was one design. The one i'm going through has a list of matchers where a matcher can function on the resource, scope, attrs, or telemetry specific data. This should be flexible enough to allow a user to match on just a service.name or a log body or a trace scope.

I think we should add a recommendation that if possible overrides should be supported, and in the order opamp > http > file, ie a change specified by an opamp provider would override other provider changes - including if the other providers have subsequent updates. If we don't specify this detail, we're going to have free-for-all inconsistent implementations about overrides.

This follows the natural (and I think most existing implementations for non-otel agent management) that central config overrides all other changes

yup this makes sense to me, I've been working through the same thought/issue, and would be good to mandate this.

Here's a suggestion

Overrides SHOULD be supported. A policy should retain the provider source as one of OPAMP(1), HTTP(2), FILE(3), CUSTOM(?) (priority in brackets; custom priority is specified by the implementer; there can be multiple custom providers, 1 is higher priority than 2, etc). To be implemented, any new policy field has to have priority at least as high as the already implemented policy field otherwise it is dropped. Where fields cannot be merged consistently while still applying the priority override , the lower priority policy is dropped in entirety

Can you give an example, I don't get id vs name

id is a global alphanumeric stable identifier, name is more like metadata for the user.

id: abd123noifoiqwnfl
name: Filter healthcheck spans

So name could be used as ID. I don't really see the need for an extra field

This section assumes perfect transport order, or at least doesn't make it clear to me what happens with out of order updates.
For example policy "export-traces: on" is sent, but get's delayed. "export-traces: off" is sent and received, the agent turns off export of traces. Then the delayed "export-traces: on" gets pushed through. I have 2 problems now

1. The order of application is clearly wrong, the intention is to have no exporting but the last policy turns it on
2. Even if I wanted to combine these, it's impossible to come up with a commutative reduction for these two operations

I would favour a naive last timestamp approach (since we've got the prioritization, it's reasonable to assume that a given policy source has a specific consistent timestamper)

So, at high level we did not want to constraint it to timestamp only since there are operations that can be resolved without it, but it is fair to add an example with it as well. It is up to the specific policy to decide how to resolve conflicts.

I'll add one ASAP.

Can you please clarify why should the server parse configuration? I think the server is supposed to generate configuration and send to the Collector, not parse it.

Let's imagine we try to create a UI to control collectors and configure them. Don't I need to understand the commands I'm sending to collectors, or do you envision this always being "blob of bytes"?

Let's then imagine I try to create some kind of abstraction around controlling collectors, where I want to support more than just specifically an exact version of one otel collector - how does that work?

I think both approaches are possible. I can envision a system where the UI gives you an edit box where the user is free to input a config and then that config is sent via OpAMP to the agent without the server attempting to interpret it in any way. What you describe is also possible, i.e. a UI that is more knowledgable about specifics of the config and give you richer controls to compose the config.

I now understand what you are saying but reading about "parsing" was confusing. Specifically, the rich UI functionality you describe doesn't necessarily need to parse a config. It does need to know the schema of the config and be able to generate valid config blobs that match that schema. Parsing may be part of the picture for example to visualize the effective config but that seems to be a topic that is outside of this OTEP's scope.

A config pushed through OpAMP can be applied dynamically if the recipient (the Collector or SDK) chooses to apply it dynamically. What inherently in OpAMP or file-based config prevents the dynamic behavior?

I can implement the OpAMP spec + Configuration spec without the requirement to support dynamic adaptation, right?

That's what this is saying. There's nothing required in OpAMP to allow this to work, we need a component that requires dynamic behaivor.

There's nuances to this point as well, but the basic gist is - We aren't requiring dynamic loading, so we'd need to change our stance or create something that DOES require it.

OK, that makes sense. I think the part that says "cannot" confused me.

i updated the language based on this conversation

Can you please clarify idempotency on a different example. For example if the policy is "multiply all numeric gauge values by 100" (a policy that converts from fractions to percentages), can I expect it to do the multiplication only once if the policy is applied both in the SDK and Collector?

If you wanted to enforce such a policy, YES - you'd need a way to track it was performed in the SDK so the Collector would not also do the work.

Policies are not generic transformations though, so I'm not sure we'd have a "multiply all gauge values by 100", but we MAY have a "convert milliseconds to seconds" where you can determine, via the contents of OTLP, if the policy was already enforced.

What I want to primarily understand is whether we are implying a communication mechanism that helps ensure idempotency (e.g. signal that the policy was already applied and should not be applied again) or we are limiting policies to only be designed in such a way that without such communication mechanism idempotency is still ensured (just like it is in the example you brought where you only apply the transform if unit=="milliseconds").

The OTEP does not answer this question but I think it is an important one since it defines the design space.