Skip to content

Add Metrics Annotations#4260

Closed
fstab wants to merge 4 commits intoopen-telemetry:mainfrom
fstab:metric-annotations
Closed

Add Metrics Annotations#4260
fstab wants to merge 4 commits intoopen-telemetry:mainfrom
fstab:metric-annotations

Conversation

@fstab
Copy link
Copy Markdown
Member

@fstab fstab commented Mar 12, 2022

Add annotations @Timed and @Counted for automatically instrumenting method invocations.

The annotations are similar to the annotations offered by Micrometer, MicroProfile metrics, and Dropwizard metrics.

The default attributes class, method, and exception are similar to the default attributes provided by Micrometer's annotations.

Here's an example of what a @Counted metric would look like in Prometheus format:

method_invocations_total{class="com.example.demo.Example", method="example", exception="None"} 1.0

This PR is a follow-up to the discussion in open-telemetry/opentelemetry-java-instrumentation#7030.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 12, 2022

Codecov Report

Merging #4260 (eaa90a3) into main (5f32636) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    open-telemetry/opentelemetry-java#4260   +/-   ##
=========================================
  Coverage     90.04%   90.04%           
  Complexity     5057     5057           
=========================================
  Files           580      580           
  Lines         15573    15573           
  Branches       1497     1497           
=========================================
  Hits          14022    14022           
  Misses         1090     1090           
  Partials        461      461           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f32636...eaa90a3. Read the comment docs.

*/
@Target(ElementType.PARAMETER)
@Retention(RetentionPolicy.RUNTIME)
public @interface TimerAttribute {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really need a separate annotation for each type of instrument attribute? Could this just be "MetricAttribute" and apply to both? Or do you need to separate the attributes for the counter and the attributes for a timer, if you happen to have both of them? I would argue, since timers generate histograms, and histograms implicitly have counts of the total invocations in them, that you wouldn't want to or need to annotation a method with both, unless the use-case is extremely unusual.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we perhaps reuse SpanAttribute somehow? Or deprecate it, and introduce a new annotation that would apply to both signals?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkwatson true, it does not make sense to use both @Timed and @Counted since timers already include the count. I'm happy to merge TimerAttribute and CounterAttribute into a single MetricAttribute.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed an update replacing TimerAttribute and CounterAttribute with a single MetricAttribute.

*/
@Target(ElementType.PARAMETER)
@Retention(RetentionPolicy.RUNTIME)
public @interface TimerAttribute {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we perhaps reuse SpanAttribute somehow? Or deprecate it, and introduce a new annotation that would apply to both signals?

*
* <p>By default, the instrument will not have an attribute with the return value.
*/
String returnValueAttribute() default "";
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple days ago somebody asked for the same thing for spans: open-telemetry/opentelemetry-java-instrumentation#5527
Instead of an attribute, perhaps this could be a separate annotation? That's used by both signals (traces and metrics)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be an option to use an attribute for a span but not for a metric. For metrics, each set of attributes creates a new time series, and users must be careful to use only attributes that have a limited number of possible different values. For spans that is not an issue. So it might make sense to use arbitrary parameters, return values, and exceptions as Span attributes but not use them as metric attributes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess if we want to have an option to use the return value as a span attribute but not as a metric attribute we don't need an extra annotation, because then we cannot reuse it for spans and metrics anyway. In that case we could just leave the returnValueAttribute parameter.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @mateuszrzeszutek was suggesting something like this:

  @Timed
  @ReturnAttribute("process.result")
  public int process() {
    ...
  }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, exactly that -- and this annotation could be used to fill in attributes both in spans and metrics.

Copy link
Copy Markdown
Member Author

@fstab fstab Mar 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure whether we should have common attributes for both spans and metrics, or whether we should separate them. The difference is that with metrics each new attribute value creates a new time series. Therefore you only want to choose attributes that have a limited number of possible values. With Spans that's not the case.

An example would be a method returning a unique identifier: It would be valid to use the return value as a Span attribute, but it would be a bad idea to use it as a metric attribute.

For method parameters the current proposal is to have two separate annotations: @MetricAttribute and @SpanAttribute. If you want a parameter to be used in both, metric and span, you would need to add both annotations.

If we want to have the same separation for return values, we would end up with something like @MetricReturnAttribute and @SpanReturnAttribute. That's fine. I find it shorter to have this as a parameter of @Counted, @Timed, and maybe @WithSpan, but I have no strong opinion about this.

However, having a common @ReturnAttribute for both metrics and spans would make it impossible to use a high cardinality return value as a Span attribute without using it as a metric attribute at the same time.

What do you think?

Copy link
Copy Markdown
Member

@trask trask Mar 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I understand now, yeah that makes sense that you would want control over which attributes are tagged on the span vs tagged on the metric, thx

Comment on lines +21 to +23
* <li><b>class:</b> The fully qualified name of the class whose method is invoked.
* <li><b>method:</b> The name of the annotated method, or "new" of the annotation is on a
* constructor.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of class and method, WDYT about using code.namespace and code.function? These attributes are already in the spec (for spans, at least), and already are (or should be) captured by the agent.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the heads-up. If that's the standard attribute names from the Spec we should use them.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed an update renaming this to the standard attributes.

Comment on lines +24 to +25
* <li><b>exception:</b> The {@link Class#getSimpleName() simple name} of the Exception if an
* Exception is thrown, or "None" if the method did not throw an Exception.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, can we use exception.type and Class#getCanonicalName() instead? It's what Span#recordException() uses.

Copy link
Copy Markdown
Member Author

@fstab fstab Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the heads-up. This was inspired by what Micrometer does, but if there are standard attributes in OpenTelemetry we should use them.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed an update using exception.type and the canonical name.

* <p>Setting {@code disableExceptionAttribute = true} will disable the {@code exception}
* attribute.
*/
boolean disableExceptionAttribute() default false;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this method necessary? The same thing can be achieved by a View if somebody wants that; and turning off the exception capturing does not seem that useful.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if there is a situation where Exceptions can result in cardinality explosion because a large number of different Exception classes is thrown. If this can happen it would make sense to have a way to disable the exception attribute. If we are sure that Exception classes are always low cardinality we could remove this.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's technically possible to have hight cardinality here (like autogenerating exception classes at runtime 🙈 ), but it's very low likely to happen -- and again, you can use Views to filter this attribute out if you somehow happen to run into such scenario.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed an update removing the disableExceptionAttribute parameter.

*
* <ul>
* <li><b>code.namespace:</b> The fully qualified name of the class whose method is invoked.
* <li><b>code.function:</b> The name of the annotated method, or "new" of the annotation is on a
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* <li><b>code.function:</b> The name of the annotated method, or "new" of the annotation is on a
* <li><b>code.function:</b> The name of the annotated method, or "new" if the annotation is on a

* <li><b>code.namespace:</b> The fully qualified name of the class whose method is invoked.
* <li><b>code.function:</b> The name of the annotated method, or "new" of the annotation is on a
* constructor.
* <li><b>exception.type:</b> The {@link Class#getCanonicalName()} () canonical name} of the
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* <li><b>exception.type:</b> The {@link Class#getCanonicalName()} () canonical name} of the
* <li><b>exception.type:</b> The {@link Class#getCanonicalName() canonical name} of the

*
* <p>The default name is {@code method.invocations.total}.
*/
String value() default "method.invocations.total";
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since method.invocations.total hasn't been spec'd, maybe we shouldn't have a default, or maybe we should move the metrics annotations to an alpha package for now?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving to an alpha package sounds good. Could you let me know how to do this? I don't know the conventions too well.

* })
* </pre>
*/
String[] additionalAttributes() default {};
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious, do you have a particular use case in mind for these additional attributes?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something that comes to my mind would be this:

  @Timed(attributes = {"paymentType", "CreditCard"})
  public void processPayment(CreditCard creditCard) {
    // ...
  }

  @Timed(attributes = {"paymentType", "DebitCard"})
  public void processPayment(DebitCard debitCard) {
    // ...
  }

Does this make sense?

* <li><b>code.function:</b> The name of the annotated method, or "new" of the annotation is on a
* constructor.
* <li><b>exception.type:</b> The {@link Class#getCanonicalName()} () canonical name} of the
* Exception if an Exception is thrown, or "None" if the method did not throw an Exception.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on the instrumentation side, we don't populate metric attributes when they are n/a, I definitely don't understand some of the nuances of metric design, is populating "None" needed for some reason?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The reason why it's there is because that's what Micrometer does, but I guess not populating it would be better.

In Prometheus missing attributes are treated the same way as empty attributes. I pushed an update saying that the attribute is only present if an Exception is thrown. Is this ok, or would it be better to always keep the attribute but with an empty value?

*
* <p>By default, the instrument will not have an attribute with the return value.
*/
String returnValueAttribute() default "";
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @mateuszrzeszutek was suggesting something like this:

  @Timed
  @ReturnAttribute("process.result")
  public int process() {
    ...
  }

@tylerlance
Copy link
Copy Markdown

Do we have a planned version for this to be merge for? This seems like a valuable feature. Hoping it wasn't abandoned.

@jack-berg
Copy link
Copy Markdown
Member

It feels strange accepting additions to the stable extension annotations module without knowing what an implementation of a corresponding annotation processor looks like.

Can opentelemetry-extension-annotations move to opentelemetry-java-instrumentation to live alongside the annotation processor implementation?

If not, can we we put this in opentelemetry-extension-incubator while we build confidence in the API / implementation?

@fstab
Copy link
Copy Markdown
Member Author

fstab commented Jun 2, 2022

My initial impulse is that opentelemetry-extension-incubator is a better fit because users would explicitly use these annotations in their code. Should I move it?

@fstab fstab force-pushed the metric-annotations branch from 498a873 to eaa90a3 Compare June 11, 2022 15:04
@fstab
Copy link
Copy Markdown
Member Author

fstab commented Jun 11, 2022

I moved it to extensions/incubator/. The new annotations are now in package io.opentelemetry.extension.incubator.annotations.

From the discussion above I think the only unresolved point is how to use return values as attributes.

Current implementation:

  @Timed(returnValueAttribute = "process.result")
  public int process() {
    return 3;
  }

Alternative proposal:

  @Timed
  @ReturnAttribute("process.result")
  public int process() {
    return 3;
  }

I think the idea behind the alternative proposal is that @ReturnAttribute could work for both, spans and metrics. However, there is a fundamental difference between spans and metrics: For metrics, each attribute value creates a new time series. If you use a high cardinality return value as a metric attribute, you might overload the monitoring backend. For spans, high cardinality attributes are not an issue.

For that reason I think we should provide a way to say that a return value will be a span attribute, but not a metric attribute. With returnValueAttribute being a property of the @Timed and @Counted annotations, it is clear that this is only for the metric but does not affect any spans.

Anyway, I'm happy to change this if consensus is that we should go with a dedicated @ReturnAttribute annotation.

@jkwatson
Copy link
Copy Markdown
Contributor

@fstab sorry about the delay on this. We're going to discuss at today's OTel java office hours and will respond with a recommendation on next steps ASAP.

@jack-berg
Copy link
Copy Markdown
Member

Hi @fstab just following up on @jkwatson's comment. Here's where we're at:

  • We don't think opentelemetry-java is the right long term home for these annotations, given that the primary implementation lives in opentelemetry-java-instrumentation. We'd like to make opentelemetry-java-instrumentation the long term home of annotations.
  • Modules in opentelemetry-java-instrumentation generally live in io.opentelemetry.instrumentation.* package, versus io.opentelemetry.* for opentelemetry-java projects. In order to continue following this pattern while not breaking compatibility for the annotations module, we're going to start a new annotations module in opentelemetry-java-instrumentation, deprecating the opentelemetry-java opentelemetry-extension-annotations module until we ultimately stop publishing it starting with the next major version.
  • We'd like to not get further away from this target, and merging this until the span annotations have been moved to opentelemetry-java-instrumentation. At that point, we can open a PR adding these annotations to that new module, and simultaneously include an implementation for the annotations.

Again, sorry for the delay.

@trask
Copy link
Copy Markdown
Member

trask commented Jun 30, 2022

I've kicked off the discussion in the instrumentation repo open-telemetry/opentelemetry-java-instrumentation#6245, and added the topic of naming to the SIG agenda for tomorrow. Once we close on naming, I'll do the initial copy of existing annotations over to the instrumentation repo.

* <p>Unit strings should follow the instrument unit rules: <a
* href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#instrument-unit">https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md#instrument-unit</a>
*/
String unit() default "";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this default to "1" ?

*
* <p>Default is seconds.
*/
TimeUnit unit() default TimeUnit.SECONDS;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@github-actions
Copy link
Copy Markdown
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions Bot added the Stale label Jul 22, 2022
@jack-berg jack-berg removed the Stale label Jul 22, 2022
@trask
Copy link
Copy Markdown
Member

trask commented Aug 1, 2022

hey @fstab! we've finished moving the @WithSpan instrumentation annotations to the instrumentation repo (https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation-annotations). we're keeping the existing annotations around in this repository (and still supporting them in auto-instrumentation for backwards compatibility), but would like to add new features going forwards in the new module now

@jack-berg
Copy link
Copy Markdown
Member

I'm going to close this because now that we've moved forward with the decision to move annotations to opentelemetry-java-instrumentation.

@cptkng
Copy link
Copy Markdown

cptkng commented Apr 1, 2023

@fstab Could you summarize what the current status of your (great) idea with annotations for metrics is?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants