Skip to content

Add ExponentialHistogram to Metrics data model#1935

Merged
jsuereth merged 26 commits into
open-telemetry:mainfrom
jmacd:jmacd/expohistomodel
Oct 7, 2021
Merged

Add ExponentialHistogram to Metrics data model#1935
jsuereth merged 26 commits into
open-telemetry:mainfrom
jmacd:jmacd/expohistomodel

Conversation

@jmacd

@jmacd jmacd commented Sep 16, 2021

Copy link
Copy Markdown
Contributor

Fixes #1932.

Changes

Adds statements describing the ExponentialHistogram data point introduced by open-telemetry/opentelemetry-proto#322.

This is meant to capture details about acceptable precision limits, how to handle out-of-range values for producers and consumers. See open-telemetry/opentelemetry-proto#322 (comment)

Related OTEPs:
https://github.com/open-telemetry/oteps/blob/main/text/0149-exponential-histogram.md

Comment thread specification/metrics/datamodel.md
@reyang reyang added area:data-model For issues related to data model spec:metrics Related to the specification/metrics directory labels Sep 17, 2021
@jmacd
jmacd marked this pull request as ready for review September 20, 2021 21:01
@jmacd
jmacd requested review from a team September 20, 2021 21:01
@jmacd

jmacd commented Sep 20, 2021

Copy link
Copy Markdown
Contributor Author

@open-telemetry/specs-metrics-approvers this is ready for review.

Joshua MacDonald added 2 commits September 20, 2021 14:43
@jmacd

jmacd commented Sep 21, 2021

Copy link
Copy Markdown
Contributor Author

@yzhuge and @oertl please review

Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
@jmacd

jmacd commented Sep 21, 2021

Copy link
Copy Markdown
Contributor Author

FYI I have posted reference implementations that calculate bucket indexes described by the formulas here. These were captured from the review thread of open-telemetry/opentelemetry-proto#322 here, now available:
open-telemetry/oteps#179

Comment thread specification/metrics/datamodel.md
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md
Comment thread specification/metrics/datamodel.md Outdated

@jmacd jmacd left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @yzhuge!

Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
@yzhuge

yzhuge commented Sep 24, 2021

Copy link
Copy Markdown
Contributor

@jmacd all looks good now.

@yzhuge

yzhuge commented Oct 5, 2021

Copy link
Copy Markdown
Contributor

@jmacd "the lower-boundary of index -68 cannot be represented using a double".
In negative scales, the lowest bucket may be a "partial bucket" (partially representable by "double" range).
An implementation's lower bound function will need to guard against floating point underflow. In NrSketch, I have https://github.com/newrelic-experimental/newrelic-sketch-java/blob/main/src/main/java/com/newrelic/nrsketch/indexer/ExponentIndexer.java#L43

Similarly, to be bullet proof, we need guard against overflow at the high end of "double":
https://github.com/newrelic-experimental/newrelic-sketch-java/blob/main/src/main/java/com/newrelic/nrsketch/indexer/ScaledExpIndexer.java#L114

Not sure it is worth mentioning such extreme cases in the spec though. Maybe it is, spec has to be bullet proof.

"Should the producer bother to check that an index can be reverse-mapped before using it?"
I think producer should still map it to the lowest (partial) bucket. Its other choices are: round off to zero; throw an exception. None is great.

And I assume you are talking about the exponent extraction method. The log method generally does not work well in the subnormal range. I had to skip it in subnormal tests (https://github.com/newrelic-experimental/newrelic-sketch-java/blob/main/src/test/java/com/newrelic/nrsketch/indexer/BucketIndexerTest.java#L400)

Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated
Comment thread specification/metrics/datamodel.md Outdated

@reyang reyang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread specification/metrics/datamodel.md Outdated
@jsuereth
jsuereth merged commit 7b52806 into open-telemetry:main Oct 7, 2021
carlosalberto pushed a commit to carlosalberto/opentelemetry-specification that referenced this pull request Oct 31, 2024
* Add ExponentialHistogram to Metrics data model

* draft expectations

* toc

* remove one 'exponential'

* mention the use of logarithm and inexact computation

* manual edit TOC

* typo

* reduce precision

* from Yuke's feedback

* mapping methods

* several fixes from yzhuge

* lint

* update links

* Changelog

* mention min/max

* let consumers deal with overflow and underflow

* yzhuge's remarks

* whitespace

* Apply suggestions from code review

Co-authored-by: Reiley Yang <[email protected]>

* revert TOC trouble etc

* upcase

* Update specification/metrics/datamodel.md

Co-authored-by: Aaron Abbott <[email protected]>

Co-authored-by: Reiley Yang <[email protected]>
Co-authored-by: Aaron Abbott <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:data-model For issues related to data model spec:metrics Related to the specification/metrics directory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Specify data model details for ExponentialHistogram

9 participants