Environment

SaaS (https://sentry.io/)

Steps to Reproduce

Run a Java service with Sentry SDK v7.9.0
Have a piece of code that logs higher-volume metrics and lower-volume metrics
Notice that while higher-volume metrics generally appear in Sentry dashboards, lower-volume metrics rarely make it.

Expected Result

Both high-volume and low-volume metrics are sent and are not sampled on the client side.

Actual Result

High volume metrics appear in the dashboard, although at much lower rates than the incidence of those metrics.
Low volume metrics frequently don't appear on the dashboard nor in the list of emitted metrics under "Settings > {Org} > {Project} > Metrics". Sometimes those low-volume metrics appear days after the initial metric has been emitted.

Additionally, both types of metrics appear to be sampled (on the client?) since the values seen in the Sentry dashboards are much lower than from estimating event incidence by our logs.

Here's a sample of the events for Google Drive API "exportSizeLimitExceeded" error from our logs:

(base) backend-server-dev:~/backend-server-dev$ zgrep -c exportSizeLimitExceeded logs/backend.2024-0* | egrep -v ':0$'
logs/backend.2024-05-29.0.log.zip:6
logs/backend.2024-05-31.0.log.zip:4
logs/backend.2024-06-03.0.log.zip:4
logs/backend.2024-06-04.0.log.zip:2

Whereas here's the screenshot of the Sentry dashboard showing that metrics for the last 14d -- note that it only shows up once on May 31st

Likewise here are some counts from our preview environment of the number of files transferred via the Google Drive API from logs vs. the Sentry metric; note that the max counts/day are 200+ for the high-volume days where as our max value is 71 for that date range.

(base) backend-server-dev:~/backend-server-dev$ zgrep -c 'Transferring file' logs/backend.2024-0* |egrep -v ':0$'
logs/backend.2024-05-31.0.log.zip:257
logs/backend.2024-06-03.0.log.zip:217
logs/backend.2024-06-04.0.log.zip:30
logs/backend.2024-06-05.0.log.zip:21
logs/backend.2024-06-07.0.log.zip:44
logs/backend.2024-06-11.0.log.zip:105
logs/backend.2024-06-12.0.log.zip:128

Note that I've run the Java SDK with debug enabled, and the only messages re: metrics I see are the following types of messages; per your support team I was told I should see the names of captured metrics here but that doesn't appear to be the case for any of the environments I've enabled Sentry debug in:

DEBUG: Metrics: nothing to flush
DEBUG: Metrics: nothing to flush
DEBUG: Metrics: flushing 1 buckets
DEBUG: Metrics: capturing metrics
DEBUG: Envelope sent successfully.
DEBUG: Envelope flushed

Product Area

DDM

Link

https://memyard.sentry.io/metrics/?interval=1d&project=4507091072385024&statsPeriod=30d&utc=true&widgets=%5B%7B%22mri%22%3A%22c%3Acustom%2Fgoogle_drive.transfer_file.start%40none%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A0%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%2C%7B%22mri%22%3A%22c%3Acustom%2Fgoogle_drive.transfer_file.success%40none%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A1%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%2C%7B%22mri%22%3A%22c%3Acustom%2Fgoogle_drive.transfer_file.failure%40none%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A2%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%2C%7B%22mri%22%3A%22d%3Acustom%2Fstorage_service.train_drive.available_files%40count%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A3%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%2C%7B%22mri%22%3A%22d%3Acustom%2Fstorage_service.train_drive.indexable_files%40count%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A4%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%2C%7B%22mri%22%3A%22d%3Acustom%2Fstorage_service.train_drive.transferred_files%40count%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A5%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%2C%7B%22mri%22%3A%22c%3Acustom%2Fgoogle_drive.transfer_file.failure.file_not_found%40none%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A6%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%2C%7B%22mri%22%3A%22c%3Acustom%2Fgoogle_drive.transfer_file.failure.export_size_limit_exceeded%40none%22%2C%22op%22%3A%22sum%22%2C%22query%22%3A%22%22%2C%22groupBy%22%3A%5B%5D%2C%22powerUserMode%22%3Afalse%2C%22type%22%3A1%2C%22id%22%3A7%2C%22displayType%22%3A%22line%22%2C%22focusedSeries%22%3A%5B%5D%2C%22sort%22%3A%7B%22order%22%3A%22asc%22%7D%2C%22isHidden%22%3Afalse%2C%22overlays%22%3A%5B%22samples%22%5D%7D%5D

DSN

https://8f29233cca683e7b0a93e6a2b859bb17@o4506853392973824.ingest.us.sentry.io/4507091072385024

Version

No response

┆Issue is synchronized with this Jira Improvement by Unito