migrate MessagingMetrics to micrometer #27848

entangled90 · 2025-02-07T16:00:56Z

Description

Migrates MessagingMetrics to micrometers.
There are a couple of timers, so dashboard changes are needed.

Dashboard changes

Related issues

closes #27082

npepinpe

Found some issues already, I'll review Monday when it's ready

npepinpe · 2025-02-07T17:05:26Z

zeebe/atomix/cluster/src/main/java/io/atomix/cluster/messaging/impl/MessagingMetricsDoc.java

+  REQUEST_RESPONSE_LATENCY {
+    @Override
+    public String getName() {
+      return "zeebe.messaging_request.response.latency";


❓ Kept a _ there on purpose, or?

Suggested change

return "zeebe.messaging_request.response.latency";

return "zeebe.messaging.request.response.latency";

npepinpe · 2025-02-07T17:07:03Z

zeebe/atomix/cluster/src/main/java/io/atomix/cluster/messaging/impl/MessagingMetricsDoc.java

+  IN_FLIGHT_REQUESTS {
+    @Override
+    public String getName() {
+      return "messaging_inflight_requests";


Suggested change

return "messaging_inflight_requests";

return "zeebe.messaging.inflight.requests";

npepinpe · 2025-02-07T17:07:26Z

zeebe/atomix/cluster/src/main/java/io/atomix/cluster/messaging/impl/MessagingMetricsDoc.java

+    }
+  };
+
+  enum MessagingKeyNames implements KeyName {


🔧 Please add some documentation for the key names

npepinpe · 2025-02-07T17:09:29Z

zeebe/atomix/cluster/src/main/java/io/atomix/cluster/messaging/impl/MessagingMetricsImpl.java

+    return Counter.builder(RESPONSE_COUNT.getName())
+        .description(RESPONSE_COUNT.getDescription())
+        .tags(
+            MessagingKeyNames.TYPE.asString(),


❌ Should be "outcome", not "type".

npepinpe · 2025-02-07T17:10:24Z

zeebe/atomix/cluster/src/main/java/io/atomix/cluster/messaging/impl/MessagingMetricsImpl.java

+  private static final String MESSAGE = "MESSAGE";
+  private static final String REQ_RESP = "REQ_RESP";


🔧 If I understood correctly, these are the two possible values for the type label? We should make it an enum.

npepinpe

🚀 Did you test it? As this is quite a substantial PR.

npepinpe · 2025-02-10T16:23:28Z

monitor/grafana/zeebe.json

              },
              "editorMode": "code",
-              "expr": "histogram_quantile(0.50, sum(rate(zeebe_messaging_request_response_latency_bucket{cluster=~\"$cluster\", namespace=~\"$namespace\",pod=~\"$pod\", partition=~\"$partition\"}[$__rate_interval])) by (topic, le))",
+              "expr": "histogram_quantile(0.50, sum(rate(zeebe_messaging_request_response_latency_bucket{cluster=~\"$cluster\", namespace=~\"$namespace\",pod=~\"$pod\", partition=~\"$partition\"}[$__rate_interval]) or rate(zeebe_messaging_request_response_latency_seconds_bucket{cluster=~\"$cluster\", namespace=~\"$namespace\",pod=~\"$pod\", partition=~\"$partition\"}[$__rate_interval])) by (topic, le))",


🔧 Does this show things properly? I ran into an issue with the union of series and heatmap data earlier, though I was not using histogram_quantile, so maybe that works better.

They look fine to me, similar to other benchmarks

npepinpe · 2025-02-10T16:23:50Z

monitor/grafana/zeebe.json

                "uid": "$DS_PROMETHEUS"
              },
              "editorMode": "code",
-              "expr": "sum(increase(zeebe_messaging_request_response_latency_bucket{cluster=~\"$cluster\", namespace=~\"$namespace\",partition=~\"$partition\",pod=~\"$pod\", topic=~\"atomix-membership-probe-request\"}[$__rate_interval])) by (le)",


🔧 Same as above, for the others in general. Did you double check?

Look fine to me?

entangled90 · 2025-02-10T16:52:47Z

Yes I did check the dashboard it looks fine to me, I hope I didn't miss anything

relates #26078

…etrics

backport-action · 2025-02-10T18:18:51Z

Created backport PR for stable/8.5:

[Backport stable/8.5] migrate MessagingMetrics to micrometer #27947 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-27848-to-stable/8.5
git worktree add --checkout .worktree/backport-27848-to-stable/8.5 backport-27848-to-stable/8.5
cd .worktree/backport-27848-to-stable/8.5
git reset --hard HEAD^
git cherry-pick -x 4a4a767762119b45828ef5449b1d6302b4861846 f4324309af5d04d61cef9eef07267314cc0efb55 bdb959f4b9a2527d89b4fb3dde953529313e578d 89ec8f33e6fa63c3d80beba6ea5d2b21485809c4 b7fd8ceb3dbf2040315d9819a49d9eaaacac1298 1cf487dfa4a8559e579c0be2400a43a066a77fb5 a88665609955707e6346e5e1ad4a79c35336aec3 6c7df9d081f0c3ca65d6c05059005e905b7e5284 8774e00c3d64928ed220ce5a51d07e64b129039c 8261d97293dbb5c6234db03a147dbcc4d4062cfa 28ddf16cc98d2a79ee1981fa89b53459e6355611 bcc389615bec1af9faac9ef3ae130f9b18ba6b22 9bd85d6bf41dc26227a7a1228b34d93e4b20e699 e99390ec0296c1cf277cd52ee902ec53f24099a3 5034256a8a0bad915cf76c5d0f2efffa595cbea3 688f49286655b7da789c85a7034a895d6b7f2a01 74153bb784864cecbfc3760947019a35fee9ca2d 16108868b4fe9c0de9b83dbdbb555460285a9b90 2bff71b3ece40c34d2fe39c7c6c91ef32c3da1ad
git push --force-with-lease

backport-action · 2025-02-10T18:19:02Z

Created backport PR for stable/8.6:

[Backport stable/8.6] migrate MessagingMetrics to micrometer #27948 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-27848-to-stable/8.6
git worktree add --checkout .worktree/backport-27848-to-stable/8.6 backport-27848-to-stable/8.6
cd .worktree/backport-27848-to-stable/8.6
git reset --hard HEAD^
git cherry-pick -x 4a4a767762119b45828ef5449b1d6302b4861846 f4324309af5d04d61cef9eef07267314cc0efb55 bdb959f4b9a2527d89b4fb3dde953529313e578d 89ec8f33e6fa63c3d80beba6ea5d2b21485809c4 b7fd8ceb3dbf2040315d9819a49d9eaaacac1298 1cf487dfa4a8559e579c0be2400a43a066a77fb5 a88665609955707e6346e5e1ad4a79c35336aec3 6c7df9d081f0c3ca65d6c05059005e905b7e5284 8774e00c3d64928ed220ce5a51d07e64b129039c 8261d97293dbb5c6234db03a147dbcc4d4062cfa 28ddf16cc98d2a79ee1981fa89b53459e6355611 bcc389615bec1af9faac9ef3ae130f9b18ba6b22 9bd85d6bf41dc26227a7a1228b34d93e4b20e699 e99390ec0296c1cf277cd52ee902ec53f24099a3 5034256a8a0bad915cf76c5d0f2efffa595cbea3 688f49286655b7da789c85a7034a895d6b7f2a01 74153bb784864cecbfc3760947019a35fee9ca2d 16108868b4fe9c0de9b83dbdbb555460285a9b90 2bff71b3ece40c34d2fe39c7c6c91ef32c3da1ad
git push --force-with-lease

backport-action · 2025-02-10T18:19:07Z

Created backport PR for stable/8.7:

[Backport stable/8.7] migrate MessagingMetrics to micrometer #27949 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-27848-to-stable/8.7
git worktree add --checkout .worktree/backport-27848-to-stable/8.7 backport-27848-to-stable/8.7
cd .worktree/backport-27848-to-stable/8.7
git reset --hard HEAD^
git cherry-pick -x 4a4a767762119b45828ef5449b1d6302b4861846 f4324309af5d04d61cef9eef07267314cc0efb55 bdb959f4b9a2527d89b4fb3dde953529313e578d 89ec8f33e6fa63c3d80beba6ea5d2b21485809c4 b7fd8ceb3dbf2040315d9819a49d9eaaacac1298 1cf487dfa4a8559e579c0be2400a43a066a77fb5 a88665609955707e6346e5e1ad4a79c35336aec3 6c7df9d081f0c3ca65d6c05059005e905b7e5284 8774e00c3d64928ed220ce5a51d07e64b129039c 8261d97293dbb5c6234db03a147dbcc4d4062cfa 28ddf16cc98d2a79ee1981fa89b53459e6355611 bcc389615bec1af9faac9ef3ae130f9b18ba6b22 9bd85d6bf41dc26227a7a1228b34d93e4b20e699 e99390ec0296c1cf277cd52ee902ec53f24099a3 5034256a8a0bad915cf76c5d0f2efffa595cbea3 688f49286655b7da789c85a7034a895d6b7f2a01 74153bb784864cecbfc3760947019a35fee9ca2d 16108868b4fe9c0de9b83dbdbb555460285a9b90 2bff71b3ece40c34d2fe39c7c6c91ef32c3da1ad
git push --force-with-lease

@entangled90

# Description Backport of #27848 to `stable/8.5`. relates to #27082 original author: @entangled90

@entangled90

…28192) ## Description Backport of #27848 to stable/8.7. original author: @entangled90 Backport of #27848 from stable/8.5 to stable/8.7. Original PR was made in main, but there are less conflicts from stable/8.5 Does not contain backport of #28108, it must be merged afterwards ## Related issues relates to #27082

@entangled90

…28594) # Description Backport of #28192 to `stable/8.6`. relates to #27848 #27848 #27082 original author: @entangled90

entangled90 self-assigned this Feb 7, 2025

entangled90 marked this pull request as draft February 7, 2025 16:01

github-actions bot added the component/zeebe Related to the Zeebe component/team label Feb 7, 2025

npepinpe requested changes Feb 7, 2025

View reviewed changes

entangled90 force-pushed the cs-26078-migrate-messaging-metrics branch from 16cce10 to 6ce28c5 Compare February 7, 2025 17:17

entangled90 marked this pull request as ready for review February 10, 2025 08:14

entangled90 force-pushed the cs-26078-migrate-messaging-metrics branch from a1a149f to 507508b Compare February 10, 2025 08:24

github-actions bot added component/operate Related to the Operate component/team component/tasklist Related to the Tasklist component/team component/optimize Related to Optimize component/team labels Feb 10, 2025

entangled90 force-pushed the cs-26078-migrate-messaging-metrics branch 3 times, most recently from dc96efc to 7986087 Compare February 10, 2025 09:41

entangled90 requested a review from npepinpe February 10, 2025 11:35

npepinpe approved these changes Feb 10, 2025

View reviewed changes

entangled90 added 8 commits February 10, 2025 18:07

feat: migrated MessagingMetrics to micrometer

4a4a767

relates #26078

feat: migrate NettyDnsMetric to micrometer

f432430

feat: migrate SwimMembershipProtocolMetrics to micrometer

bdb959f

deps: remove prometheus from atomix-cluster

89ec8f3

fix: wrong parameters order for RESPONSE_COUNT metric

b7fd8ce

fix: distributionSLOs for request size equal to prometheus buckets

1cf487d

docs: add JavaDoc to MessagingKeyNames

a886656

style: format

6c7df9d

entangled90 added 5 commits February 10, 2025 18:07

fix: do not perform nested computeIfAbsent in SwimMembershipProtocolM…

bcc3896

…etrics

deps: missing dependency in zeebe-protocol-test-util

9bd85d6

deps: add micrometer-core to zeebe/broker-client

e99390e

deps: scope micrometer-core to test only in broker-clientt

5034256

deps: add micrometer-core to gateway-grpc for tests

688f492

entangled90 force-pushed the cs-26078-migrate-messaging-metrics branch from e3538f3 to c91e636 Compare February 10, 2025 17:07

entangled90 enabled auto-merge February 10, 2025 17:16

entangled90 added 3 commits February 10, 2025 18:40

deps: add micrometer-core to gateway (not just tests)

74153bb

deps: remove duplicated micrometer dep in gateway-grpc

1610886

test: fix variable naming

2bff71b

entangled90 force-pushed the cs-26078-migrate-messaging-metrics branch from f4c5036 to 2bff71b Compare February 10, 2025 17:44

entangled90 added this pull request to the merge queue Feb 10, 2025

Merged via the queue into main with commit 0a66ca8 Feb 10, 2025
71 of 72 checks passed

entangled90 deleted the cs-26078-migrate-messaging-metrics branch February 10, 2025 18:18

backport-action mentioned this pull request Feb 10, 2025

[Backport stable/8.5] migrate MessagingMetrics to micrometer #27947

Merged

backport-action mentioned this pull request Feb 10, 2025

[Backport stable/8.6] migrate MessagingMetrics to micrometer #27948

Closed

backport-action mentioned this pull request Feb 10, 2025

[Backport stable/8.7] migrate MessagingMetrics to micrometer #27949

Closed

github-merge-queue bot pushed a commit that referenced this pull request Feb 13, 2025

[Backport stable/8.5] migrate MessagingMetrics to micrometer (#27947)

ebda3e5

# Description Backport of #27848 to `stable/8.5`. relates to #27082 original author: @entangled90

github-actions bot added the version:8.5.15 label Feb 14, 2025

entangled90 mentioned this pull request Feb 14, 2025

[Backport stable/8.7] migrate MessagingMetrics to micrometer #27949 #28192

Merged

backport-action mentioned this pull request Feb 21, 2025

[Backport stable/8.6] Migrate MessagingMetrics to micrometer #27949 #28594

Merged

github-merge-queue bot pushed a commit that referenced this pull request Mar 2, 2025

[Backport stable/8.6] Migrate MessagingMetrics to micrometer #27949 (#…

e3dbafd

…28594) # Description Backport of #28192 to `stable/8.6`. relates to #27848 #27848 #27082 original author: @entangled90

koevskinikola added the version:8.7.0-alpha5 label Mar 6, 2025

github-actions bot added the version:8.6.11 label Mar 6, 2025

korthout added the version:8.7.0 Marks an issue as being completely or in parts released in 8.7.0 label Apr 2, 2025

	return "zeebe.messaging_request.response.latency";
	return "zeebe.messaging.request.response.latency";

	return "messaging_inflight_requests";
	return "zeebe.messaging.inflight.requests";

		private static final String MESSAGE = "MESSAGE";
		private static final String REQ_RESP = "REQ_RESP";

migrate MessagingMetrics to micrometer #27848

migrate MessagingMetrics to micrometer #27848

Uh oh!

Conversation

entangled90 commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Uh oh!

npepinpe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

npepinpe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

entangled90 commented Feb 10, 2025

Uh oh!

Uh oh!

backport-action commented Feb 10, 2025

Uh oh!

backport-action commented Feb 10, 2025

Uh oh!

backport-action commented Feb 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

entangled90 commented Feb 7, 2025 •

edited

Loading