Metric Error Codes Recording + Error Code Prefixing by jake-engelberg · Pull Request #8256 · dapr/dapr

jake-engelberg · 2024-10-30T23:30:59Z

Description

Error codes that were defined and used on a per-developer basis for debug logging has been temporarily moved to a singular errorcodes package to be defined and prefixed for cohesion. Eventually, they will be converted to structures similar to predefined.go for all errors and associated messages.

Additionally, error codes are now retrieved using a function which also records a count metric for tracking errors.

No renaming has been done to assure backwards compatability.

The metric can be filtered by its keys for app-id, error code, and error category.

The error metric recording is opt-in via the flag metrics.recordErrorCodes.

Metric fields/schema:

dapr_error_code_total
{
    app_id = "errorgen-app",
    category = "pubsub",
    dapr_io_enabled = "true",
    error_code = "ERR_PUBSUB_NOT_FOUND",
    instance = "10.244.0.35:9090",
    job = "kubernetes-service-endpoints",
    namespace = "crud-app",
    node = "dapr-master-control-plane",
    service = "errorgen-app-dapr"
}

Example of query dapr_error_code_total in Grafana:

Example of query rate(dapr_error_code_total[2m]) in Grafana:

Another img:

Issue reference

Please reference the issue this PR will close: dapr/proposals#67

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

Code compiles correctly
Created/updated tests
Unit tests passing
End-to-end tests passing
Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: Add exhaustive list of error codes and description of metric flag docs#4422
Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: Add exhaustive list of error codes and description of metric flag docs#4422
Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: Add exhaustive list of error codes and description of metric flag docs#4422

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

Signed-off-by: Jake Engelberg <[email protected]>

This reverts commit 53a02b5. Done so to implement in a separate PR. Signed-off-by: Jake Engelberg <[email protected]>

This reverts commit 44c31be. Signed-off-by: Jake Engelberg <[email protected]>

… var will be logged Signed-off-by: Jake Engelberg <[email protected]>

Signed-off-by: Jake Engelberg <[email protected]>

…fix unit tests Signed-off-by: Jake Engelberg <[email protected]>

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg · 2024-11-22T02:13:23Z

dapr/kit#109

This PR makes some changes to dapr/kit that are necessary for the latest commit I made.

Signed-off-by: Jake Engelberg <[email protected]>

elena-kolevska

Looks like we're almost done! Just a few small nits.

elena-kolevska · 2024-11-26T14:08:56Z

 	}

-	return kiterrors.NewBuilder(s.Code(), httpCode, s.Message(), info.GetMetadata()[tagField]).
+	return kiterrors.NewBuilder(s.Code(), httpCode, s.Message(), info.GetMetadata()[tagField], "unknown").


We need to look a bit deeper into this, if I remember correctly how pluggable components work, we won't be able to extract the original category, but we could maybe create a new generic one, for example, pluggable-state. @JoshVanL what do you think?
For context (so you don't have to look through the whole PR), we added a category field needed for recording metrics to kit and APIErrors, but we never return it as part of the final error the client sees. Does that mean a pluggable component won't have access to it?

I'm also ok marking this as category pluggable-component for now, and split it in the future if necessary.

Signed-off-by: Jake Engelberg <[email protected]>

elena-kolevska · 2024-11-27T01:18:09Z

Looking great! Let's wait for the tests the pass and I'll approve. Good job Jake!

Signed-off-by: Jake Engelberg <[email protected]>

marcduiker · 2024-12-03T08:19:02Z

@holopin-bot @jake-engelberg Thanks Jake!

holopin-bot · 2024-12-03T08:19:05Z

Congratulations @jake-engelberg, the maintainer of this repository has issued you a badge! Here it is: https://holopin.io/claim/cm486tyw200580cl88wmkqfj2

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!

* further fine tuning Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * try fixing borked e2e Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * try fixing standalone validation Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * install numpy too Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * update pip & install pkgs globally instead of venv Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * install requests Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * // -> # for commented line Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * match only installed version of powershell Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> * wip: errorcode const consolidation Signed-off-by: Jake Engelberg <[email protected]> * wip: error code monitoring/metric recording Signed-off-by: Jake Engelberg <[email protected]> * style: error_code -> errorcode Signed-off-by: Jake Engelberg <[email protected]> * fix: error changes from refactor Signed-off-by: Jake Engelberg <[email protected]> * remove 2 unused error codes Signed-off-by: Jake Engelberg <[email protected]> * Revert "wip: error code monitoring/metric recording" This reverts commit 53a02b5. Done so to implement in a separate PR. Signed-off-by: Jake Engelberg <[email protected]> * Revert "Revert "wip: error code monitoring/metric recording"" This reverts commit 44c31be. Signed-off-by: Jake Engelberg <[email protected]> * use RecordAndGet() to record error code metric only whenever ApiError var will be logged Signed-off-by: Jake Engelberg <[email protected]> * undo invalid error code recording Signed-off-by: Jake Engelberg <[email protected]> * apply cohesive prefixes to error codes Signed-off-by: Jake Engelberg <[email protected]> * correctly apply metric recording to inline string error definitions, fix unit tests Signed-off-by: Jake Engelberg <[email protected]> * add explicit "type" field for error code metrics Signed-off-by: Jake Engelberg <[email protected]> * linting improvements Signed-off-by: Jake Engelberg <[email protected]> * errorcode recording unit test, fix: register type field on init Signed-off-by: Jake Engelberg <[email protected]> * add category to error codes for O(1) operation on recording Signed-off-by: Jake Engelberg <[email protected]> * use error code vars in tests Signed-off-by: Jake Engelberg <[email protected]> * fumpt Signed-off-by: Jake Engelberg <[email protected]> * convert last test strings, fix some integration tests Signed-off-by: Jake Engelberg <[email protected]> * fix errorcodes compile error Signed-off-by: Jake Engelberg <[email protected]> * golangci-lint fixes Signed-off-by: Jake Engelberg <[email protected]> * lint fix Signed-off-by: Jake Engelberg <[email protected]> * remove debug binary Signed-off-by: Jake Engelberg <[email protected]> * Revert "lint fix" This reverts commit 392080a. Revert "golangci-lint fixes" This reverts commit d6c6066. Revert "fix errorcodes compile error" This reverts commit 9739534. Revert "convert last test strings, fix some integration tests" This reverts commit ae39228. Revert "fumpt" This reverts commit 7f04bb3. Revert "use error code vars in tests" This reverts commit c6ae704. remove addtl err code var in test Signed-off-by: Jake Engelberg <[email protected]> * revise "key" field in metric to "category" Signed-off-by: Jake Engelberg <[email protected]> * restore original error codes + bug fixes Signed-off-by: Jake Engelberg <[email protected]> * fix configuration test Signed-off-by: Jake Engelberg <[email protected]> fix copyright headers Signed-off-by: Jake Engelberg <[email protected]> utilize todo context w/o struct member Signed-off-by: Jake Engelberg <[email protected]> nelson style improvements Signed-off-by: Jake Engelberg <[email protected]> * fix: remove doubled calls of RecordAndGet() Signed-off-by: Jake Engelberg <[email protected]> * reduce repitition in pubsub.go by recording at parent build() Signed-off-by: Jake Engelberg <[email protected]> * add specific recorder logic for Jobs API composite err codes Signed-off-by: Jake Engelberg <[email protected]> * refactor metric RecordX() funcs to reduce repetition Signed-off-by: Jake Engelberg <[email protected]> * lint fix Signed-off-by: Jake Engelberg <[email protected]> * metric naming: count -> total Signed-off-by: Jake Engelberg <[email protected]> * fix: style/naming Signed-off-by: Jake Engelberg <[email protected]> * fix: correct crypto metric recording w/ wrapper funcs Signed-off-by: Jake Engelberg <[email protected]> * fix: remove extra metric recordings Signed-off-by: Jake Engelberg <[email protected]> * integtest: error code metrics Signed-off-by: Jake Engelberg <[email protected]> * fix duplicated/missing http recordings Signed-off-by: Jake Engelberg <[email protected]> * clarify/further consolidate recording funcs Signed-off-by: Jake Engelberg <[email protected]> * further reduction of recording, requires dapr/kit PR Signed-off-by: Jake Engelberg <[email protected]> * Add metric spec plural copy, remove some comments Signed-off-by: Jake Engelberg <[email protected]> * nit improvements Signed-off-by: Jake Engelberg <[email protected]> * provide newest dapr/kit to properly record error code Signed-off-by: Jake Engelberg <[email protected]> * remove dapr/kit replace, add log for malformed error code Signed-off-by: Jake Engelberg <[email protected]> * log: error -> warn Signed-off-by: Jake Engelberg <[email protected]> --------- Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]> Co-authored-by: Cassandra Coyle <[email protected]> Co-authored-by: Elena Kolevska <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Artur Souza <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg force-pushed the metric-error-codes-impl branch from 9d751be to 6f94db0 Compare October 31, 2024 20:10

cicoyle and others added 20 commits October 31, 2024 16:11

further fine tuning

18b5afe

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

try fixing borked e2e

067b799

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

try fixing standalone validation

5a03bb0

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

install numpy too

b497232

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

update pip & install pkgs globally instead of venv

26dccd8

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

install requests

648ea13

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

// -> # for commented line

d815142

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

match only installed version of powershell

20f247d

Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Jake Engelberg <[email protected]>

wip: errorcode const consolidation

b486692

Signed-off-by: Jake Engelberg <[email protected]>

wip: error code monitoring/metric recording

94dabf8

Signed-off-by: Jake Engelberg <[email protected]>

style: error_code -> errorcode

af9d3b4

Signed-off-by: Jake Engelberg <[email protected]>

fix: error changes from refactor

31f845b

Signed-off-by: Jake Engelberg <[email protected]>

remove 2 unused error codes

c548580

Signed-off-by: Jake Engelberg <[email protected]>

Revert "wip: error code monitoring/metric recording"

9dc97d7

This reverts commit 53a02b5. Done so to implement in a separate PR. Signed-off-by: Jake Engelberg <[email protected]>

Revert "Revert "wip: error code monitoring/metric recording""

80d748c

This reverts commit 44c31be. Signed-off-by: Jake Engelberg <[email protected]>

use RecordAndGet() to record error code metric only whenever ApiError…

683c8f7

… var will be logged Signed-off-by: Jake Engelberg <[email protected]>

undo invalid error code recording

1942b75

Signed-off-by: Jake Engelberg <[email protected]>

apply cohesive prefixes to error codes

24391c1

Signed-off-by: Jake Engelberg <[email protected]>

correctly apply metric recording to inline string error definitions, …

23b0f4c

…fix unit tests Signed-off-by: Jake Engelberg <[email protected]>

add explicit "type" field for error code metrics

4d78709

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg force-pushed the metric-error-codes-impl branch from 6f94db0 to 4d78709 Compare October 31, 2024 20:12

jake-engelberg added 2 commits October 31, 2024 16:23

linting improvements

babd9af

Signed-off-by: Jake Engelberg <[email protected]>

errorcode recording unit test, fix: register type field on init

5288f83

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg marked this pull request as ready for review November 1, 2024 19:56

jake-engelberg requested review from a team as code owners November 1, 2024 19:56

filintod reviewed Nov 1, 2024

View reviewed changes

Comment thread tests/e2e/healthcheck/healthcheck_test.go Outdated

jake-engelberg added 2 commits November 3, 2024 21:44

add category to error codes for O(1) operation on recording

fb61001

Signed-off-by: Jake Engelberg <[email protected]>

use error code vars in tests

c6ae704

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg force-pushed the metric-error-codes-impl branch 2 times, most recently from b5ac1a4 to fb33a61 Compare November 22, 2024 00:16

further reduction of recording, requires dapr/kit PR

2cc84c5

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg force-pushed the metric-error-codes-impl branch from fb33a61 to 2cc84c5 Compare November 22, 2024 01:21

Add metric spec plural copy, remove some comments

fb1984a

Signed-off-by: Jake Engelberg <[email protected]>

elena-kolevska suggested changes Nov 26, 2024

View reviewed changes

nit improvements

f691d69

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg force-pushed the metric-error-codes-impl branch from 787df94 to f691d69 Compare November 26, 2024 18:22

provide newest dapr/kit to properly record error code

647fb64

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg force-pushed the metric-error-codes-impl branch from 1568630 to 647fb64 Compare November 27, 2024 03:05

Merge branch 'master' into metric-error-codes-impl

b2c0b65

elena-kolevska previously approved these changes Nov 27, 2024

View reviewed changes

jake-engelberg dismissed elena-kolevska’s stale review via 5eaf491 November 27, 2024 17:05

remove dapr/kit replace, add log for malformed error code

a059c03

Signed-off-by: Jake Engelberg <[email protected]>

jake-engelberg force-pushed the metric-error-codes-impl branch from 5eaf491 to a059c03 Compare November 27, 2024 17:06

jake-engelberg and others added 2 commits November 27, 2024 12:06

Merge branch 'master' into metric-error-codes-impl

c0adb93

log: error -> warn

115bb8b

Signed-off-by: Jake Engelberg <[email protected]>

elena-kolevska approved these changes Nov 27, 2024

View reviewed changes

Merge branch 'master' into metric-error-codes-impl

cc2dc2a

jake-engelberg requested a review from JoshVanL November 27, 2024 18:53

Merge branch 'master' into metric-error-codes-impl

81f1732

artursouza approved these changes Nov 27, 2024

View reviewed changes

artursouza merged commit fdd642e into dapr:master Nov 27, 2024

jake-engelberg mentioned this pull request Dec 11, 2024

Error Code Metrics Improved Testing+Recording #8345

Merged

7 tasks

jake-engelberg mentioned this pull request Jan 27, 2026

REQUEST: New membership for jake-engelberg dapr/community#697

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Metric Error Codes Recording + Error Code Prefixing#8256

Metric Error Codes Recording + Error Code Prefixing#8256
artursouza merged 61 commits into
dapr:masterfrom
jake-engelberg:metric-error-codes-impl

jake-engelberg commented Oct 30, 2024 •

edited

Loading

Uh oh!

Uh oh!

jake-engelberg commented Nov 22, 2024

Uh oh!

elena-kolevska left a comment

Uh oh!

Uh oh!

Uh oh!

elena-kolevska Nov 26, 2024

Uh oh!

elena-kolevska Nov 26, 2024

Uh oh!

Uh oh!

Uh oh!

elena-kolevska commented Nov 27, 2024

Uh oh!

marcduiker commented Dec 3, 2024

Uh oh!

holopin-bot Bot commented Dec 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Uh oh!

Conversation

jake-engelberg commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue reference

Checklist

Uh oh!

Uh oh!

jake-engelberg commented Nov 22, 2024

Uh oh!

elena-kolevska left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elena-kolevska Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

elena-kolevska Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elena-kolevska commented Nov 27, 2024

Uh oh!

marcduiker commented Dec 3, 2024

Uh oh!

holopin-bot Bot commented Dec 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

jake-engelberg commented Oct 30, 2024 •

edited

Loading