feat(datadog): Improve Datadog plugin tag support#11943
Conversation
826b279 to
7767c7d
Compare
|
Hi @deiwin, I see PR is still in draft status, is there still time to advance? |
|
Hi! I haven't been able to work on this recently, but would love to see this completed. Based on how I read the code, the code changes should work as-is, but the docs & tests still need work. Unfortunately I wasn't able to get the tests to run on mac (following this guide), so I put this on hold for now. |
Can you share the specific problem? |
|
Following the linked guide, everything seems to work up to and including the |
|
I actually opened this PR hoping that I could use the CI system to validate my changes, but unfortunately it looks like not all PRs are automatically validated by CI. |
|
Hi @deiwin, the running Docker container may have limited resources, causing the process to execute slowly and fail to complete within the time expected by the test framework. You can try to modify the timeout parameters of the test framework to give the process more time to complete the operation. You can check the configuration of the Test::Nginx::Socket module to see if the timeout parameter can be adjusted. |
|
Any specific conf changes (file & parameter) you could recommend? My Docker should have 8 (M1) cores & 15.5GB of memory available, so it shouldn't be too constrained. |
For example timeout in |
7767c7d to
216c7b6
Compare
|
Looks like that worked and I am now able to reliably run at least the relevant Datadog plugin tests! 🎉 Thank you, @Baoyuantop! 🙇 I've finished the changes now and updated the PR with them. I'll now open it for review. |
216c7b6 to
da581f7
Compare
|
I fixed the lint issue (I hope) but the other CI errors seemed unrelated to my changes. |
9b34b89 to
ab93afb
Compare
|
I realized there's one more change required for what I'm trying to achieve, so I added another commit to also add a |
ab93afb to
5120f05
Compare
|
I fixed the lint issue with the too long line: https://github.com/apache/apisix/compare/ab93afb5125b041bbc90ffab1cd2a0ec1b8834af..5120f05bed6eef9248d0a8a8f0dccc59cafdedcd I also found a way to run the lint (or at least luacheck) locally in the container ( The other CI failures seem unrelated to my changes again. |
| core.table.insert(tags, "route_name:" .. entry.route_id) | ||
| end | ||
|
|
||
| if entry.path and entry.path ~= "" then |
There was a problem hiding this comment.
Can you use functions to reduce duplicate code?
There was a problem hiding this comment.
Did you have something like this in mind: 004b574?
| if entry.route_id and entry.route_id ~= "" then | ||
| core.table.insert(tags, "route_name:" .. entry.route_id) | ||
| if entry.constant_tags and #entry.constant_tags > 0 then | ||
| for _, tag in pairs(entry.constant_tags) do |
There was a problem hiding this comment.
should use ipairs to iterate over the array.
There was a problem hiding this comment.
004b574 to
3d61e59
Compare
Currently there is no way to distinguish Datadog metrics for different HTTP endpoints if these endpoints are served through a single Apisix route. With these changes, if `include_path` is set to true, the path pattern by which the HTTP request was matched to a route is included as a metric tag with the `path:` key. This allows different endpoints to be distinguished in metrics.
Currently there is no way to distinguish Datadog metrics for e.g. GET & POST requests for the same endpoint within a single Apisix route, although their performance characteristics are likely to be quite different. With these changes, if `include_method` is set to true, HTTP method is included as a metric tag with the `method:` key, enabling such requests to be differentiated.
`constant_tags` are already supported at the plugin configuration level. However, sometimes different values may be required for each route, but this was previously not possible. For example, if some routes are owned by team A and some routes by team B, they could add `constant_tags: ["owner:team_a"]` or `constant_tags: ["owner:team_b"]` to each route, and would then be able to group metrics by team on Datadog.
Having a separate tag for the HTTP response code class allows useful grouping in Datadog by e.g. successful requests (2xx), client errors (4xx), and server errors (5xx). The existing `response_status` already essentially includes that information, but because of how Datadog works, one would need to list out each possible value between 400 and 431 to group all client errors, for example. Having a separate tag for the class greatly simplifies this. Generally it may be problematic to add new tags by default without an opt-in configuration option, because Datadog charges based on [custom metrics count][1] and additional tags can increase that count. However, the additional tag is safe to add here, because it is guaranteed not to increase the custom metric count. That is because the new tag is based on the already included `response_status` value, which is more granular than the new `response_status_class` value, so the number of unique tag combinations will not change. [1]: https://docs.datadoghq.com/metrics/custom_metrics/
The constraints are based on [Datadog documentation][1]. This applies to both tags provided in plugin configuration and also for tags provided in route configuration. [1]: https://docs.datadoghq.com/getting_started/tagging/#define-tags
Use a table-based approach to simplify the code and reduce duplication.
3d61e59 to
cfbc307
Compare
|
I also rebased this onto latest |
|
Hi @deiwin, could you please add some descriptive information so that other maintainers can understand your intention and review this PR more effectively, such as the reason for adding this feature improvement and the application of these features in an actual business context? |
|
@Baoyuantop each commit already has detailed reasoning for the specific change within that commit. But I can try improving the overall PR description as well. |
|
Updated the PR description as well |
many thx for your nice explain, I totally understand why we need this PR |
Description
Currently the metrics reported by Apisix's Datadog plugin are not sufficient for our observability needs (which I believe are common) and we need to rely on service-reported metrics instead for most use-cases. This is problematic for us, because service-reported metrics are fundamentally unable to include some important information. For example: rate-limited response counts, gateway errors when the service is unreachable, and delays within the network between gateway and the service.
The changes included in this PR make the metrics reported by the plugin compatible with most common observability use-cases in a backwards-compatible and cost-conscious way. Each commit includes detailed reasoning within the commit message for the specific change but an overview of the changes is provided here below as well.
pathpattern in a tag allows metrics to be grouped by endpoint in Datadog dashboards and monitors.methodis required to distinguish different endpoints with the same path that use a different HTTP method. A GET and a PUT for the same path can have very different performance characteristics and failure reasons, so need to be distinguished.response_status_classtag is required to be able to create monitors that alert when the rate of user errors (4xx responses) is high, for example. Without such a tag, the monitor would have to explicitly group by every possible 4xx value (400, 401, 402, etc) and then add the values reported by such groups together, which is not practical in most cases.Checklist