Repository navigation
docs: Add Hubble as observability source - #1782
Andrei Kvapil (kvaps) merged 1 commit into
Conversation
Add documentation explaining how to enable Hubble for network observability in Grafana. Include four pre-built Hubble dashboards (overview, dns-namespace, l7-http-metrics, network-overview) and register them in the monitoring hub's dashboard list. Closes #749 Signed-off-by: majiayu000 <[email protected]>
dosubot
Bot
added
size/XXL
Summary of ChangesHello lif (@majiayu000), I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances network observability by integrating Hubble with Grafana. It provides clear instructions for enabling Hubble within the Cilium setup and introduces a suite of specialized Grafana dashboards. These dashboards offer detailed insights into DNS queries, L7 HTTP traffic, and general network flow patterns, allowing users to monitor and troubleshoot network behavior more effectively. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
📝 WalkthroughWalkthroughThe pull request introduces Hubble as a network observability source for the Cozystack monitoring stack by adding three Grafana dashboard definitions for DNS, HTTP, and network metrics, along with implementation documentation and dashboard registry entries. Changes
Sequence Diagram(s)sequenceDiagram
actor User
participant Cozystack
participant Hubble as Hubble Component
participant Prometheus
participant Grafana
User->>Cozystack: Enable Hubble in Monitoring stack
activate Cozystack
Cozystack->>Hubble: Deploy & configure Hubble
activate Hubble
Hubble->>Hubble: Collect network metrics<br/>(flows, DNS, HTTP, policies)
deactivate Hubble
Hubble->>Prometheus: Expose metrics via endpoint
activate Prometheus
Prometheus->>Prometheus: Scrape Hubble metrics<br/>(at configured intervals)
deactivate Prometheus
deactivate Cozystack
User->>Grafana: Access Hubble dashboards
activate Grafana
Grafana->>Prometheus: Query metrics<br/>(DNS, HTTP, Network panels)
Prometheus-->>Grafana: Return time-series data
Grafana->>Grafana: Render panels &<br/>visualizations
Grafana-->>User: Display dashboards
deactivate Grafana
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
Pre-merge checks✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces Hubble observability by adding new documentation, four Grafana dashboards, and registering them. The changes are a great addition. I've found a few issues in the new dashboards, including incorrect queries, inconsistent units, and leftover demo tags. Specifically, one query for CPU usage by source uses the wrong namespace variable, which will lead to incorrect data. Several other panels are missing the method label in their by clauses, which will result in incorrect legends. I've also suggested updating a dashboard that uses a deprecated panel type for better maintainability. Please see my detailed comments for suggestions on how to fix these issues.
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=~\"${source_workload}\"}\n) by (namespace, workload)", |
There was a problem hiding this comment.
The PromQL query for the "CPU Usage by Source" panel incorrectly uses ${destination_namespace} to filter metrics, but the panel is intended to show data for the source. This will lead to incorrect data being displayed. You should use ${source_namespace} instead.
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=~\"${source_workload}\"}\n) by (namespace, workload)", | |
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${source_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${source_namespace}\", workload=~\"${source_workload}\"}\n) by (namespace, workload)", |
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "expr": "topk(10, sum(rate(hubble_dns_queries_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])*60) by (query))", |
There was a problem hiding this comment.
The expression for this panel multiplies the rate by 60, which likely converts it to requests per minute. However, the panel's unit is set to reqps (requests per second). This is inconsistent. To fix this, you should either change the unit to reqpm or remove the *60 from the expression to match the reqps unit. Given the unit, removing *60 is the more direct fix.
| "expr": "topk(10, sum(rate(hubble_dns_queries_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])*60) by (query))", | |
| "expr": "topk(10, sum(rate(hubble_dns_queries_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])) by (query))", |
| "schemaVersion": 38, | ||
| "style": "dark", | ||
| "tags": [ | ||
| "kubecon-demo" |
There was a problem hiding this comment.
The tag kubecon-demo seems to be a leftover from a demonstration. It should be replaced with a more generic and descriptive tag like hubble to better categorize the dashboard.
| "kubecon-demo" | |
| "hubble" |
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload, status), 0.001)", |
There was a problem hiding this comment.
The legendFormat for this panel includes {{ method }}, but the method label is missing from the by clause in the PromQL query. This will cause the literal string {{ method }} to be displayed in the legend. Please add method to the by clause to correctly display the HTTP method.
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload, status), 0.001)", | |
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, method, source_namespace, source_workload, status), 0.001)", |
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)", |
There was a problem hiding this comment.
The legendFormat for this panel includes {{ method }}, but the method label is missing from the by clause in the PromQL query. This will cause the literal string {{ method }} to be displayed in the legend. Please add method to the by clause in both parts of the query to correctly display the HTTP method.
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)", | |
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, method, source_namespace, source_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, method, source_namespace, source_workload)", |
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload, status), 0.001)", |
There was a problem hiding this comment.
The legendFormat for this panel includes {{ method }}, but the method label is missing from the by clause in the PromQL query. This will cause the literal string {{ method }} to be displayed in the legend. Please add method to the by clause to correctly display the HTTP method.
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload, status), 0.001)", | |
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, method, destination_namespace, destination_workload, status), 0.001)", |
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)", |
There was a problem hiding this comment.
The legendFormat for this panel includes {{ method }}, but the method label is missing from the by clause in the PromQL query. This will cause the literal string {{ method }} to be displayed in the legend. Please add method to the by clause in both parts of the query to correctly display the HTTP method.
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)", | |
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, method, destination_namespace, destination_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, method, destination_namespace, destination_workload)", |
| }, | ||
| "editorMode": "code", | ||
| "expr": "topk(10, sum(rate(hubble_flows_processed_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])) by (destination))", | ||
| "legendFormat": "{{source}}", |
There was a problem hiding this comment.
The query for this panel groups by destination, but the legendFormat is set to {{source}}. This will result in incorrect labels in the legend. It should be {{destination}}.
| "legendFormat": "{{source}}", | |
| "legendFormat": "{{destination}}", |
| "schemaVersion": 38, | ||
| "style": "dark", | ||
| "tags": [ | ||
| "kubecon-demo" |
There was a problem hiding this comment.
The tag kubecon-demo seems to be a leftover from a demonstration. It should be replaced with a more generic and descriptive tag like hubble to better categorize the dashboard.
| "kubecon-demo" | |
| "hubble" |
| clickhouse/altinity-clickhouse-operator-dashboard | ||
| storage/linstor | ||
| seaweedfs/seaweedfs | ||
| hubble/overview |
There was a problem hiding this comment.
The hubble/overview dashboard uses the legacy graph panel type, which is deprecated in newer Grafana versions. The other Hubble dashboards added in this PR use the modern timeseries panel type. For consistency and future maintainability, please update the hubble/overview dashboard to use timeseries panels instead of graph panels.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (1)
dashboards/hubble/network-overview.json (1)
801-826: Hardcoded pod name in override reduces portability.The override references a specific pod name
kube-dns-7d44cdb5d5-g85vgwhich is environment-specific and will not work correctly in other clusters. Consider removing this override or using a pattern-based approach if this was intended to filter out specific noise.
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
dashboards/hubble/dns-namespace.jsondashboards/hubble/l7-http-metrics.jsondashboards/hubble/network-overview.jsondashboards/hubble/overview.jsondocs/hubble-observability.mdpackages/extra/monitoring/dashboards.list
🧰 Additional context used
📓 Path-based instructions (1)
**/*.md
📄 CodeRabbit inference engine (AGENTS.md)
Git commits should follow
[component] Descriptionformat and include--signoffflag
Files:
docs/hubble-observability.md
🪛 LanguageTool
docs/hubble-observability.md
[style] ~3-~3: Consider a different adjective to strengthen your wording.
Context: ...orm built on top of Cilium. It provides deep visibility into the communication and b...
(DEEP_PROFOUND)
🔇 Additional comments (3)
dashboards/hubble/dns-namespace.json (1)
1-602: Dashboard looks well-structured.The DNS namespace dashboard is properly configured with consistent query expressions, appropriate templating variables, and correct panel configurations.
docs/hubble-observability.md (1)
1-99: Documentation is comprehensive and well-organized.The documentation provides clear guidance on prerequisites, configuration, accessing dashboards, available metrics, and troubleshooting steps. This will help users effectively enable and use Hubble for network observability.
packages/extra/monitoring/dashboards.list (1)
42-45: No action needed. Thedashboards/hubble/overview.jsonfile exists in the repository, and the entry in dashboards.list is correct.Likely an incorrect or invalid review comment.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "expr": "round(sum(rate(hubble_http_requests_total{reporter=~\"${reporter}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\"}[$__rate_interval])), 0.001)", | ||
| "refId": "A" | ||
| } | ||
| ], | ||
| "title": "Incoming Request Volume", | ||
| "type": "stat" |
There was a problem hiding this comment.
Missing cluster filter in query.
This panel's query does not include the cluster filter (cluster=~"${cluster}"), unlike other panels in this dashboard. This inconsistency may cause unexpected results when users select a specific cluster.
🔎 Proposed fix
- "expr": "round(sum(rate(hubble_http_requests_total{reporter=~\"${reporter}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\"}[$__rate_interval])), 0.001)",
+ "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", reporter=~\"${reporter}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\"}[$__rate_interval])), 0.001)",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "expr": "round(sum(rate(hubble_http_requests_total{reporter=~\"${reporter}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\"}[$__rate_interval])), 0.001)", | |
| "refId": "A" | |
| } | |
| ], | |
| "title": "Incoming Request Volume", | |
| "type": "stat" | |
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", reporter=~\"${reporter}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\"}[$__rate_interval])), 0.001)", | |
| "refId": "A" | |
| } | |
| ], | |
| "title": "Incoming Request Volume", | |
| "type": "stat" |
🤖 Prompt for AI Agents
In dashboards/hubble/l7-http-metrics.json around lines 131 to 142, the
Prometheus expression for the "Incoming Request Volume" panel is missing the
cluster label filter; update the metric selector to include
cluster=~"${cluster}" (i.e., add cluster=~"${cluster}" inside the {...} selector
alongside reporter, destination_namespace and destination_workload) so the query
is consistent with other panels and respects the dashboard's cluster variable.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload, status), 0.001)", | ||
| "interval": "", | ||
| "legendFormat": "{{ cluster }} {{ method }} {{ source_namespace }}/{{ source_workload }}: {{ status }}", | ||
| "range": true, | ||
| "refId": "A" | ||
| } |
There was a problem hiding this comment.
Legend format references non-existent labels.
The legendFormat includes {{ method }} but the query aggregates by cluster, source_namespace, source_workload, status only. The method label won't be available, resulting in empty values in the legend. Either add method to the by clause or remove it from the legend format.
🔎 Proposed fix (remove from legend)
- "legendFormat": "{{ cluster }} {{ method }} {{ source_namespace }}/{{ source_workload }}: {{ status }}",
+ "legendFormat": "{{ cluster }} {{ source_namespace }}/{{ source_workload }}: {{ status }}",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload, status), 0.001)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ method }} {{ source_namespace }}/{{ source_workload }}: {{ status }}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload, status), 0.001)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ source_namespace }}/{{ source_workload }}: {{ status }}", | |
| "range": true, | |
| "refId": "A" | |
| } |
🤖 Prompt for AI Agents
In dashboards/hubble/l7-http-metrics.json around lines 436 to 449, the
legendFormat references {{ method }} but the Prometheus expr groups by only
cluster, source_namespace, source_workload, and status so method will be
missing; fix by including method in the aggregation clause (add method to the
"by (...)" list in the expr) so the label exists for the legend, or
alternatively remove {{ method }} from legendFormat if you prefer not to group
by it.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)", | ||
| "interval": "", | ||
| "legendFormat": "{{ cluster }} {{ method }} {{ source_namespace }}/{{ source_workload }}", | ||
| "range": true, | ||
| "refId": "A" | ||
| } |
There was a problem hiding this comment.
Legend format references non-existent label.
Same issue: {{ method }} in the legend format but method is not in the aggregation. This will display empty values.
🔎 Proposed fix
- "legendFormat": "{{ cluster }} {{ method }} {{ source_namespace }}/{{ source_workload }}",
+ "legendFormat": "{{ cluster }} {{ source_namespace }}/{{ source_workload }}",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ method }} {{ source_namespace }}/{{ source_workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, source_namespace, source_workload)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ source_namespace }}/{{ source_workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } |
🤖 Prompt for AI Agents
In dashboards/hubble/l7-http-metrics.json around lines 537 to 550, the legend
uses {{ method }} but the PromQL aggregations do not include a method label so
it will render empty; fix by either removing {{ method }} from the legend or
adding method to the aggregation grouping on both numerator and denominator
(e.g., include method in the by(...) lists and in both sum(rate(...))
expressions) so the label exists and is consistent across the ratio.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=~\"${source_workload}\"}\n) by (namespace, workload)", | ||
| "interval": "", | ||
| "legendFormat": "{{ cluster }} {{ namespace }}/{{ workload }}", | ||
| "range": true, | ||
| "refId": "A" | ||
| } | ||
| ], |
There was a problem hiding this comment.
Legend format references non-existent label.
The legendFormat references {{ cluster }} but the query aggregates by namespace, workload only. The cluster label won't be available in the result.
🔎 Proposed fix
- "legendFormat": "{{ cluster }} {{ namespace }}/{{ workload }}",
+ "legendFormat": "{{ namespace }}/{{ workload }}",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=~\"${source_workload}\"}\n) by (namespace, workload)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ namespace }}/{{ workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| ], | |
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=~\"${source_workload}\"}\n) by (namespace, workload)", | |
| "interval": "", | |
| "legendFormat": "{{ namespace }}/{{ workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| ], |
🤖 Prompt for AI Agents
In dashboards/hubble/l7-http-metrics.json around lines 763 to 777, the
legendFormat references {{ cluster }} but the Prometheus query only aggregates
by namespace and workload so the cluster label will not be present; fix by
either adding cluster to the aggregation (e.g., include cluster in the by(...)
clause and any necessary grouping keys in the inner expressions) or remove {{
cluster }} from legendFormat (e.g., use "{{ namespace }}/{{ workload }}" or
another existing label) so the legend only references labels returned by the
query.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "round(sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload, status), 0.001)", | ||
| "interval": "", | ||
| "legendFormat": "{{ cluster }} {{ method }} {{ destination_namespace }}/{{ destination_workload }}: {{ status }}", | ||
| "range": true, | ||
| "refId": "A" | ||
| } | ||
| ], |
There was a problem hiding this comment.
Legend format references non-existent label.
The legendFormat includes {{ method }} but the aggregation is by cluster, destination_namespace, destination_workload, status — method is not included.
🔎 Proposed fix
- "legendFormat": "{{ cluster }} {{ method }} {{ destination_namespace }}/{{ destination_workload }}: {{ status }}",
+ "legendFormat": "{{ cluster }} {{ destination_namespace }}/{{ destination_workload }}: {{ status }}",🤖 Prompt for AI Agents
In dashboards/hubble/l7-http-metrics.json around lines 877 to 891, the legend
references {{ method }} but the Prometheus aggregation groups only by cluster,
destination_namespace, destination_workload, and status; either remove {{ method
}} from the legend or include method in the aggregation and query so the label
exists. Fix by updating the expr to aggregate by (cluster,
destination_namespace, destination_workload, status, method) if you want method
shown, or remove {{ method }} from legendFormat if method should not be
displayed; ensure the labels used in legendFormat exactly match the labels
present in the grouped query.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)", | ||
| "interval": "", | ||
| "legendFormat": "{{ cluster }} {{ method }} {{ destination_namespace }}/{{ destination_workload }}", | ||
| "range": true, | ||
| "refId": "A" | ||
| } |
There was a problem hiding this comment.
Legend format references non-existent label.
Same issue with {{ method }} not being in the aggregation.
🔎 Proposed fix
- "legendFormat": "{{ cluster }} {{ method }} {{ destination_namespace }}/{{ destination_workload }}",
+ "legendFormat": "{{ cluster }} {{ destination_namespace }}/{{ destination_workload }}",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ method }} {{ destination_namespace }}/{{ destination_workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\",status!~\"5.*\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)\n/\nsum(rate(hubble_http_requests_total{cluster=~\"${cluster}\", destination_namespace=~\"${destination_namespace}\", destination_workload=~\"${destination_workload}\", reporter=\"${reporter}\", source_namespace=~\"${source_namespace}\", source_workload=~\"${source_workload}\"}[$__rate_interval])) by (cluster, destination_namespace, destination_workload)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ destination_namespace }}/{{ destination_workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } |
🤖 Prompt for AI Agents
In dashboards/hubble/l7-http-metrics.json around lines 978-991 the legend
references {{ method }} but the PromQL aggregation doesn't include the method
label, so either remove {{ method }} from legend or add method to both
aggregations' by(...) clauses; to fix, update both sum(rate(...)[...])) by (...)
expressions to include method (e.g., by (cluster, destination_namespace,
destination_workload, method)) so the method label exists for the legend, or
alternatively delete {{ method }} from legendFormat if you prefer not to
aggregate by method.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "exemplar": false, | ||
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=\"${destination_workload}\"}\n) by (namespace, workload)", | ||
| "interval": "", | ||
| "legendFormat": "{{ cluster }} {{ namespace }}/{{ workload }}", | ||
| "range": true, | ||
| "refId": "A" | ||
| } | ||
| ], |
There was a problem hiding this comment.
Legend format references non-existent label.
The query aggregates by namespace, workload but the legend references {{ cluster }}.
🔎 Proposed fix
- "legendFormat": "{{ cluster }} {{ namespace }}/{{ workload }}",
+ "legendFormat": "{{ namespace }}/{{ workload }}",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=\"${destination_workload}\"}\n) by (namespace, workload)", | |
| "interval": "", | |
| "legendFormat": "{{ cluster }} {{ namespace }}/{{ workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| ], | |
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "exemplar": false, | |
| "expr": "sum(\n node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\"}\n * on(namespace,pod)\n group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{cluster=~\"${cluster}\", namespace=~\"${destination_namespace}\", workload=\"${destination_workload}\"}\n) by (namespace, workload)", | |
| "interval": "", | |
| "legendFormat": "{{ namespace }}/{{ workload }}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| ], |
🤖 Prompt for AI Agents
In dashboards/hubble/l7-http-metrics.json around lines 1202-1216 the Prometheus
query groups metrics by namespace and workload but the panel legend references
{{ cluster }} which is not included in the aggregation, causing an empty label;
fix by either adding cluster to the aggregation (e.g., include cluster in the
by(...) clause or ensure the joined metric exposes a cluster label) so the label
exists, or remove {{ cluster }} from legendFormat (e.g., change legendFormat to
"{{ namespace }}/{{ workload }}") depending on whether you want cluster shown.
| "targets": [ | ||
| { | ||
| "datasource": { | ||
| "type": "prometheus", | ||
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "expr": "topk(10, sum(rate(hubble_flows_processed_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])) by (destination))", | ||
| "legendFormat": "{{source}}", | ||
| "range": true, | ||
| "refId": "A" | ||
| } | ||
| ], | ||
| "title": "Top 10 destinations", | ||
| "type": "bargauge" |
There was a problem hiding this comment.
Legend format mismatch with aggregation.
The panel title is "Top 10 destinations" and aggregates by destination, but the legendFormat uses {{source}}. This appears to be a copy-paste error from the "Top 10 sources" panel.
🔎 Proposed fix
- "legendFormat": "{{source}}",
+ "legendFormat": "{{destination}}",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "expr": "topk(10, sum(rate(hubble_flows_processed_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])) by (destination))", | |
| "legendFormat": "{{source}}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| ], | |
| "title": "Top 10 destinations", | |
| "type": "bargauge" | |
| "targets": [ | |
| { | |
| "datasource": { | |
| "type": "prometheus", | |
| "uid": "${DS_PROMETHEUS}" | |
| }, | |
| "editorMode": "code", | |
| "expr": "topk(10, sum(rate(hubble_flows_processed_total{cluster=~\"$cluster\", source_namespace=~\"$source_namespace\", destination_namespace=~\"$destination_namespace\"}[$__rate_interval])) by (destination))", | |
| "legendFormat": "{{destination}}", | |
| "range": true, | |
| "refId": "A" | |
| } | |
| ], | |
| "title": "Top 10 destinations", | |
| "type": "bargauge" |
🤖 Prompt for AI Agents
In dashboards/hubble/network-overview.json around lines 410 to 424, the panel
aggregates by destination (title "Top 10 destinations") but the legendFormat is
incorrectly set to "{{source}}"; change the legendFormat to "{{destination}}" so
the displayed legend matches the aggregation, ensuring the label reflects the
destination dimension used in the query.
| "title": "Missing ICMP Echo Replys", | ||
| "type": "timeseries" |
There was a problem hiding this comment.
Typo in panel title.
"Replys" should be "Replies".
🔎 Proposed fix
- "title": "Missing ICMP Echo Replys",
+ "title": "Missing ICMP Echo Replies",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "title": "Missing ICMP Echo Replys", | |
| "type": "timeseries" | |
| "title": "Missing ICMP Echo Replies", | |
| "type": "timeseries" |
🤖 Prompt for AI Agents
In dashboards/hubble/network-overview.json around lines 631-632, the panel title
contains a typo: "Missing ICMP Echo Replys" should be corrected to "Missing ICMP
Echo Replies"; update the JSON title string accordingly to use the correct
spelling "Replies" and ensure the file remains valid JSON after the change.
Andrei Kvapil (kvaps)
left a comment
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
LGTM, thank you!
Fixes #749
Changes
Summary by CodeRabbit
New Features
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.