Skip to content

docs: enhance Proxy Metrics documentation. #5568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
arkodg merged 9 commits into from
Mar 27, 2025
+202 −18
Merged

docs: enhance Proxy Metrics documentation. #5568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c94aa89
docs: enhance Proxy Metrics documentation.
sudiptob2 Mar 21, 2025
7c237c4
docs: update wording.
sudiptob2 Mar 21, 2025
fbea98d
docs: Viewing otel metrics in pod logs.
sudiptob2 Mar 26, 2025
85ba303
docs: use boilerplate
sudiptob2 Mar 26, 2025
f42b6ec
docs: enable debug exporter in otel section.
sudiptob2 Mar 27, 2025
7c27415
docs: Add next steps.
sudiptob2 Mar 27, 2025
5c9cc51
docs: Use tab
sudiptob2 Mar 27, 2025
fcebab3
docs: Link Envoyproxy with Gateway instead of GatewayClass.
sudiptob2 Mar 27, 2025
d0840b2
docs: Update wording.
sudiptob2 Mar 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions site/content/en/latest/boilerplates/o11y_prerequisites.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,25 @@
---
---

### Install Envoy Gateway

{{< boilerplate prerequisites >}}

Envoy Gateway provides an add-ons Helm Chart, which includes all the needing components for observability.
By default, the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) is disabled.
### Install Add-ons

Envoy Gateway provides an add-ons Helm chart to simplify the installation of observability components.
The documentation for the add-ons chart can be found
[here](https://gateway.envoyproxy.io/docs/install/gateway-addons-helm-api/).

Install the add-ons Helm Chart:
Follow the instructions below to install the add-ons Helm chart.

```shell
helm install eg-addons oci://docker.io/envoyproxy/gateway-addons-helm --version {{< helm-version >}} --set opentelemetry-collector.enabled=true -n monitoring --create-namespace
helm install eg-addons oci://docker.io/envoyproxy/gateway-addons-helm --version {{< helm-version >}} -n monitoring --create-namespace
```

By default, the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) is **disabled.**
To install add-ons with OpenTelemetry Collector enabled, use the following command.

```shell
helm install eg-addons oci://docker.io/envoyproxy/gateway-addons-helm --version {{< helm-version >}} --set opentelemetry-collector.enabled=true -n monitoring --create-namespace
```
200 changes: 186 additions & 14 deletions site/content/en/latest/tasks/observability/proxy-metric.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,46 +2,218 @@
title: "Proxy Metrics"
---

Envoy Gateway provides observability for the ControlPlane and the underlying EnvoyProxy instances.
This task show you how to config proxy metrics.
Envoy Gateway offers observability for both the Control Plane and the underlying Envoy Proxy instances.
This task shows you how to configure proxy metrics.

## Prerequisites

{{< boilerplate o11y_prerequisites >}}

## Metrics

By default, Envoy Gateway expose metrics with prometheus endpoint.
### Prometheus Metrics

Verify metrics:
To query metrics using Prometheus API, follow the steps below.

```shell
export PROMETHEUS_PORT=$(kubectl get service prometheus -n monitoring -o jsonpath='{.spec.ports[0].port}')
kubectl port-forward service/prometheus -n monitoring 19001:$PROMETHEUS_PORT
```

Query metrics using Prometheus API:

```shell
curl -s 'http://localhost:19001/api/v1/query?query=topk(1,envoy_cluster_upstream_cx_connect_ms_sum)' | jq .
```

To directly view the metrics in Prometheus format from the Envoy's `/stats/prometheus`
[admin endpoint](https://www.envoyproxy.io/docs/envoy/latest/operations/admin), follow the steps below.

```shell
export ENVOY_POD_NAME=$(kubectl get pod -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward pod/$ENVOY_POD_NAME -n envoy-gateway-system 19001:19001
```

View the metrics:

# check metrics
```shell
curl localhost:19001/stats/prometheus | grep "default/backend/rule/0"
```

You can disable metrics by setting the `telemetry.metrics.prometheus.disable` to `true` in the `EnvoyProxy` CRD.
If you are only using the OpenTelemetry sink, you might want to set the `telemetry.metrics.prometheus.disable` to `true`
in the _EnvoyProxy CRD_ as shown in the following command.

{{< tabpane text=true >}}
{{% tab header="Apply from stdin" %}}
```shell
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/metric/disable-prometheus.yaml
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: envoy-gateway-system
spec:
gatewayClassName: eg
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: prometheus
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: prometheus
namespace: envoy-gateway-system
spec:
telemetry:
metrics:
prometheus:
disable: true
EOF
```
{{% /tab %}}
{{% tab header="Apply from file" %}}
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: envoy-gateway-system
spec:
gatewayClassName: eg
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: prometheus
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: prometheus
namespace: envoy-gateway-system
spec:
telemetry:
metrics:
prometheus:
disable: true
```
{{% /tab %}}
{{< /tabpane >}}

Envoy Gateway can send metrics to OpenTelemetry Sink.
Send metrics to OTel-Collector:

To completely remove Prometheus resources from the cluster, set the `prometheus.enabled` Helm value to `false`.

```shell
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/metric/otel-sink.yaml
helm upgrade eg-addons oci://docker.io/envoyproxy/gateway-addons-helm --version {{< helm-version >}} -n monitoring --reuse-values --set prometheus.enabled=false
```

Verify OTel-Collector metrics:
### OpenTelemetry Metrics

Envoy Gateway can export metrics to an OpenTelemetry sink. Use the following command to send metrics to the
OpenTelemetry Collector. Ensure that the OpenTelemetry components are enabled,
as mentioned in the [Prerequisites](#prerequisites).

{{< tabpane text=true >}}
{{% tab header="Apply from stdin" %}}
```shell
cat <<EOF | kubectl apply -f -
Copy link
Copy Markdown
Contributor

@arkodg arkodg Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use tabs here ?

gateway/site/content/en/v1.3/tasks/traffic/direct-response.md

Line 17 in 953ccc1

{{% tab header="Apply from stdin" %}}

Copy link
Copy Markdown
Member Author

@sudiptob2 sudiptob2 Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated with a similar approach now.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: envoy-gateway-system
spec:
gatewayClassName: eg
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: otel-sink
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: otel-sink
namespace: envoy-gateway-system
spec:
telemetry:
metrics:
sinks:
- type: OpenTelemetry
openTelemetry:
host: otel-collector.monitoring.svc.cluster.local
port: 4317
EOF
```
{{% /tab %}}
{{% tab header="Apply from file" %}}
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: envoy-gateway-system
spec:
gatewayClassName: eg
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: otel-sink
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: otel-sink
namespace: envoy-gateway-system
spec:
telemetry:
metrics:
sinks:
- type: OpenTelemetry
openTelemetry:
host: otel-collector.monitoring.svc.cluster.local
port: 4317
```
{{% /tab %}}
{{< /tabpane >}}


Temporarily enable the `debug` exporter in the OpenTelemetry Collector
to view metrics in the pod logs using the following command. Debug exporter is enabled for demonstration purposes and
should not be used in production.

```shell
helm upgrade eg-addons oci://docker.io/envoyproxy/gateway-addons-helm --version {{< helm-version >}} -n monitoring --reuse-values --set opentelemetry-collector.config.service.pipelines.metrics.exporters='{debug,prometheus}'

```

To view the logs of the OpenTelemetry Collector, use the following command:

```shell
export OTEL_POD_NAME=$(kubectl get pod -n monitoring --selector=app.kubernetes.io/name=opentelemetry-collector -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward pod/$OTEL_POD_NAME -n monitoring 19001:19001
kubectl logs -n monitoring -f $OTEL_POD_NAME --tail=100

# check metrics
curl localhost:19001/metrics | grep "default/backend/rule/0"
```

## Next Steps

Check out the [Visualising metrics using Grafana](./grafana-integration.md) section to learn more about how you can observe all the metrics in one place.