Skip to content

Commit 3977aef

Browse files
authored
feat(monitoring): add GPU observability dashboards and recording rules (#2418)
## What this PR does Adds GPU observability to Cozystack: five Grafana dashboards, a set of VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter on Talos when running GPU workloads directly in pods. ### Components 1. **Dashboards** — `dashboards/gpu/*.json`, registered in `packages/system/monitoring/dashboards-infra.list`. All follow the project convention: the Prometheus data source is selected through the `$ds_prometheus` template variable. - `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics engine, memory copy), VRAM, power, temperature, and health (XID errors, power/thermal throttling). All metrics are shown per-GPU, filterable by the `$Hostname` selector. - `gpu-fleet` — cluster-wide inventory, capacity, utilization, and per-node power draw as percentage of TDP. - `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh accounting; utilization and power are shown per-node (DCGM cannot attribute hardware metrics to namespaces). - `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a per-namespace requests/limits table. All counts derive from the `namespace:gpu_count:allocated` recording rule, which excludes `Failed`/`Succeeded` pod carryover. - `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation, util-per-watt, and per-GPU power/thermal throttle fractions. 2. **Recording rules and alert** — `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`. Four rule groups: - `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview panels. - `gpu.recording.node.1m` — per-node hardware aggregates (grouped by `Hostname`) plus per-namespace allocation counts from kube resource requests; filters `cozy-*`/`kube-*`. - `gpu.recording.efficiency.1m` — per-GPU tensor saturation and util-per-watt, plus per-GPU power/thermal throttle fractions bounded via `clamp_max(..., 1)`. - `gpu.recording.throttle.validation.5m` — houses the `GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp throttle fraction exceeds 1.0. This guards against DCGM counter-unit drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10 with DCGM 3.x. Rules are safe on clusters without DCGM — they evaluate to empty series when no matching metrics are scraped. 3. **Reference manifests** under `packages/system/gpu-operator/examples/`. These are **not** templates; they document one working configuration: - `values-native-talos.yaml` — Cozystack Package values for the native-pod scenario. - `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with profiling, ECC, throttling, and energy counters used by the dashboards and recording rules. - `nvidia-driver-compat.yaml` — DaemonSet that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a location the GPU Operator validator inspects. Workaround for [NVIDIA/gpu-operator#1687](NVIDIA/gpu-operator#1687). Requires `pod-security.kubernetes.io/enforce: privileged` on the target namespace when the cluster enforces baseline/restricted PodSecurityStandards. - `README.md` — explains the two deployment paths (sandbox vs. native), which DCGM metrics each dashboard depends on, and how the pieces fit together. 4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats` enforces that every recording-rule reference in tracked dashboards resolves, and that every DCGM metric referenced by the rules is declared in either the upstream default CSV snapshot (`hack/dcgm-default-counters.csv`) or the custom CSV shipped under `examples/`. ### Why reference manifests, not templates The out-of-the-box `packages/system/gpu-operator/values-talos.yaml` targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via VFIO, driver disabled on the host, device plugin disabled. That is the sensible default for Cozystack today. Running GPU workloads directly in pods on Talos is also possible but requires a different configuration that makes assumptions not every user shares: - the NVIDIA Talos system extension must be installed on GPU nodes, - the operator's own driver and toolkit components must be disabled because the extension provides them, - the validator's hardcoded paths need a workaround until NVIDIA/gpu-operator#1687 lands upstream. Shipping those as active templates would silently impose those assumptions. Shipping them as reference files next to the package lets operators opt in with eyes open and understand the moving parts before applying them. ### Scope note The dashboards and recording rules are useful independently of the reference manifests — any cluster that scrapes DCGM Exporter (however it is installed) will populate them. The reference manifests are self-contained and do not change any default behavior of the `gpu-operator` package. ### Screenshots <img width="2824" height="1055" alt="2026-04-27_12-36-52" src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a" /> <img width="2913" height="1255" alt="2026-04-27_12-36-39" src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229" /> <img width="2932" height="1251" alt="2026-04-27_12-36-30" src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23" /> <img width="2864" height="1222" alt="2026-04-27_12-36-00" src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9" /> <img width="2958" height="1226" alt="image" src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76" /> ### Release note ```release-note feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added five GPU dashboards for performance, efficiency, fleet overview, quotas/allocation, and tenant usage (utilization, power, throttling, allocation, billing). * Added cluster recording rules and a throttling validation alert; new Prometheus-derived metrics and per-GPU rankings/timeseries. * **Documentation** * Added GPU Operator examples and README documenting native Talos setup and required DCGM counters. * **Tests** * Added validation tests ensuring dashboard ↔ recording-rule and DCGM metric consistency. * **Chores** * Registered new GPU dashboards in infra listings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents a961a90 + b2a8cca commit 3977aef

13 files changed

Lines changed: 5525 additions & 0 deletions

‎dashboards/gpu/gpu-efficiency.json‎

Lines changed: 819 additions & 0 deletions
Large diffs are not rendered by default.

‎dashboards/gpu/gpu-fleet.json‎

Lines changed: 1157 additions & 0 deletions
Large diffs are not rendered by default.

‎dashboards/gpu/gpu-performance.json‎

Lines changed: 957 additions & 0 deletions
Large diffs are not rendered by default.

‎dashboards/gpu/gpu-quotas.json‎

Lines changed: 637 additions & 0 deletions
Large diffs are not rendered by default.

‎dashboards/gpu/gpu-tenants.json‎

Lines changed: 1098 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Cross-validation between GPU recording rules, the dashboards that consume
4+
# them, and the DCGM Exporter metric set the cluster actually scrapes. Catches:
5+
#
6+
# 1. dangling references — a dashboard query mentions a recording rule name
7+
# that doesn't exist in gpu-recording.rules.yaml. This is the bug the
8+
# pre-merge review caught: gpu-efficiency.json shipped panels keyed on
9+
# pod:tensor_saturation:avg5m without the rule being defined, so the
10+
# panel showed "No data" everywhere.
11+
#
12+
# 2. typos in rule names — same bug class, manifested as a single-character
13+
# difference between rule and reference.
14+
#
15+
# 3. undeclared DCGM metrics — a dashboard query or recording rule mentions
16+
# a DCGM_FI_* metric that is neither in the upstream default CSV nor in
17+
# the project's custom CSV (dcgm-custom-metrics.yaml), meaning DCGM
18+
# Exporter would never emit it and the panel silently shows "No data".
19+
# Example regression: gpu-fleet.json shipped a TDP panel referencing
20+
# DCGM_FI_DEV_POWER_MGMT_LIMIT before the custom CSV declared it.
21+
#
22+
# Scope: only dashboards listed in packages/system/monitoring/dashboards-infra.list
23+
# under the "gpu/" prefix (i.e. shipped to production), not every JSON file in
24+
# dashboards/gpu/. Untracked drafts stay out of scope on purpose — adding one
25+
# to dashboards-infra.list is what brings it under the test.
26+
#
27+
# Reverse direction (rule defined but never consumed) is intentionally NOT
28+
# enforced: some rules exist for ad-hoc PromQL or upcoming dashboards. Treat
29+
# unused rules as an editorial concern, not a regression.
30+
#
31+
# Title syntax constraints from cozytest.sh's awk parser:
32+
# - Titles delimited by ASCII double quotes; embedded quotes truncate.
33+
# - Only [A-Za-z0-9] from the title survives into the function name; titles
34+
# differing only in punctuation collapse to the same function.
35+
#
36+
# Run with: hack/cozytest.sh hack/check-gpu-recording-rules.bats
37+
# -----------------------------------------------------------------------------
38+
39+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
40+
RULES_FILE="$REPO_ROOT/packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml"
41+
DASHBOARDS_LIST="$REPO_ROOT/packages/system/monitoring/dashboards-infra.list"
42+
DASHBOARDS_DIR="$REPO_ROOT/dashboards"
43+
DCGM_DEFAULT_CSV="$REPO_ROOT/hack/dcgm-default-counters.csv"
44+
DCGM_CUSTOM_CSV="$REPO_ROOT/packages/system/gpu-operator/examples/dcgm-custom-metrics.yaml"
45+
46+
# Extract the set of "- record: NAME" entries from the rules YAML.
47+
# Outputs one rule name per line, sorted and deduplicated.
48+
extract_rules() {
49+
awk '/^[[:space:]]*-[[:space:]]*record:[[:space:]]/ {
50+
sub(/^[[:space:]]*-[[:space:]]*record:[[:space:]]*/, "")
51+
sub(/[[:space:]]*$/, "")
52+
print
53+
}' "$RULES_FILE" | sort -u
54+
}
55+
56+
# Extract the set of recording-rule references from a dashboard JSON.
57+
# A recording-rule reference is matched by the pattern
58+
# <segment>:<segment>(:<segment>)+
59+
# where each <segment> is [a-z0-9_]. Raw DCGM metrics (DCGM_FI_*),
60+
# kube-state-metrics (kube_*) and similar uppercase / single-word metric
61+
# names do not match because the leading segment must be lowercase and the
62+
# whole expression must contain at least two ':' characters.
63+
extract_refs() {
64+
json_file=$1
65+
# Prometheus convention allows 2-segment rule names (level:metric); this
66+
# regex is tuned to the 3+ segment convention used in this repo
67+
# (level:metric:op — e.g. cluster:gpu_count:total). Update if future
68+
# rules use 2 segments, otherwise they will be silently skipped.
69+
grep -hoE '[a-z][a-z0-9_]*:[a-z0-9_]+:[a-z0-9_]+' "$json_file" | sort -u
70+
}
71+
72+
# Resolve "gpu/foo" -> "$DASHBOARDS_DIR/gpu/foo.json"
73+
list_tracked_gpu_dashboards() {
74+
awk '/^gpu\// { print $0 ".json" }' "$DASHBOARDS_LIST"
75+
}
76+
77+
# Extract the set of DCGM_FI_* metric names declared in a CSV file. Handles
78+
# both the upstream-style default CSV (unindented) and the ConfigMap-style
79+
# custom CSV (YAML-indented). A declaration line starts — after any leading
80+
# whitespace — with "DCGM_FI_<NAME>," ; comment lines begin with "#" and are
81+
# skipped. Uses POSIX awk's match()+RSTART/RLENGTH so no GNU extensions
82+
# are required.
83+
extract_csv_metrics() {
84+
file=$1
85+
awk '
86+
{
87+
line = $0
88+
sub(/^[[:space:]]+/, "", line)
89+
if (line ~ /^#/) next
90+
if (match(line, /^DCGM_FI_[A-Z0-9_]+/)) {
91+
print substr(line, RSTART, RLENGTH)
92+
}
93+
}
94+
' "$file" | sort -u
95+
}
96+
97+
# Extract the set of DCGM_FI_* metric references from a text file (dashboard
98+
# JSON or rules YAML). A DCGM metric name has at least two underscore-delimited
99+
# segments after the "DCGM_FI_" prefix (e.g. DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_
100+
# PIPE_TENSOR_ACTIVE, DCGM_FI_DRIVER_VERSION). Requiring two segments keeps
101+
# the matcher from latching onto glob stubs like "DCGM_FI_DEV_*_VIOLATION" that
102+
# appear in comments.
103+
extract_dcgm_refs() {
104+
file=$1
105+
grep -hoE 'DCGM_FI_[A-Z0-9]+(_[A-Z0-9]+)+' "$file" | sort -u
106+
}
107+
108+
@test "every recording rule reference in tracked GPU dashboards has a matching record" {
109+
TMP=$(mktemp -d)
110+
trap 'rm -rf "$TMP"' EXIT
111+
112+
extract_rules > "$TMP/rules.txt"
113+
[ -s "$TMP/rules.txt" ] || { echo "no recording rules extracted from $RULES_FILE" >&2; exit 1; }
114+
115+
list_tracked_gpu_dashboards > "$TMP/dashboards.txt"
116+
[ -s "$TMP/dashboards.txt" ] || { echo "no gpu/* dashboards listed in $DASHBOARDS_LIST" >&2; exit 1; }
117+
118+
failed=0
119+
while IFS= read -r dashboard_rel; do
120+
dashboard="$DASHBOARDS_DIR/$dashboard_rel"
121+
if [ ! -f "$dashboard" ]; then
122+
echo "ERROR: dashboard listed but file missing: $dashboard" >&2
123+
failed=1
124+
continue
125+
fi
126+
127+
extract_refs "$dashboard" > "$TMP/refs.txt"
128+
# comm -23: lines unique to refs.txt (referenced but not defined)
129+
# Both inputs must be sorted; extract_* helpers already sort.
130+
comm -23 "$TMP/refs.txt" "$TMP/rules.txt" > "$TMP/missing.txt"
131+
if [ -s "$TMP/missing.txt" ]; then
132+
echo "ERROR: $dashboard_rel references undefined recording rules:" >&2
133+
sed 's/^/ - /' "$TMP/missing.txt" >&2
134+
failed=1
135+
fi
136+
done < "$TMP/dashboards.txt"
137+
138+
[ "$failed" -eq 0 ]
139+
}
140+
141+
@test "every DCGM metric referenced in tracked dashboards and rules is declared" {
142+
TMP=$(mktemp -d)
143+
trap 'rm -rf "$TMP"' EXIT
144+
145+
[ -f "$DCGM_DEFAULT_CSV" ] || { echo "missing $DCGM_DEFAULT_CSV" >&2; exit 1; }
146+
[ -f "$DCGM_CUSTOM_CSV" ] || { echo "missing $DCGM_CUSTOM_CSV" >&2; exit 1; }
147+
148+
{
149+
extract_csv_metrics "$DCGM_DEFAULT_CSV"
150+
extract_csv_metrics "$DCGM_CUSTOM_CSV"
151+
} | sort -u > "$TMP/declared.txt"
152+
[ -s "$TMP/declared.txt" ] || { echo "no DCGM metrics extracted from CSVs" >&2; exit 1; }
153+
154+
list_tracked_gpu_dashboards > "$TMP/dashboards.txt"
155+
[ -s "$TMP/dashboards.txt" ] || { echo "no gpu/* dashboards listed in $DASHBOARDS_LIST" >&2; exit 1; }
156+
157+
failed=0
158+
159+
# Dashboard coverage — every dashboard's DCGM references must resolve.
160+
while IFS= read -r dashboard_rel; do
161+
dashboard="$DASHBOARDS_DIR/$dashboard_rel"
162+
[ -f "$dashboard" ] || continue # handled by the existence test
163+
extract_dcgm_refs "$dashboard" > "$TMP/refs.txt"
164+
[ -s "$TMP/refs.txt" ] || continue # dashboard relies entirely on recording rules
165+
comm -23 "$TMP/refs.txt" "$TMP/declared.txt" > "$TMP/missing.txt"
166+
if [ -s "$TMP/missing.txt" ]; then
167+
echo "ERROR: $dashboard_rel references DCGM metrics not declared in any CSV:" >&2
168+
sed 's/^/ - /' "$TMP/missing.txt" >&2
169+
failed=1
170+
fi
171+
done < "$TMP/dashboards.txt"
172+
173+
# Rules coverage — recording rules consume DCGM directly, so their set
174+
# must be declared too, otherwise derived series on every dashboard
175+
# collapse to empty.
176+
extract_dcgm_refs "$RULES_FILE" > "$TMP/rule-refs.txt"
177+
if [ -s "$TMP/rule-refs.txt" ]; then
178+
comm -23 "$TMP/rule-refs.txt" "$TMP/declared.txt" > "$TMP/rule-missing.txt"
179+
if [ -s "$TMP/rule-missing.txt" ]; then
180+
echo "ERROR: gpu-recording.rules.yaml references DCGM metrics not declared in any CSV:" >&2
181+
sed 's/^/ - /' "$TMP/rule-missing.txt" >&2
182+
failed=1
183+
fi
184+
fi
185+
186+
[ "$failed" -eq 0 ]
187+
}
188+
189+
@test "every tracked GPU dashboard listed in dashboards-infra.list exists on disk" {
190+
TMP=$(mktemp -d)
191+
trap 'rm -rf "$TMP"' EXIT
192+
193+
list_tracked_gpu_dashboards > "$TMP/dashboards.txt"
194+
[ -s "$TMP/dashboards.txt" ] || { echo "no gpu/* dashboards listed in $DASHBOARDS_LIST" >&2; exit 1; }
195+
196+
failed=0
197+
while IFS= read -r dashboard_rel; do
198+
dashboard="$DASHBOARDS_DIR/$dashboard_rel"
199+
if [ ! -f "$dashboard" ]; then
200+
echo "ERROR: $dashboard_rel listed in dashboards-infra.list but $dashboard does not exist" >&2
201+
failed=1
202+
fi
203+
done < "$TMP/dashboards.txt"
204+
205+
[ "$failed" -eq 0 ]
206+
}

‎hack/dcgm-default-counters.csv‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Snapshot of the upstream DCGM Exporter default-counters.csv used as a
2+
# fixture by hack/check-gpu-recording-rules.bats. The test verifies that
3+
# every DCGM_FI_* metric referenced by a tracked GPU dashboard is either
4+
# declared here (upstream defaults) or in
5+
# packages/system/gpu-operator/examples/dcgm-custom-metrics.yaml
6+
# (the project's custom CSV).
7+
#
8+
# Source: https://github.com/NVIDIA/dcgm-exporter/blob/4.1.1-4.0.4/etc/default-counters.csv
9+
# Pinned to the DCGM Exporter image tag shipped by the gpu-operator
10+
# chart under packages/system/gpu-operator/charts/gpu-operator/values.yaml
11+
# (dcgmExporter.version = 4.1.1-4.0.4-ubuntu22.04). When that image is
12+
# bumped, refresh this file from the matching tag in the NVIDIA/dcgm-exporter
13+
# repo and re-run `./hack/cozytest.sh hack/check-gpu-recording-rules.bats`.
14+
15+
# Format
16+
# If line starts with a '#' it is considered a comment
17+
# DCGM FIELD, Prometheus metric type, help message
18+
19+
# Clocks
20+
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
21+
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
22+
23+
# Temperature
24+
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
25+
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
26+
27+
# Power
28+
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).
29+
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).
30+
31+
# PCIE
32+
# DCGM_FI_PROF_PCIE_TX_BYTES, counter, Total number of bytes transmitted through PCIe TX via NVML.
33+
# DCGM_FI_PROF_PCIE_RX_BYTES, counter, Total number of bytes received through PCIe RX via NVML.
34+
DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total number of PCIe retries.
35+
36+
# Utilization (the sample period varies depending on the product)
37+
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
38+
DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
39+
DCGM_FI_DEV_ENC_UTIL, gauge, Encoder utilization (in %).
40+
DCGM_FI_DEV_DEC_UTIL , gauge, Decoder utilization (in %).
41+
42+
# Errors and violations
43+
DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.
44+
# DCGM_FI_DEV_POWER_VIOLATION, counter, Throttling duration due to power constraints (in us).
45+
# DCGM_FI_DEV_THERMAL_VIOLATION, counter, Throttling duration due to thermal constraints (in us).
46+
# DCGM_FI_DEV_SYNC_BOOST_VIOLATION, counter, Throttling duration due to sync-boost constraints (in us).
47+
# DCGM_FI_DEV_BOARD_LIMIT_VIOLATION, counter, Throttling duration due to board limit constraints (in us).
48+
# DCGM_FI_DEV_LOW_UTIL_VIOLATION, counter, Throttling duration due to low utilization (in us).
49+
# DCGM_FI_DEV_RELIABILITY_VIOLATION, counter, Throttling duration due to reliability constraints (in us).
50+
51+
# Memory usage
52+
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
53+
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
54+
55+
# ECC
56+
# DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, counter, Total number of single-bit volatile ECC errors.
57+
# DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, Total number of double-bit volatile ECC errors.
58+
# DCGM_FI_DEV_ECC_SBE_AGG_TOTAL, counter, Total number of single-bit persistent ECC errors.
59+
# DCGM_FI_DEV_ECC_DBE_AGG_TOTAL, counter, Total number of double-bit persistent ECC errors.
60+
61+
# Retired pages
62+
# DCGM_FI_DEV_RETIRED_SBE, counter, Total number of retired pages due to single-bit errors.
63+
# DCGM_FI_DEV_RETIRED_DBE, counter, Total number of retired pages due to double-bit errors.
64+
# DCGM_FI_DEV_RETIRED_PENDING, counter, Total number of pages pending retirement.
65+
66+
# NVLink
67+
# DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, Total number of NVLink flow-control CRC errors.
68+
# DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL, counter, Total number of NVLink data CRC errors.
69+
# DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL, counter, Total number of NVLink retries.
70+
# DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, counter, Total number of NVLink recovery errors.
71+
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, counter, Total number of NVLink bandwidth counters for all lanes.
72+
# DCGM_FI_DEV_NVLINK_BANDWIDTH_L0, counter, The number of bytes of active NVLink rx or tx data including both header and payload.
73+
74+
# VGPU License status
75+
DCGM_FI_DEV_VGPU_LICENSE_STATUS, gauge, vGPU License status
76+
77+
# Remapped rows
78+
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for uncorrectable errors
79+
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for correctable errors
80+
DCGM_FI_DEV_ROW_REMAP_FAILURE, gauge, Whether remapping of rows has failed
81+
82+
# Static configuration information. These appear as labels on the other metrics
83+
DCGM_FI_DRIVER_VERSION, label, Driver Version
84+
# DCGM_FI_NVML_VERSION, label, NVML Version
85+
# DCGM_FI_DEV_BRAND, label, Device Brand
86+
# DCGM_FI_DEV_SERIAL, label, Device Serial Number
87+
# DCGM_FI_DEV_OEM_INFOROM_VER, label, OEM inforom version
88+
# DCGM_FI_DEV_ECC_INFOROM_VER, label, ECC inforom version
89+
# DCGM_FI_DEV_POWER_INFOROM_VER, label, Power management object inforom version
90+
# DCGM_FI_DEV_INFOROM_IMAGE_VER, label, Inforom image version
91+
# DCGM_FI_DEV_VBIOS_VERSION, label, VBIOS version of the device
92+
93+
# Datacenter Profiling (DCP) metrics
94+
# NOTE: supported on Nvidia datacenter Volta GPUs and newer
95+
DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, Ratio of time the graphics engine is active.
96+
# DCGM_FI_PROF_SM_ACTIVE, gauge, The ratio of cycles an SM has at least 1 warp assigned.
97+
# DCGM_FI_PROF_SM_OCCUPANCY, gauge, The ratio of number of warps resident on an SM.
98+
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
99+
DCGM_FI_PROF_DRAM_ACTIVE, gauge, Ratio of cycles the device memory interface is active sending or receiving data.
100+
# DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, Ratio of cycles the fp64 pipes are active.
101+
# DCGM_FI_PROF_PIPE_FP32_ACTIVE, gauge, Ratio of cycles the fp32 pipes are active.
102+
# DCGM_FI_PROF_PIPE_FP16_ACTIVE, gauge, Ratio of cycles the fp16 pipes are active.
103+
DCGM_FI_PROF_PCIE_TX_BYTES, gauge, The rate of data transmitted over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
104+
DCGM_FI_PROF_PCIE_RX_BYTES, gauge, The rate of data received over the PCIe bus - including both protocol headers and data payloads - in bytes per second.

0 commit comments

Comments
 (0)