A lightweight C tool for reading GPU Performance Metrics (GPM) on NVIDIA GPUs, with support for MIG (Multi-Instance GPU) slices. Includes a Python wrapper for continuous monitoring and CSV export.
- NVIDIA GPU with GPM support (Hopper or newer: H100, GH200, etc.)
- NVIDIA driver 525.60.13 or newer
- CUDA Toolkit (for NVML headers)
- GCC compiler
- Python 3.6+ (only for monitoring wrapper)
git clone <repository>
cd gpm-metrics-readergcc -o gpm_metrics_reader gpm_metrics_reader.c -lnvidia-ml -I/usr/local/cuda/include -L/usr/local/cuda/lib64Run once to see current metrics:
./gpm_metrics_readerUse the Python wrapper for continuous sampling:
# Monitor continuously, save to CSV
python gpm_monitor.py --output metrics.csv
# Sample every 500ms
python gpm_monitor.py --output metrics.csv --interval 500
# Run for 60 seconds
python gpm_monitor.py --output metrics.csv --duration 60
# Custom binary path
python gpm_monitor.py --output metrics.csv --binary /path/to/gpm_metrics_readerArguments:
--output FILE(required): Output CSV file path--interval MSEC(default: 1000): Sampling interval in milliseconds--duration SEC(default: infinite): Total duration in seconds--binary PATH(default: ./gpm_metrics_reader): Path to binary
CSV Output Format:
timestamp,device_id,device_name,gpu_instance_id,compute_instance_id,metric_id,metric_name,value,unit
2025-11-21T10:30:45.123,0,NVIDIA GH200 480GB,,,1,GRAPHICS_UTIL,0.01,%
2025-11-21T10:30:45.123,0,NVIDIA GH200 480GB,,,2,SM_UTIL,0.00,%Edit the METRICS_TO_QUERY array in gpm_metrics_reader.c to select which metrics to monitor:
// List of metrics to query - add/remove them here
static const nvmlGpmMetricId_t METRICS_TO_QUERY[] = {
NVML_GPM_METRIC_SM_UTIL, // ID 2
NVML_GPM_METRIC_DRAM_BW_UTIL, // ID 10
NVML_GPM_METRIC_FP16_UTIL, // ID 13
};Recompile after making changes:
gcc -o gpm_metrics_reader gpm_metrics_reader.c -lnvidia-ml -I/usr/local/cuda/include -L/usr/local/cuda/lib64See NVIDIA NVML documentation for the complete list.