Hi, thanks for the tool!
We used the configuration file to enable the collection of some extra fields.
When we perform a curl request to the prometheus endpoint, however, we are missing the following enabled fields:
- DCGM_FI_DEV_MEMORY_TEMP
- DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS
- DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS
- DCGM_FI_DEV_ROW_REMAP_FAILURE
- DCGM_FI_PROF_PCIE_TX_BYTES
- DCGM_FI_PROF_PCIE_RX_BYTES
- DCGM_FI_DEV_FAN_SPEED
After some digging we made the following observations:
- The exporter package has generic filtering for null values. This suggests that the missing fields were not available from the underlying drivers.
- For DCGM_FI_DEV_FAN_SPEED we couldn't find any fan sensors. I guess that explains why it's missing.
- Fields related to remapping may have been missing because no remapping had been triggered.
- We still don't know why the PCIE fields are missing.
Here's our question:
How do you recommend that dcgm-exporter users troubleshoot missing fields? Is there some diagnostic/debugging tool we can use instead of hunting down each field (group) differently?
Thanks in advance!
Hi, thanks for the tool!
We used the configuration file to enable the collection of some extra fields.
When we perform a curl request to the prometheus endpoint, however, we are missing the following enabled fields:
After some digging we made the following observations:
Here's our question:
How do you recommend that dcgm-exporter users troubleshoot missing fields? Is there some diagnostic/debugging tool we can use instead of hunting down each field (group) differently?
Thanks in advance!