Skip to content

How to troubleshoot missing fields? #163

Description

@iliakur

Hi, thanks for the tool!

We used the configuration file to enable the collection of some extra fields.

When we perform a curl request to the prometheus endpoint, however, we are missing the following enabled fields:

  • DCGM_FI_DEV_MEMORY_TEMP
  • DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS
  • DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS
  • DCGM_FI_DEV_ROW_REMAP_FAILURE
  • DCGM_FI_PROF_PCIE_TX_BYTES
  • DCGM_FI_PROF_PCIE_RX_BYTES
  • DCGM_FI_DEV_FAN_SPEED

After some digging we made the following observations:

  1. The exporter package has generic filtering for null values. This suggests that the missing fields were not available from the underlying drivers.
  2. For DCGM_FI_DEV_FAN_SPEED we couldn't find any fan sensors. I guess that explains why it's missing.
  3. Fields related to remapping may have been missing because no remapping had been triggered.
  4. We still don't know why the PCIE fields are missing.

Here's our question:
How do you recommend that dcgm-exporter users troubleshoot missing fields? Is there some diagnostic/debugging tool we can use instead of hunting down each field (group) differently?

Thanks in advance!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions