Prerequisites

• I searched existing issues

Feature Summary

In the previous iteration of the XID detection using DCGM, we were using DGCM to query the GPU serial numbers and include it as a part of the health event metadata. In the current Syslog health monitor approach, we don't have access to DCGM, so we end up losing the GPU serial number.

Problem/Use Case

Tracking component serial number make querying health event data at a component level easier and it may also help with tack down faulty components that may be hot potatoing between clusters

Proposed Solution

• Query nvidia-smi to find the GPU PCI to UUID mapping on start up of syslog-health-monitor. Bonus points if the go-nvml library is used to avoid relying on the nvidia-smi binary being present
• Store the results of the query in a lookup table
• Lookup up the GPU information before a healthy or an unhealthy event is sent.

Component

Health Monitor