Context
In response to @lalitadithya ’s question raised in topograph#179 earlier, https://github.com/NVIDIA/topograph/blob/main/docs/reference/node-labels.md has been published NVIDIA/topograph publishes four canonical node labels describing physical network topology:
network.topology.nvidia.com/accelerator — NVLink domain (clique) IDnetwork.topology.nvidia.com/leaf — leaf switch identifiernetwork.topology.nvidia.com/spine — spine switch identifiernetwork.topology.nvidia.com/core — core switch identifier
These labels can be consumed today by KAI Scheduler, Grove, Kueue & similar for topology-aware placement. On clusters where Topograph is deployed, they describe the physical state NVSentinel observes when correlating faults — and potentially enable topology-aware blast-radius analysis.
It looks like NVSentinel's Metadata Augmentor includes some topology labels in allowedLabels but doesn’t presently include these four.
Is there a preferred approach for NVSentinel vis-a-vis topograph's labels? One of these? Or perhaps there’s a compelling reason to avoid?
- Default-on in Metadata Augmentor — append the four labels to
allowedLabels in distros/kubernetes/nvsentinel/values.yaml so NVSentinel picks them up automatically when they exist on nodes.
- Default-off, operator opt-in — document the values override; operators wanting the integration add the labels themselves.
- Auto-detect — conditionally include the labels if topograph is detected in the cluster (harder to test).
- Separate topograph-aware sensor — don't touch Metadata Augmentor; build a dedicated NVSentinel component that understands topograph semantics.
Related
Context
In response to @lalitadithya ’s question raised in topograph#179 earlier, https://github.com/NVIDIA/topograph/blob/main/docs/reference/node-labels.md has been published NVIDIA/topograph publishes four canonical node labels describing physical network topology:
network.topology.nvidia.com/accelerator— NVLink domain (clique) IDnetwork.topology.nvidia.com/leaf— leaf switch identifiernetwork.topology.nvidia.com/spine— spine switch identifiernetwork.topology.nvidia.com/core— core switch identifierThese labels can be consumed today by KAI Scheduler, Grove, Kueue & similar for topology-aware placement. On clusters where Topograph is deployed, they describe the physical state NVSentinel observes when correlating faults — and potentially enable topology-aware blast-radius analysis.
It looks like NVSentinel's Metadata Augmentor includes some topology labels in
allowedLabelsbut doesn’t presently include these four.Is there a preferred approach for NVSentinel vis-a-vis topograph's labels? One of these? Or perhaps there’s a compelling reason to avoid?
allowedLabelsindistros/kubernetes/nvsentinel/values.yamlso NVSentinel picks them up automatically when they exist on nodes.Related