Skip to content

feat: implement NFD-based GPU hardware detection#494

Merged
mchmarny merged 1 commit into
NVIDIA:mainfrom
ArangoGutierrez:feat/nfd-hardware-detector
Apr 7, 2026
Merged

feat: implement NFD-based GPU hardware detection#494
mchmarny merged 1 commit into
NVIDIA:mainfrom
ArangoGutierrez:feat/nfd-hardware-detector

Conversation

@ArangoGutierrez

Copy link
Copy Markdown
Contributor

Summary

Add NFDHardwareDetector that uses NFD source packages (PCI + kernel) to detect NVIDIA GPU hardware via PCI enumeration and check nvidia kernel module state. This enables day-0 GPU detection without nvidia-smi or drivers installed.

Part of: NFD Snapshot Enrichment (Track A) — Task 1 of 3
Depends on: #482 (merged)

Changes

pkg/collector/gpu/hardware.go

  • Add PCI identification constants (nvidiaVendorID, pciClassVGA, pciClass3D, nvidiaKernelModule)
  • Add NFD source/feature key constants (nfdSourcePCI, nfdSourceKernel, etc.)
  • Implement NFDHardwareDetector.Detect() with defaults.NFDDetectionTimeout (5s)
    • PCI source discovery (required) for GPU device enumeration
    • Kernel source discovery (best-effort) for driver module state
    • Context checks before start and between discovery phases
  • Implement extractHardwareInfo() pure function for PCI device filtering
    • Filters by NVIDIA vendor ID (10de) + VGA (0300) or 3D controller (0302) class
    • Checks nvidia kernel module in loaded modules flags

pkg/collector/gpu/hardware_test.go

  • 7 new tests for extractHardwareInfo:
    • NVIDIA GPUs present (2 NVIDIA + 1 Intel = count 2)
    • No NVIDIA GPUs (Intel only)
    • 3D controller class (0302)
    • Driver loaded / not loaded
    • Empty features / nil features

go.mod

Testing

  • All 20 tests pass with -race flag
  • golangci-lint reports 0 issues
  • Full project builds cleanly

Design Notes

  • Blank imports of source/pci and source/kernel only register into NFD's internal source map — no K8s scheme init() side-effects (lesson from feat: add HardwareDetector interface and measurement keys for NFD integration #482 review)
  • NFDHardwareDetector is not safe for concurrent use (documented) — NFD source singletons are shared state, but GPU collector runs once per snapshot
  • extractHardwareInfo is a pure function for maximum testability

@ArangoGutierrez

Copy link
Copy Markdown
Contributor Author

cc @mchmarny

lockwobr
lockwobr previously approved these changes Apr 6, 2026

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice clean split between the pure extractHardwareInfo and the NFD-dependent Detect. A few things to address — the vendor footprint is the big one, and the tests should be table-driven per project conventions. See inline comments.

Comment thread go.mod
Comment thread pkg/collector/gpu/hardware.go Outdated
Comment thread pkg/collector/gpu/hardware.go Outdated
Comment thread pkg/collector/gpu/hardware.go Outdated
Comment thread pkg/collector/gpu/hardware_test.go Outdated
@mchmarny mchmarny enabled auto-merge (squash) April 7, 2026 11:45
Add NFDHardwareDetector that uses NFD source packages (PCI + kernel) to
detect NVIDIA GPU hardware via PCI enumeration and check nvidia kernel
module state. This enables day-0 GPU detection without nvidia-smi.

- Add NFD v0.18.3 dependency
- Isolate NFD imports in nfd.go to limit init() blast radius
- Implement NFDHardwareDetector.Detect() with timeout and platform docs
- Implement extractHardwareInfo() pure function for PCI device filtering
- Add table-driven tests for extractHardwareInfo
- Add context check between PCI and kernel discovery phases
- Document thread-safety and platform constraints

Signed-off-by: Carlos Eduardo Arango Gutierrez <[email protected]>
auto-merge was automatically disabled April 7, 2026 14:07

Head branch was pushed to by a user without write access

@ArangoGutierrez ArangoGutierrez force-pushed the feat/nfd-hardware-detector branch from 3c52e09 to 384af3b Compare April 7, 2026 14:07
@mchmarny mchmarny merged commit 81cf701 into NVIDIA:main Apr 7, 2026
32 of 44 checks passed
@ArangoGutierrez ArangoGutierrez deleted the feat/nfd-hardware-detector branch April 8, 2026 20:02
ArangoGutierrez added a commit to ArangoGutierrez/aicr that referenced this pull request Apr 28, 2026
Adds an nfd componentRefs entry to the 12 production GPU leaf overlays
(H100, GB200, RTX Pro 6000 on EKS / AKS / GKE / OKE / LKE) flipping
overrides.topologyUpdater.enable to true. Each cluster running these
recipes now publishes per-node NodeResourceTopology (NRT) CRDs
describing NUMA zones, GPU-to-NUMA affinity, and NIC-to-NUMA affinity
— the fact base downstream NUMA-aware schedulers and recipe
auto-resolution flows can consume.

Overlay change: each leaf gains a fresh componentRefs entry with
name=nfd, type=Helm, and overrides.topologyUpdater.enable=true.
mergeComponentRef (pkg/recipe/metadata.go:532) deep-merges the
overrides onto the base nfd entry from recipes/overlays/base.yaml
without replacing inherited source/version/valuesFile.

Chart configuration (recipes/components/nfd/values.yaml):

* topologyUpdater.createCRDs: true ensures the noderesourcetopologies
  CRD is installed by the chart on overlays that flip enable=true.
  Managed K8s control planes (EKS / AKS / GKE / OKE / LKE) do not
  preinstall it. The chart guards the CRD template on
  (enable && createCRDs), so this is a no-op when TU is off.
* topologyUpdater.kubeletStateDir: "" disables the broad
  /var/lib/kubelet hostPath mount. TU only needs the pod-resources
  gRPC socket, which is mounted via a dedicated hostPath; the empty
  state-dir stops the chart from exposing kubelet state files (TLS
  material, pod manifests, checkpoint data) into the TU container.

Scheduling (recipes/registry.yaml): nfd.nodeScheduling.{system,
accelerated}.tolerationPaths now include topologyUpdater.tolerations
so the bundler injects --accelerated-node-toleration values into TU
pods. Without this, on every targeted GPU cluster (which carry
nvidia.com/gpu=present:NoSchedule), the TU DaemonSet would have been
unschedulable. nodeSelector deliberately not added — TU runs on all
nodes (same rationale as worker), and topology data on system nodes
is needed for cross-NUMA scheduling decisions.

Kind-chain overlays (h100-kind-*, kind-*) are intentionally excluded:
KWOK and kind clusters lack a real kubelet pod-resources gRPC socket,
so TU would CrashLoopBackOff. The KubeletPodResources feature gate
has been Beta-default since Kubernetes 1.15 and reached GA in 1.28
(KEP-606); AICR's affected leaves require K8s >= 1.30, so the
prerequisite is satisfied in practice.

Coverage:

* New chainsaw step assert-bundle-topology-nfd in CUJ1-training
  asserts the rendered NFD bundle contains topologyUpdater.enable=
  true, createCRDs=true, the GPU-taint toleration, master.enable=
  true, gc.enable=true, and enableNodeFeatureApi=true.
* New TestNFDTopologyUpdater_OverlayCoverage in pkg/recipe/
  metadata_test.go covers all 12 GPU platform+intent overlays
  (expected ON) and both kind-chain leaves (expected OFF).
  Type-assertion failures on ON cases promote to t.Fatal so a
  malformed override produces a loud regression rather than a
  silent false-negative.
* docs/user/component-catalog.md gains the missing NFD row and a
  Topology Updater section documenting which recipes run TU and
  the kubelet pod-resources prerequisite.

Closes the local NFD adoption initiative (Tracks A and B previously
shipped via PRs NVIDIA#482, NVIDIA#494, NVIDIA#495, NVIDIA#502, NVIDIA#511, NVIDIA#518, NVIDIA#688). Track C
(recipe auto-resolution from NRT data) and scheduler-integration
work remain open for future PRs.

Signed-off-by: Carlos Eduardo Arango Gutierrez <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants