Summary

aicr validate fails on BCM/Ubuntu nodes because a consumer in the gpu-operator / DRA driver stack reads the kernel config from the upstream source-tree path /usr/src/linux-$(uname -r)/.config, but Ubuntu installs that file under the -headers- infix path /usr/src/linux-headers-$(uname -r)/.config. NodeWright should ship this path alias as a node setup/tuning so it survives reboots and kernel upgrades, replacing the current manual pdsh workaround.

Context

Found while validating AICR's BCM service end-to-end on node00[1-7]. The nvidia-driver-daemonset / DRA stack expects the source-tree convention /usr/src/linux-$(uname -r)/.config; on Ubuntu the file only exists at /usr/src/linux-headers-$(uname -r)/.config. Without the alias, validation stalls (DRA pod Init:0/1, driver daemonset unhealthy).

Current manual workaround (does not survive reboot/upgrade)

pdsh -w node00[1-7] '
  mkdir -p /usr/src/linux-$(uname -r) &&
  ln /usr/src/linux-headers-$(uname -r)/.config /usr/src/linux-$(uname -r)/.config
'

This has to be re-applied by hand after every reboot or kernel upgrade. Confirmed required for aicr validate to pass.

Proposed NodeWright implementations

Option A — single-file symlink (matches workaround scope):

mkdir -p /usr/src/linux-$(uname -r)
ln -sf /usr/src/linux-headers-$(uname -r)/.config /usr/src/linux-$(uname -r)/.config

Option B — full-directory symlink (also covers consumers probing the headers tree for Makefile / include / Module.symvers):

[ ! -e /usr/src/linux-$(uname -r) ] && \
  ln -s linux-headers-$(uname -r) /usr/src/linux-$(uname -r)

Likely belongs as a setup rather than a tuning if it's needed across all Ubuntu cases.

Acceptance criteria

• After reboot / kernel upgrade, /usr/src/linux-$(uname -r)/.config resolves without manual intervention.
• aicr validate --phase deployment and --phase conformance pass with no manual pdsh hardlinks: nvidia-driver-daemonset clean, DRA pod reaches Running (no Init:0/1 stall), and all health checks green (platform-health, gpu-operator-health, dra-support, accelerator-metrics, ai-service-metrics).

Notes

• This is a NodeWright deliverable tracked in AICR; the package can be published publicly (confirmed with the NodeWright owner).