Repository navigation
docs(gfd): correct the gpu.mode, gpu.replicas, gfd.timestamp and cuda.runtime-version descriptions - #2104
docs(gfd): correct the gpu.mode, gpu.replicas, gfd.timestamp and cuda.runtime-version descriptions#2104fjvbn2003 wants to merge 1 commit into
Conversation
….runtime-version descriptions gpu.mode reports "graphics" rather than "display", gpu.replicas is the number of replicas per GPU, gfd.timestamp is the time GFD started, and cuda.runtime-version is the CUDA version the driver supports. Signed-off-by: YoungjuKim <[email protected]>
| | nvidia.com/cuda.runtime-version.major | Integer | Major of the highest CUDA version supported by the NVIDIA driver | 12 | | ||
| | nvidia.com/cuda.runtime-version.minor | Integer | Minor of the highest CUDA version supported by the NVIDIA driver | 5 | | ||
| | nvidia.com/cuda.runtime-version.full | Integer | Highest CUDA version supported by the NVIDIA driver. This is not the version of the CUDA runtime in a container image | 12.5 | | ||
| | nvidia.com/gfd.timestamp | Integer | Unix time at which GFD started and generated the first set of labels (optional). It is not updated when GFD regenerates the labels after each sleep interval | 1724632719 | |
There was a problem hiding this comment.
We should rather update the label value, isn't it?
There was a problem hiding this comment.
As far as I can tell, keeping the value fixed is intentional, so this PR only documents current behavior:
- The timestamp labeler was moved out of the loop in NVIDIA/gpu-feature-discovery@51bbda5 ("Instantiate timestamp labeler only once") and is still created before
rerun:today (main.go#L260-L261). - Drop Timestamp Label gpu-feature-discovery#8 describes the cost of a changing value: every node's labels would change on every sleep interval, which wakes every node watcher.
One thing I found while checking: TestRunSleep asserts the label stays the same (main_test.go#L368), but it doesn't actually catch a change. I moved the timestamp labeler into the loop and ran go test -run TestRunSleep -count=5: it passed 5/5. For i == 0 the modification-time wait loop doesn't run, so outFileModificationTime[0] stays 0 and the i == 1 read doesn't wait for a second run. I can fix that test separately.
If you'd rather have the label show the time of the last run, I'm happy to make that code change in a separate PR, or drop the gfd.timestamp row here so the other three rows can go in. Which do you prefer?
(Written with help from Claude (AI); the test run above was done locally.)
Description
Four rows of the "Generated Labels" table in
docs/gpu-feature-discovery/README.mddo not match what GFD writes. I compared them with the labels on five nodes running GFD v0.19.3 (GPU Operator v26.3.3) and with the code.nvidia.com/gpu.modecomputeordisplaygraphics(PCI class 0x030000),compute(0x030200) orunknown—getModeForClasses()ininternal/lm/nvml.go. All five nodes (GeForce, class0x030000) are labelledgraphics. No code path writesdisplay.nvidia.com/gpu.replicas1without sharing —getReplicas()ininternal/lm/resource.go. WithNewGPUResourceLabelerand 4 GPUs: no sharing →gpu.count=4,gpu.replicas=1; time-slicing with 4 replicas →gpu.count=4,gpu.replicas=4. On a time-sliced RTX 3090 withreplicas: 5, the node hasgpu.count=1,gpu.replicas=5and capacitynvidia.com/gpu: 5.nvidia.com/gfd.timestampNewTimestampLabeleris called once before thererun:loop incmd/gpu-feature-discovery/main.go, so the value does not change when GFD relabels every sleep interval. On all five nodes the label equals the container start time to the second (for example1789974767=2026-09-21T07:12:47Z=startedAt), while the log showsCreating Labelsevery 60 s.nvidia.com/cuda.runtime-version.*(and the deprecatedcuda.runtime.*)nvmlSystemGetCudaDriverVersion(internal/resource/nvml-lib.go) orcuDriverGetVersion(internal/resource/cuda-lib.go). Nodes with driver 595.99.02 report13.2and a node with 570.195.03 reports12.8, regardless of the CUDA runtime in the workload images.This change only updates the descriptions. It does not change behaviour.
Checklist
make lint) — documentation onlymake test) — documentation onlymake check-modules) — not affectedmake check-third-party-notices) — not affectedTesting
Compared the table with
kubectl get node -o jsonon five GPU nodes (RTX 4070 Laptop x2, RTX 4060 Laptop, RTX 3060 Laptop, time-sliced RTX 3090) and with GFD's container start times and logs. Forgpu.replicaswith more than one GPU, I ranNewGPUResourceLabelerwith a 4-GPU count in a throwaway test (not included).I used an AI assistant (Claude) to help investigate and write this change, including the checks described under Testing.
🤖 Generated with Claude Code