Skip to content

Add NVIDIA GPU passive health checks #2952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
un-def merged 5 commits into from
Aug 7, 2025
Merged

Add NVIDIA GPU passive health checks #2952

un-def merged 5 commits into from
Aug 7, 2025

Conversation

@un-def
Copy link
Collaborator

@un-def un-def commented Aug 6, 2025

  • shim: Integrate libdcgm, add a new endpoint returning overall GPU health with list of incidents.
  • Periodically pull instance health from shim, store the raw response in a new DB table. Infer overall instance health and store it in a new column of the "instances" table.
  • Don't consider failed instances for submitted jobs. Note: instances with warnings are still considered for jobs.
  • API: add a new method returning a list of instance health checks with unified structure.
  • CLI: display "warning" and "failure" health statuses in the same way as "unreachable", below the instance status.

Closes: #2930

un-def added 4 commits August 6, 2025 16:59
@un-def
Add NVIDIA GPU passive health checks
f7ed2f6 
* shim: Integrate libdcgm, add a new endpoint returning overall GPU
  health with list of incidents.
* Periodically pull instance health from shim, store the raw
  response in a new DB table. Infer overall instance health and
  store it in a new column of the "instances" table.
* Don't consider failed instances for submitted jobs. Note: instances
  with warnings are still considered for jobs.
* API: add a new method returning a list of instance health checks
  with unified structure.
* CLI: display "warning" and "failure" health statuses in the same way
  as "unreachable", below the instance status.

Closes: #2930
@un-def
CI: Fix shim build
50e19c4
@un-def
CI: disable shim/runner tests on macOS
d0fbba7 
cannot use _Ctype_long(ts) (value of type _Ctype_long)
as _Ctype_int64_t value in struct literal

cannot use _Ctype_ulong(0) (constant 0 of type _Ctype_ulong)
as _Ctype_uint64_t value in argument to (_Cfunc_dcgmPolicyRegister_v2))
@un-def
CI: compile ARM binaries on ARM runners
bbdaac3
@un-def un-def requested a review from r4victor August 6, 2025 17:42
r4victor
r4victor reviewed Aug 7, 2025
View reviewed changes
r4victor
r4victor reviewed Aug 7, 2025
View reviewed changes
@un-def un-def merged commit 28012cf into master Aug 7, 2025
25 checks passed
@un-def un-def deleted the issue_2930_dcgm_passive_health_checks branch August 7, 2025 08:29
un-def added a commit that referenced this pull request Aug 7, 2025
@un-def
[shim] Fix macOS build
48f5f32 
#2952 broke macOS because of
go-dcgm
@un-def un-def mentioned this pull request Aug 7, 2025
Merged
jvstme added a commit that referenced this pull request Aug 7, 2025
@jvstme
Fix release workflow
b0c6655 
Port changes from #2936 and #2952
@jvstme jvstme mentioned this pull request Aug 7, 2025
Merged
jvstme added a commit that referenced this pull request Aug 7, 2025
@jvstme
[Internal]: Fix release workflow (#2959)
cd07427 
Port changes from #2936 and #2952
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@r4victor r4victor r4victor approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: NVIDIA GPU passive health checks

3 participants

@un-def @r4victor