Skip to content

Add example nccl test script for slurm on gke#4960

Merged
LAVEEN merged 1 commit intoGoogleCloudPlatform:developfrom
ACW101:nccl-run
Dec 10, 2025
Merged

Add example nccl test script for slurm on gke#4960
LAVEEN merged 1 commit intoGoogleCloudPlatform:developfrom
ACW101:nccl-run

Conversation

@ACW101
Copy link
Copy Markdown
Collaborator

@ACW101 ACW101 commented Dec 9, 2025

Submission Checklist

Add example NCCL script specific for running Slurm on GKE. Based on the previous work in https://github.com/GoogleCloudPlatform/cluster-toolkit/tree/main/examples/machine-learning/a3-megagpu-8g/nccl-tests and https://github.com/GoogleCloudPlatform/cluster-toolkit/tree/main/examples/machine-learning/a3-ultragpu-8g

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

  • Fork your PR branch from the Toolkit "develop" branch (not main)
  • Test all changes with pre-commit in a local branch #
  • Confirm that "make tests" passes all tests
  • Add or modify unit tests to cover code changes
  • Ensure that unit test coverage remains above 80%
  • Update all applicable documentation
  • Follow Cluster Toolkit Contribution guidelines #

@ACW101 ACW101 requested review from a team and samskillman as code owners December 9, 2025 06:15
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello @ACW101, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a new example to the community/examples/slurm-gke/nccl-tests directory, providing a complete guide for setting up and running NCCL performance benchmarks on a Slurm-managed GKE cluster. The example includes scripts for building the NCCL tests inside a container and then executing them with configurations optimized for different A3 GPU types and network technologies (RDMA and TCPXO). This aims to assist users in effectively deploying and evaluating GPU-accelerated, containerized applications within a GKE Slurm environment.

Highlights

  • New Example for NCCL Tests on GKE Slurm: Introduced a comprehensive example demonstrating how to run NVIDIA Collective Communications Library (NCCL) performance tests on a Slurm cluster deployed on Google Kubernetes Engine (GKE).
  • Containerized Build Process: Added a Slurm batch script (build-nccl-tests.sh) that builds the NCCL tests within a specified container image using srun and pyxis, ensuring compatibility and efficient setup.
  • RDMA and TCPXO Test Scripts: Provided two distinct Slurm batch scripts for running NCCL all_gather_perf benchmarks: run-nccl-tests-rdma.sh for A3 Ultra, A4, and A4X GPU instances utilizing RDMA, and run-nccl-tests-tcpxo.sh for A3 Mega GPU instances using TCPXO.
  • Enroot and Pyxis Integration: The examples showcase the use of enroot and pyxis for launching containerized GPU workloads, including environment variable setup and necessary volume mounts for network interconnects like gIB (Google InfiniBand) and GPUDirect-TCPXO.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@ACW101 ACW101 added the enhancement New feature or request label Dec 9, 2025
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds helpful example scripts for running NCCL tests on Slurm on GKE. The scripts and documentation are a good starting point, but there are several inconsistencies and errors that need to be addressed. Most importantly, as per the repository style guide (line 33), new examples must be added to the index in examples/README.md. This change is missing and is critical for discoverability. Additionally, there is a critical syntax error in run-nccl-tests-tcpxo.sh due to a missing closing quote. I've left detailed comments on the README and shell scripts to correct these issues, as well as benchmark names, script references, container version inconsistencies, and other items to improve clarity and ensure the examples work as expected.

Comment thread community/examples/slurm-gke/nccl-tests/run-nccl-tests-tcpxo.sh Outdated
Comment thread community/examples/slurm-gke/nccl-tests/README.md Outdated
Comment thread community/examples/slurm-gke/nccl-tests/README.md
Comment thread community/examples/slurm-gke/nccl-tests/build-nccl-tests.sh Outdated
Comment thread community/examples/slurm-gke/nccl-tests/README.md Outdated
Comment thread community/examples/slurm-gke/nccl-tests/README.md
Comment thread community/examples/slurm-gke/nccl-tests/build-nccl-tests.sh Outdated
Comment thread community/examples/slurm-gke/nccl-tests/run-nccl-tests-rdma.sh
Comment thread community/examples/slurm-gke/nccl-tests/run-nccl-tests-rdma.sh Outdated
Comment thread community/examples/slurm-gke/nccl-tests/run-nccl-tests-tcpxo.sh
@ACW101 ACW101 added release-improvements Added to release notes under the "Improvements" heading. and removed enhancement New feature or request labels Dec 9, 2025
@ACW101 ACW101 force-pushed the nccl-run branch 2 times, most recently from b823081 to 76baf7f Compare December 9, 2025 06:40
Comment thread community/examples/slurm-gke/nccl-tests/README.md Outdated
Comment thread community/examples/slurm-gke/nccl-tests/run-nccl-tests-rdma.sh
Comment thread community/examples/slurm-gke/nccl-tests/build-nccl-tests.sh
Comment thread community/examples/slurm-gke/nccl-tests/run-nccl-tests-tcpxo.sh
Comment thread community/examples/slurm-gke/nccl-tests/run-nccl-tests-rdma.sh Outdated
@ACW101 ACW101 requested a review from nick-stroud December 9, 2025 17:29
nick-stroud
nick-stroud previously approved these changes Dec 10, 2025
LAVEEN
LAVEEN previously approved these changes Dec 10, 2025
@LAVEEN LAVEEN merged commit f7a4428 into GoogleCloudPlatform:develop Dec 10, 2025
13 of 74 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-improvements Added to release notes under the "Improvements" heading.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants