Skip to content

feat: Add resource-policy accelerator_topology_mode#5393

Merged
SwarnaBharathiMantena merged 14 commits into
GoogleCloudPlatform:developfrom
jamOne-:resource-policy
Apr 1, 2026
Merged

feat: Add resource-policy accelerator_topology_mode#5393
SwarnaBharathiMantena merged 14 commits into
GoogleCloudPlatform:developfrom
jamOne-:resource-policy

Conversation

@jamOne-
Copy link
Copy Markdown
Contributor

@jamOne- jamOne- commented Mar 24, 2026 •
edited
Loading

Description

This PR introduces generic support for the accelerator_topology_mode field in the resource-policy module, enabling the use of PROVISION_ONLY for Compute Engine Super-slicing architectures. It also enforces necessary architectural constraints (like Dynamic Workload Scheduler exclusivity) without hardcoding marketing terminology into the modules.

Motivation and Context

Super-slicing support in CT.

Key Changes

  • Added accelerator_topology_mode: Integrated this optional variable into the resource-policy module's workload_policy block.
  • Explicit workload_policy.type: Ensured explicit assignment of workload_policy.type when topologies are specified, preventing the masking of future GCP API additions.
  • API Validation Guards: Added lifecycle { precondition {} } blocks to ensure both workload_policy.type and accelerator_topology are provided whenever an accelerator_topology_mode is defined.
  • Provider Version Bumps: Bumped the minimum required google-beta provider to 7.24.0 in affected modules (resource-policy and gke-node-pool) and adjusted the compiler's GoogleProviderVersion upper bound to <= 7.24.0 in pkg/config/expand.go.
  • DWS / Queued Provisioning Exclusivity: Added a lifecycle { precondition {} } block to the gke-node-pool module to explicitly prevent combining enable_queued_provisioning (DWS) with custom topology modes like PROVISION_ONLY.
    • Reasoning: GKE's Dynamic Workload Scheduler (Queued Provisioning) dynamically manages its own compute resource policies. Forcing a static custom resource policy onto a node pool while DWS is enabled causes a direct GCP API collision. (DWS Documentation)

Verification

  • Manual End-to-End Testing: Verified the module behavior manually in a live GCP project. Successfully generated a blueprint using ghpc and provisioned a compute resource policy with accelerator_topology = "16x16" and accelerator_topology_mode = "PROVISION_ONLY" via Terraform. Confirmed the configuration is correctly parsed, sent by the updated google-beta provider, and accepted as READY by the GCP API.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

  • Fork your PR branch from the Toolkit "develop" branch (not main)
  • Test all changes with pre-commit in a local branch #
  • Confirm that "make tests" passes all tests
  • Add or modify unit tests to cover code changes
  • Ensure that unit test coverage remains above 80%
  • Update all applicable documentation
  • Follow Cluster Toolkit Contribution guidelines #

@jamOne- jamOne- changed the base branch from main to develop March 24, 2026 10:44
@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Mar 24, 2026

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the developer experience by automating the management of core external dependencies like Terraform and Packer. It also introduces advanced capabilities for GKE, such as the GKE Slice Controller for GPU resource optimization, and refines Slurm cluster configurations with more robust topology generation. A new example blueprint showcases high-performance GCS RAPID storage with Anywhere Cache, expanding the toolkit's storage integration options. Additionally, various internal dependencies have been updated to maintain a modern and stable codebase.

Highlights

  • Automatic Dependency Management: Introduced a new mechanism to automatically download and manage terraform and packer binaries, ensuring that required tools are available and correctly versioned. This includes a new --download-dependencies flag for explicit control over this behavior.
  • GKE Enhancements: Added support for the GKE Slice Controller, enabling advanced GPU resource management (Super-slicing) for GKE clusters. Also, included commented-out configurations for Managed Lustre integration in GKE blueprints, indicating future expansion for high-performance file systems.
  • Slurm Topology Generation Improvements: Enhanced the Slurm topology generation script to include unique block UUIDs, improving the robustness and uniqueness of generated block names. Docstrings were also added to clarify the logic for topology reconfiguration and compression.
  • New GCS RAPID Storage Example: Added a new example blueprint (rapid-storage-slurm.yaml) demonstrating the use of Google Cloud Storage with the RAPID storage class, hierarchical namespaces, and Anywhere Cache for high-performance data access within a Slurm cluster.
  • General Dependency Updates: Updated various Go and Python dependencies, including shfmt, goimports, and numerous Python packages in requirements.txt and requirements-dev.txt, to ensure compatibility and leverage the latest features and fixes.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the external PR from external contributor label Mar 24, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Mar 24, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for "Super-slicing" and the GKE Slice Controller. It includes new variables such as accelerator_topology_mode and enable_slice_controller, and updates Terraform configurations to integrate these features. Crucial preconditions have been added to ensure compatibility with GKE versions for the Slice Controller. Additionally, the Google provider versions have been updated across relevant files. The changes appear to be well-implemented and include necessary validations for the new functionalities.

@aslam-quad
Copy link
Copy Markdown
Contributor

aslam-quad commented Mar 24, 2026

/gcbrun

jamOne- added 7 commits March 24, 2026 15:14
@jamOne-
feat: integrate PROVISION_ONLY resource policy mode for Super-slicing
86e8aaa 
Adds accelerator_topology_mode to the resource-policy module mapped to workload_policy. Includes an exclusivity precondition within the gke-node-pool module to ensure PROVISION_ONLY clusters are never incorrectly configured alongside queued provisioning.
@jamOne-
refactor: decouple PROVISION_ONLY references from Super-slicing and m…
985e5a2 
…ake workload_policy type explicit
@jamOne-
build: bump minimum google-beta provider version to 7.24.0
b778eb8
@jamOne-
fix: ensure workload_policy type is provided when specifying accelera…
ea6fc07 
…tor_topology_mode
@jamOne-
fix: ensure workload_policy.accelerator_topology is set with accelera…
83b0fda 
…tor_topology_mode
@jamOne-
fix: adjust google provider version upper bound to < 8.0.0
e6a0cd7
@jamOne-
revert: reset google provider version upper bound to <= 7.24.0
213d57b
@jamOne- jamOne- force-pushed the resource-policy branch 6 times, most recently from eac1c38 to a2b8179 Compare March 24, 2026 16:15
@jamOne- jamOne- marked this pull request as ready for review March 24, 2026 16:31
@jamOne- jamOne- requested review from a team and samskillman as code owners March 24, 2026 16:31
@jamOne- jamOne- changed the title "Resource policy feat: Add accelerator_topology_mode Mar 24, 2026
@jamOne- jamOne- changed the title feat: Add accelerator_topology_mode feat: Add resource-policy accelerator_topology_mode Mar 24, 2026
@Neelabh94
Copy link
Copy Markdown
Contributor

Neelabh94 commented Mar 26, 2026

/gcbrun

@Neelabh94 Neelabh94 added the release-module-improvements Added to release notes under the "Module Improvements" heading. label Mar 26, 2026
@jamOne- jamOne- marked this pull request as draft March 26, 2026 17:02
SwarnaBharathiMantena
SwarnaBharathiMantena requested changes Mar 27, 2026
View reviewed changes
Comment thread modules/compute/resource-policy/variables.tf Outdated
@jamOne- jamOne- marked this pull request as ready for review March 27, 2026 14:35
@jamOne- jamOne- marked this pull request as draft March 30, 2026 08:17
@jamOne- jamOne- marked this pull request as ready for review March 30, 2026 10:44
@SwarnaBharathiMantena
Copy link
Copy Markdown
Contributor

SwarnaBharathiMantena commented Mar 30, 2026

/gcbrun

@SwarnaBharathiMantena
Copy link
Copy Markdown
Contributor

SwarnaBharathiMantena commented Mar 31, 2026

/gcbrun

@SwarnaBharathiMantena
Copy link
Copy Markdown
Contributor

SwarnaBharathiMantena commented Apr 1, 2026

/gcbrun

SwarnaBharathiMantena
SwarnaBharathiMantena approved these changes Apr 1, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@SwarnaBharathiMantena SwarnaBharathiMantena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@SwarnaBharathiMantena SwarnaBharathiMantena enabled auto-merge (squash) April 1, 2026 08:50
@SwarnaBharathiMantena SwarnaBharathiMantena merged commit 5c1001d into GoogleCloudPlatform:develop Apr 1, 2026
32 of 72 checks passed
simrankaurb pushed a commit to simrankaurb/cluster-toolkit that referenced this pull request Apr 7, 2026
@jamOne- @simrankaurb
feat: Add resource-policy accelerator_topology_mode (GoogleCloudPlatf…
2aff3f0 
…orm#5393)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Neelabh94 Neelabh94 Neelabh94 approved these changes

@SwarnaBharathiMantena SwarnaBharathiMantena SwarnaBharathiMantena approved these changes

@samskillman samskillman Awaiting requested review from samskillman

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

external PR from external contributor release-module-improvements Added to release notes under the "Module Improvements" heading.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jamOne- @aslam-quad @Neelabh94 @SwarnaBharathiMantena