Build magma binary tarballs for various cuda #139888

afrittoli · 2024-11-06T15:07:52Z

This is a first step towards removing builds dependency to conda.

Currently we build magma as a conda package in a pytorch conda channel, implemented in https://github.com/pytorch/builder/tree/a1b372dbda2e9e3bd946cf135aa3b0137dfdf052/magma.

This commit adapts the logic from pytorch/builder as follows:

use pytorch/manylinux-cuda as base image
apply patches and invoke the build.sh script directly (not anymore through conda build)
stores license and build files along with the built artifact, in an info subfolder
create a tarball file which resembles that created by conda, without any conda-specific metadata

A new matrix workflow is added, which runs the build for each supported cuda version, and uploads the binaries to pyorch s3 bucket.

For the upload, define an upload.sh script, which will be used by the magma windows job as well, to upload to s3://ossci-* buckets.

The build runs on PR and push, upload runs in DRY_RUN mode in case of PR.

Fixes #139397

pytorch-bot · 2024-11-06T15:07:56Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139888

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e2987ff with merge base 3abbde9 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

atalman

This looks great! Please create another PR removing these magma builds from builder repo.
Also would make sence to move the windows magma builds workflows here as well.

afrittoli · 2024-11-06T16:04:13Z

This looks great! Please create another PR removing these magma builds from builder repo. Also would make sence to move the windows magma builds workflows here as well.

Thanks - I need to check what the Windows will look like without conda, once I figure it out I will make another PR like this one. I don't have a windows environment handy to test the Windows builds though, so there may be a bit more of trial and error there.

afrittoli · 2024-11-06T16:07:54Z

@atalman would it make sense to configure the linux-focal-py* jobs to skip for PRs that only touch the magma folder and/or this workflow?

afrittoli · 2024-11-06T16:51:43Z

This looks great! Please create another PR removing these magma builds from builder repo. Also would make sence to move the windows magma builds workflows here as well.

Thanks - I need to check what the Windows will look like without conda, once I figure it out I will make another PR like this one. I don't have a windows environment handy to test the Windows builds though, so there may be a bit more of trial and error there.

It looks like the windows path does not use conda, so it should be only a matter of copying the files over and fixing whatever linter rules break in the new repo.
I noticed that in the windows workflow we push to a dedicated bucket, not the pytorch one. Shall I do the same here?

          OSSCI_WINDOWS_S3: s3://ossci-windows/

https://github.com/pytorch/builder/blob/a1b372dbda2e9e3bd946cf135aa3b0137dfdf052/.github/workflows/build-magma-windows.yml#L46-L55

atalman · 2024-11-06T16:59:43Z

@atalman would it make sense to configure the linux-focal-py* jobs to skip for PRs that only touch the magma folder and/or this workflow?

I think so magma builds are pretty much standalone. They are used in Docker builds after being published.

malfet

LGTM, will edit PR description to specify specific commit those magma scripts are picked from

Add a version of the manylinux 2.28 image with cuda 12.6. Once this is done, cuda 12.6 can be enable for the new magma non-conda distribution provided by pytorch#139888 Signed-off-by: Andrea Frittoli <[email protected]>

.github/workflows/build-magma-linux.yml

.github/scripts/upload.sh

Add a version of the manylinux 2.28 image with cuda 12.6. Once this is done, cuda 12.6 can be enable for the new magma non-conda distribution provided by #139888 Partially-fixes #139397 Pull Request resolved: #139909 Approved by: https://github.com/atalman

atalman · 2024-11-08T13:26:42Z

@pytorchbot merge -f "signal is green"

pytorchmergebot · 2024-11-08T13:28:16Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Copy the magma for windows job and script from pytorch/builder https://github.com/pytorch/builder/blob/c9aac65e12734a16d9c285c366bcbf6cc0c67e43/.github/workflows/build-magma-windows.yml The linux version is moved here in pytorch#139888 Use the upload_aws_ossci.sh upload script used by the linux magma builds as well. Signed-off-by: Andrea Frittoli <[email protected]>

Add a version of the manylinux 2.28 image with cuda 12.6. Once this is done, cuda 12.6 can be enable for the new magma non-conda distribution provided by pytorch#139888 Partially-fixes pytorch#139397 Pull Request resolved: pytorch#139909 Approved by: https://github.com/atalman

Copy the magma for windows job and script from pytorch/builder https://github.com/pytorch/builder/blob/c9aac65e12734a16d9c285c366bcbf6cc0c67e43/.github/workflows/build-magma-windows.yml The linux version is moved here in #139888 Fixes #140001 Pull Request resolved: #139924 Approved by: https://github.com/atalman

Add a version of the manylinux 2.28 image with cuda 12.6. Once this is done, cuda 12.6 can be enable for the new magma non-conda distribution provided by pytorch#139888 Partially-fixes pytorch#139397 Pull Request resolved: pytorch#139909 Approved by: https://github.com/atalman

This is a first step towards removing builds dependency to conda. Currently we build magma as a conda package in a pytorch conda channel, implemented in https://github.com/pytorch/builder/tree/a1b372dbda2e9e3bd946cf135aa3b0137dfdf052/magma. This commit adapts the logic from pytorch/builder as follows: - use pytorch/manylinux-cuda<cuda-version> as base image - apply patches and invoke the build.sh script directly (not anymore through conda build) - stores license and build files along with the built artifact, in an info subfolder - create a tarball file which resembles that created by conda, without any conda-specific metadata A new matrix workflow is added, which runs the build for each supported cuda version, and uploads the binaries to pyorch s3 bucket. For the upload, define an upload.sh script, which will be used by the magma windows job as well, to upload to `s3://ossci-*` buckets. The build runs on PR and push, upload runs in DRY_RUN mode in case of PR. Fixes pytorch#139397 Pull Request resolved: pytorch#139888 Approved by: https://github.com/atalman, https://github.com/malfet, https://github.com/seemethere

Copy the magma for windows job and script from pytorch/builder https://github.com/pytorch/builder/blob/c9aac65e12734a16d9c285c366bcbf6cc0c67e43/.github/workflows/build-magma-windows.yml The linux version is moved here in pytorch#139888 Fixes pytorch#140001 Pull Request resolved: pytorch#139924 Approved by: https://github.com/atalman

Resolves #151707 for CUDA Nvidia Magma builds. Removes deprecated cuda 12.4 build. Using `pytorch/manylinux2_28-builder` image for magma build creates circular dependency. For a while for magma builds we used `conda-builder` image since it does not have circular dependency: https://github.com/pytorch/builder/blob/release/2.4/magma/Makefile#L13 However during migration to pytorch/pytorch: #139888 we introduced circular dependency using Manylinux 2.28 docker image. Hence using almalinux image which suppose to be general usage image Please note: Magma builds using Docker build : https://github.com/pytorch/pytorch/blob/main/.ci/magma/README.md we can look into migrating them to Docker images if required as a followup BE change if needed TODO: Make same change for rocm builds. I believe some more work for rocm is required, since maga-rocm is requires rocm dev, utils and lib to be installed : https://github.com/pytorch/pytorch/blob/main/.ci/docker/common/install_rocm.sh Pull Request resolved: #152358 Approved by: https://github.com/nWEIdia, https://github.com/huydhn

afrittoli requested a review from a team as a code owner November 6, 2024 15:07

pytorch-bot bot added the release notes: releng release notes category label Nov 6, 2024

afrittoli force-pushed the magma_build_no_conda branch from 4549d3f to 8518b99 Compare November 6, 2024 15:12

pytorchbot added the open source label Nov 6, 2024

afrittoli force-pushed the magma_build_no_conda branch 3 times, most recently from a8f0c87 to c1c4fd6 Compare November 6, 2024 15:44

atalman approved these changes Nov 6, 2024

View reviewed changes

afrittoli force-pushed the magma_build_no_conda branch from c1c4fd6 to 76e070e Compare November 6, 2024 16:42

afrittoli force-pushed the magma_build_no_conda branch from 76e070e to 1a7d9c4 Compare November 6, 2024 18:17

malfet reviewed Nov 6, 2024

View reviewed changes

malfet approved these changes Nov 6, 2024

View reviewed changes

afrittoli mentioned this pull request Nov 6, 2024

Create manylinux 2.28 cuda 12.6 image #139909

Closed

afrittoli force-pushed the magma_build_no_conda branch 2 times, most recently from 813278e to 11b6f8e Compare November 6, 2024 20:49

afrittoli mentioned this pull request Nov 6, 2024

Build magma for windows #139924

Closed

afrittoli force-pushed the magma_build_no_conda branch from 11b6f8e to 9376f96 Compare November 6, 2024 21:47

afrittoli commented Nov 6, 2024

View reviewed changes

.github/workflows/build-magma-linux.yml Outdated Show resolved Hide resolved

afrittoli force-pushed the magma_build_no_conda branch 4 times, most recently from da548c0 to f7e02ad Compare November 7, 2024 18:07

atalman reviewed Nov 7, 2024

View reviewed changes

.github/scripts/upload.sh Outdated Show resolved Hide resolved

pytorchmergebot added the merging label Nov 8, 2024

pytorchmergebot closed this in c1c94cb Nov 8, 2024

pytorchmergebot added Merged and removed merging labels Nov 8, 2024

afrittoli mentioned this pull request Nov 9, 2024

Build magma without conda and push tarball pytorch/builder#2026

Closed

atalman mentioned this pull request Apr 28, 2025

Use almalinux docker files for building Magma #152358

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build magma binary tarballs for various cuda #139888

Build magma binary tarballs for various cuda #139888

Uh oh!

afrittoli commented Nov 6, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Nov 6, 2024 •

edited

Loading

Uh oh!

atalman left a comment

Uh oh!

afrittoli commented Nov 6, 2024

Uh oh!

afrittoli commented Nov 6, 2024

Uh oh!

afrittoli commented Nov 6, 2024

Uh oh!

atalman commented Nov 6, 2024

Uh oh!

malfet left a comment

Uh oh!

Uh oh!

Uh oh!

atalman commented Nov 8, 2024

Uh oh!

pytorchmergebot commented Nov 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Build magma binary tarballs for various cuda #139888

Build magma binary tarballs for various cuda #139888

Uh oh!

Conversation

afrittoli commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139888

✅ No Failures

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

afrittoli commented Nov 6, 2024

Uh oh!

afrittoli commented Nov 6, 2024

Uh oh!

afrittoli commented Nov 6, 2024

Uh oh!

atalman commented Nov 6, 2024

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

atalman commented Nov 8, 2024

Uh oh!

pytorchmergebot commented Nov 8, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

afrittoli commented Nov 6, 2024 •

edited

Loading

pytorch-bot bot commented Nov 6, 2024 •

edited

Loading