Skip to content

Move inductor cuda 124 jobs to a separate workflow that is not triggered by ciflow/inductor #128250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
clee2000 wants to merge 1 commit into from
Closed

Move inductor cuda 124 jobs to a separate workflow that is not triggered by ciflow/inductor #128250

clee2000 wants to merge 1 commit into from

Conversation

@clee2000
Copy link
Contributor

@clee2000 clee2000 commented Jun 7, 2024 •
edited
Loading

#127825

The majority of the g5 runner usage comes from inductor (its something like 2x everything else)
in the past week, inductor ran 1300 ish times on PRs and 300 times on main. Inductor-periodic ran 50 times on main, so the previous move from inductor -> inductor-periodic only results in 250 fewer runs.

I was under the impression that cu124 is experimental currently and eventually we'll need to switch to it, so this will stay until we switch or inductor uses much fewer runners

Are we expected to be able to handle two versions of cuda in CI? Because currently we cannot, at least not comfortably

@pytorch-bot
Copy link

pytorch-bot bot commented Jun 7, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/128250

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 5 New Failures

As of commit 57bb8d1 with merge base e647ea5 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jun 7, 2024
@pytorch-bot
Copy link

pytorch-bot bot commented Jun 7, 2024

Warning: Unknown label ciflow/inductor-cu124.
Currently recognized labels are

  • ciflow/binaries
  • ciflow/binaries_conda
  • ciflow/binaries_libtorch
  • ciflow/binaries_wheel
  • ciflow/inductor
  • ciflow/inductor-perf-compare
  • ciflow/inductor-micro-benchmark
  • ciflow/linux-aarch64
  • ciflow/mps
  • ciflow/nightly
  • ciflow/periodic
  • ciflow/rocm
  • ciflow/slow
  • ciflow/trunk
  • ciflow/unstable
  • ciflow/xpu
  • ciflow/torchbench

Please add the new label to .github/pytorch-probot.yml

@clee2000 clee2000 marked this pull request as ready for review June 7, 2024 20:23
@clee2000 clee2000 requested a review from a team as a code owner June 7, 2024 20:23
huydhn
huydhn approved these changes Jun 7, 2024
View reviewed changes
Copy link
Contributor

@huydhn huydhn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! We need to remember to clean this up

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jun 7, 2024
@clee2000
Copy link
Contributor Author

clee2000 commented Jun 7, 2024

@pytorchbot merge -f "no trunk needed"

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jun 7, 2024

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@nWEIdia
Copy link
Collaborator

nWEIdia commented Jun 12, 2024 •
edited
Loading

Is there a label to trigger these cu124 inductor jobs? Currently my #128423 does not seem to run cu124 jobs
Update: ciflow/inductor-cu124 seems to be it.

TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
@clee2000 @TharinduRusira
Move inductor cuda 124 jobs to a separate workflow that is not trigge…
0e512d1 
…red by ciflow/inductor (pytorch#128250)

pytorch#127825

The majority of the g5 runner usage comes from inductor (its something like 2x everything else)
in the past week, inductor ran 1300 ish times on PRs and 300 times on main.  Inductor-periodic ran 50 times on main, so the previous move from inductor -> inductor-periodic only results in 250 fewer runs.

I was under the impression that cu124 is experimental currently and eventually we'll need to switch to it, so this will stay until we switch or inductor uses much fewer runners

Are we expected to be able to handle two versions of cuda in CI?  Because currently we cannot, at least not comfortably

Pull Request resolved: pytorch#128250
Approved by: https://github.com/huydhn
@github-actions github-actions bot deleted the csl/inductor_cu124 branch July 12, 2024 01:56
@clee2000 clee2000 mentioned this pull request Aug 9, 2024
Closed
pytorchmergebot pushed a commit that referenced this pull request Aug 12, 2024
@clee2000 @pytorchmergebot
Add inductor-cu124, inductor-rocm to upload test stats (#133143)
454713f 
Forgot to add them in #128250 and #131637

Pull Request resolved: #133143
Approved by: https://github.com/huydhn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@huydhn huydhn huydhn approved these changes

Assignees

No one assigned

Labels

ciflow/inductor-cu124 Merged topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@clee2000 @pytorchmergebot @nWEIdia @huydhn