[Pipelining] Update schedules to use I, B actions. #138886

wconstab · 2024-10-25T03:44:50Z

Stack from ghstack (oldest at bottom):

Also, update tests to use I (BACKWARD_INPUT) vs B (FULL_BACKWARD)
consistently.

Previously, schedules would issue a 'B' operation and leave it ambiguous
whether that operation should be BACKWARD_INPUT or FULL_BACKWARD,
depending on a separate flag (use_full_backward) passed to the schedule
class, which would determine which behavior was taken at runtime.

Now, use_full_backward is removed and the schedule class is required to
produce unambiguous IR. The logic for 'use_full_backward' is removed
from the runtime.

_validate_pipeline_order is replaced with _simulate_comms_compute. Both
offer similar functionality, to validate the corrrectness of a schedule
IR. 'validate' operates on compute-only IR, while simulate operates on
compute + comm IR. To convert from using validate to simulate, you have
to first insert comm actions via '_add_send_recv'.

'simulate' was inefficiently written before this PR and needed to be
optimized to run quickly for extra large schedules with >32 ranks and
microbatches per rank used in some unit tests.

cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @d4l3k @c-p-i-o

[ghstack-poisoned]

pytorch-bot · 2024-10-25T03:44:53Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138886

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[DomainsOnly] Jobs fail with GLIBC version not found

❌ 1 New Failure

As of commit 8bf3e63 with merge base failed to retrieve merge base, please contact dev infra:

NEW FAILURE - The following job has failed:

pull / cuda12.1-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh)
REGRESSION: benchmark ('basic_modules_ListOfLinears_eager', 'compile_time_instruction_count') failed, actual result 1053032414 is 2.44% higher than expected 1028000000 ±+1.50% if this is an expected regression, please update the expected results.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Also, update tests to use B, I consistently. Consider replacing '_validate_pipeline_order' with '_simulate_comms_compute' since they are somewhat duplicate functionality, and it is also annoying to update 'validate' to handle seprate BACKWARD_INPUT. Validate is different in that it considers the compute-only schedule, while simulate considers the compute+comms schedule. However, simulate is too slow for some of the unit tests with extra large numbers of ranks and microbatches, and is also failing to execute some of the schedules. After fixing the failures, I will see if it is easy to optimize the simulator or not. - Optimized simulate and add_send_recv, tests are fast now even at large scales - need to run other tests and do cleanup work, then this should be ready to land ghstack-source-id: eb263f2 Pull Request resolved: #138886

[ghstack-poisoned]

Also, update tests to use I (BACKWARD_INPUT) vs B (FULL_BACKWARD) consistently. Previously, schedules would issue a 'B' operation and leave it ambiguous whether that operation should be BACKWARD_INPUT or FULL_BACKWARD, depending on a separate flag (use_full_backward) passed to the schedule class, which would determine which behavior was taken at runtime. Now, use_full_backward is removed and the schedule class is required to produce unambiguous IR. The logic for 'use_full_backward' is removed from the runtime. _validate_pipeline_order is replaced with _simulate_comms_compute. Both offer similar functionality, to validate the corrrectness of a schedule IR. 'validate' operates on compute-only IR, while simulate operates on compute + comm IR. To convert from using validate to simulate, you have to first insert comm actions via '_add_send_recv'. 'simulate' was inefficiently written before this PR and needed to be optimized to run quickly for extra large schedules with >32 ranks and microbatches per rank used in some unit tests. ghstack-source-id: e873252 Pull Request resolved: #138886

[ghstack-poisoned]

Also, update tests to use I (BACKWARD_INPUT) vs B (FULL_BACKWARD) consistently. Previously, schedules would issue a 'B' operation and leave it ambiguous whether that operation should be BACKWARD_INPUT or FULL_BACKWARD, depending on a separate flag (use_full_backward) passed to the schedule class, which would determine which behavior was taken at runtime. Now, use_full_backward is removed and the schedule class is required to produce unambiguous IR. The logic for 'use_full_backward' is removed from the runtime. _validate_pipeline_order is replaced with _simulate_comms_compute. Both offer similar functionality, to validate the corrrectness of a schedule IR. 'validate' operates on compute-only IR, while simulate operates on compute + comm IR. To convert from using validate to simulate, you have to first insert comm actions via '_add_send_recv'. 'simulate' was inefficiently written before this PR and needed to be optimized to run quickly for extra large schedules with >32 ranks and microbatches per rank used in some unit tests. ghstack-source-id: a5a3c61 Pull Request resolved: #138886

H-Huang

LGTM

[ghstack-poisoned]

Also, update tests to use I (BACKWARD_INPUT) vs B (FULL_BACKWARD) consistently. Previously, schedules would issue a 'B' operation and leave it ambiguous whether that operation should be BACKWARD_INPUT or FULL_BACKWARD, depending on a separate flag (use_full_backward) passed to the schedule class, which would determine which behavior was taken at runtime. Now, use_full_backward is removed and the schedule class is required to produce unambiguous IR. The logic for 'use_full_backward' is removed from the runtime. _validate_pipeline_order is replaced with _simulate_comms_compute. Both offer similar functionality, to validate the corrrectness of a schedule IR. 'validate' operates on compute-only IR, while simulate operates on compute + comm IR. To convert from using validate to simulate, you have to first insert comm actions via '_add_send_recv'. 'simulate' was inefficiently written before this PR and needed to be optimized to run quickly for extra large schedules with >32 ranks and microbatches per rank used in some unit tests. ghstack-source-id: 8c833a3 Pull Request resolved: #138886

wconstab · 2024-10-31T18:14:29Z

@pytorchbot merge

pytorchmergebot · 2024-10-31T18:16:15Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-10-31T18:21:43Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, lf.linux.g4dn.12xlarge.nvidia.gpu)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

[ghstack-poisoned]

wconstab · 2024-11-01T03:46:49Z

@pytorchbot merge -i

pytorchmergebot · 2024-11-01T03:48:28Z

Merge started

Your change will be merged while ignoring the following 1 checks: pull / cuda12.1-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Also, update tests to use I (BACKWARD_INPUT) vs B (FULL_BACKWARD) consistently. Previously, schedules would issue a 'B' operation and leave it ambiguous whether that operation should be BACKWARD_INPUT or FULL_BACKWARD, depending on a separate flag (use_full_backward) passed to the schedule class, which would determine which behavior was taken at runtime. Now, use_full_backward is removed and the schedule class is required to produce unambiguous IR. The logic for 'use_full_backward' is removed from the runtime. _validate_pipeline_order is replaced with _simulate_comms_compute. Both offer similar functionality, to validate the corrrectness of a schedule IR. 'validate' operates on compute-only IR, while simulate operates on compute + comm IR. To convert from using validate to simulate, you have to first insert comm actions via '_add_send_recv'. 'simulate' was inefficiently written before this PR and needed to be optimized to run quickly for extra large schedules with >32 ranks and microbatches per rank used in some unit tests. Pull Request resolved: pytorch#138886 Approved by: https://github.com/H-Huang

Also, update tests to use I (BACKWARD_INPUT) vs B (FULL_BACKWARD) consistently. Previously, schedules would issue a 'B' operation and leave it ambiguous whether that operation should be BACKWARD_INPUT or FULL_BACKWARD, depending on a separate flag (use_full_backward) passed to the schedule class, which would determine which behavior was taken at runtime. Now, use_full_backward is removed and the schedule class is required to produce unambiguous IR. The logic for 'use_full_backward' is removed from the runtime. _validate_pipeline_order is replaced with _simulate_comms_compute. Both offer similar functionality, to validate the corrrectness of a schedule IR. 'validate' operates on compute-only IR, while simulate operates on compute + comm IR. To convert from using validate to simulate, you have to first insert comm actions via '_add_send_recv'. 'simulate' was inefficiently written before this PR and needed to be optimized to run quickly for extra large schedules with >32 ranks and microbatches per rank used in some unit tests. ghstack-source-id: 2fffcd7 Pull Request resolved: pytorch/pytorch#138886

Update

4ac6630

[ghstack-poisoned]

wconstab mentioned this pull request Oct 25, 2024

[Pipelining] add schedule simulator and chrometrace dump #138134

Closed

This was referenced Oct 25, 2024

[Pipelining] Relax multi-stage constraint #138862

Closed

[Pipelining] Support separate dI / dW and V-schedules #131762

Closed

pytorch-bot bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Oct 25, 2024

This was referenced Oct 25, 2024

Debug test failure for separate I, W execution #138863

Closed

[Pipelining] Support V-schedules in IR and Runtime #138125

Closed

Update

6da3ea0

[ghstack-poisoned]

wconstab changed the title ~~WIP Update schedules to use BACKWARD_INPUT / FULL_BACKWARD~~ [Pipelining] Update schedules to use I, B actions. Oct 25, 2024

wconstab added release notes: distributed (pipeline) release notes category module: pipelining Pipeline Parallelism labels Oct 25, 2024

Update

0432ea6

[ghstack-poisoned]

wconstab mentioned this pull request Oct 25, 2024

[Pipelining] Optimize ready_to_schedule logic #138924

Closed

Update

a8ae826

[ghstack-poisoned]

wconstab mentioned this pull request Oct 25, 2024

[Pipelining] Remove unused special case from simulator #138928

Closed

Update

4ffc700

[ghstack-poisoned]

wconstab requested review from H-Huang and kwen2501 October 30, 2024 19:44

Update

06c2821

[ghstack-poisoned]

Update

b649ff7

[ghstack-poisoned]

H-Huang approved these changes Oct 31, 2024

View reviewed changes

Update

b2b96dc

[ghstack-poisoned]

Update

9801352

[ghstack-poisoned]

Update

b17270a

[ghstack-poisoned]

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 31, 2024

pytorchmergebot added the merging label Oct 31, 2024

pytorchmergebot removed the merging label Oct 31, 2024

wconstab mentioned this pull request Oct 31, 2024

[pipelining] Fix 'last backward' counting for dI / dW #139415

Closed

wconstab added 2 commits October 31, 2024 16:09

Update

37273e3

[ghstack-poisoned]

Update

8bf3e63

[ghstack-poisoned]

wconstab mentioned this pull request Nov 1, 2024

[pipelining] RNG seed management for PP init #139445

Closed

pytorchmergebot added the merging label Nov 1, 2024

pytorchmergebot closed this in 8441661 Nov 1, 2024

pytorchmergebot added Merged and removed merging labels Nov 1, 2024

github-actions bot deleted the gh/wconstab/351/head branch December 22, 2024 02:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Pipelining] Update schedules to use I, B actions. #138886

[Pipelining] Update schedules to use I, B actions. #138886

Uh oh!

wconstab commented Oct 25, 2024 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Oct 25, 2024 •

edited

Loading

Uh oh!

H-Huang left a comment

Uh oh!

wconstab commented Oct 31, 2024

Uh oh!

pytorchmergebot commented Oct 31, 2024

Uh oh!

pytorchmergebot commented Oct 31, 2024

Uh oh!

wconstab commented Nov 1, 2024

Uh oh!

pytorchmergebot commented Nov 1, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Pipelining] Update schedules to use I, B actions. #138886

[Pipelining] Update schedules to use I, B actions. #138886

Uh oh!

Conversation

wconstab commented Oct 25, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138886

❗ 1 Active SEVs

❌ 1 New Failure

Uh oh!

H-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

wconstab commented Oct 31, 2024

Uh oh!

pytorchmergebot commented Oct 31, 2024

Merge started

Uh oh!

pytorchmergebot commented Oct 31, 2024

Merge failed

Uh oh!

wconstab commented Nov 1, 2024

Uh oh!

pytorchmergebot commented Nov 1, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wconstab commented Oct 25, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Oct 25, 2024 •

edited

Loading