Skip to content

Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to ShardedTensor #67057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bowangbj wants to merge 5 commits into from
Closed

Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to ShardedTensor #67057

bowangbj wants to merge 5 commits into from

Conversation

@bowangbj
Copy link
Contributor

@bowangbj bowangbj commented Oct 21, 2021 •
edited
Loading

Stack from ghstack:

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D31845654

@bowangbj
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
649f2d0 
…hardedTensor

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
@pytorch-probot
Copy link

pytorch-probot bot commented Oct 21, 2021 •
edited
Loading

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/2f1458fee735861373f6b1c9fb5c2e78986e1b18/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default

Workflows Labels (bold enabled) Status
Triggered Workflows
linux-bionic-py3.6-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/noarch, ciflow/xla ✅ triggered
linux-vulkan-bionic-py3.6-clang9 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/vulkan ✅ triggered
linux-xenial-cuda11.3-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/default, ciflow/linux ✅ triggered
linux-xenial-py3-clang5-mobile-build ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile ✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-dynamic ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile ✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile ✅ triggered
linux-xenial-py3.6-clang7-asan ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/sanitizers ✅ triggered
linux-xenial-py3.6-clang7-onnx ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/onnx ✅ triggered
linux-xenial-py3.6-gcc5.4 ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux ✅ triggered
linux-xenial-py3.6-gcc7-bazel-test ciflow/all, ciflow/bazel, ciflow/cpu, ciflow/default, ciflow/linux ✅ triggered
win-vs2019-cpu-py3 ciflow/all, ciflow/cpu, ciflow/default, ciflow/win ✅ triggered
win-vs2019-cuda11.3-py3 ciflow/all, ciflow/cuda, ciflow/default, ciflow/win ✅ triggered
Skipped Workflows
libtorch-linux-xenial-cuda10.2-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux 🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux 🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/slow 🚫 skipped
linux-xenial-cuda10.2-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/slow 🚫 skipped
linux-xenial-py3-clang5-mobile-code-analysis ciflow/all, ciflow/linux, ciflow/mobile 🚫 skipped
parallelnative-linux-xenial-py3.6-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux 🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled, ciflow/slow, ciflow/slow-gradcheck 🚫 skipped
periodic-linux-xenial-cuda11.1-py3.6-gcc7 ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled 🚫 skipped
periodic-win-vs2019-cuda11.1-py3 ciflow/all, ciflow/cuda, ciflow/scheduled, ciflow/win 🚫 skipped
puretorch-linux-xenial-py3.6-gcc5.4 ciflow/all, ciflow/cpu, ciflow/linux 🚫 skipped

You can add a comment to the PR and tag @pytorchbot with the following commands: 
# ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun

# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slow

For more information, please take a look at the CI Flow Wiki.

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Oct 21, 2021 •
edited
Loading

🔗 Helpful links

💊 CI failures summary and remediations

As of commit 2f1458f (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Oct 21, 2021
@bowangbj
Copy link
Contributor Author

bowangbj commented Oct 21, 2021

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@bowangbj
lint on "Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_…
d926195 
… ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
@bowangbj
rebase on "Add torch.nn.init.normal_ and torch.nn.init.kaiming_unifor…
a7d74b4 
…m_ ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
bowangbj added a commit that referenced this pull request Oct 21, 2021
@bowangbj
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
1157707 
…hardedTensor

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: 012f671
Pull Request resolved: #67057
@bowangbj
Copy link
Contributor Author

bowangbj commented Oct 21, 2021

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

albanD
albanD reviewed Oct 22, 2021
View reviewed changes
@bowangbj
resolve albanD cmt on "Add torch.nn.init.normal_ and torch.nn.init.ka…
d913ea0 
…iming_uniform_ ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
bowangbj added a commit that referenced this pull request Oct 22, 2021
@bowangbj
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
73ec3e4 
…hardedTensor

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: 54ce4ba
Pull Request resolved: #67057
@bowangbj
Copy link
Contributor Author

bowangbj commented Oct 22, 2021

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@bowangbj bowangbj requested a review from albanD October 22, 2021 17:55
@bowangbj
Copy link
Contributor Author

bowangbj commented Oct 22, 2021

Thanks Alban for your super fast review, resolved all cmts, PTAL

albanD
albanD reviewed Oct 22, 2021
View reviewed changes
@bowangbj
albanD cmt keep tensor, on "Add torch.nn.init.normal_ and torch.nn.in…
2f1458f 
…it.kaiming_uniform_ ops to ShardedTensor"

Summary:
Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D31845654](https://our.internmc.facebook.com/intern/diff/D31845654)

[ghstack-poisoned]
@bowangbj bowangbj mentioned this pull request Oct 22, 2021
Closed
@bowangbj
Copy link
Contributor Author

bowangbj commented Oct 22, 2021

@bowangbj has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@bowangbj bowangbj requested a review from albanD October 22, 2021 19:18
@bowangbj
Copy link
Contributor Author

bowangbj commented Oct 22, 2021

Thanks Alban for the super quick review, resolved ur cmts, PTAL

albanD
albanD reviewed Oct 22, 2021
View reviewed changes
Copy link
Collaborator

@albanD albanD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nn part looks good.
I'll let the distributed people comment on the rest.

pritamdamania87
pritamdamania87 approved these changes Oct 26, 2021
View reviewed changes
Copy link
Contributor

@pritamdamania87 pritamdamania87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

facebook-github-bot pushed a commit that referenced this pull request Oct 26, 2021
@bowangbj @facebook-github-bot
Add torch.nn.init.normal_ and torch.nn.init.kaiming_uniform_ ops to S…
3596e13 
…hardedTensor (#67057)

Summary:
Pull Request resolved: #67057

Extend ShardedTensor with torch.nn.init.[normal_, and kaiming_uniform_] ops
Follow up from #63997

Test Plan:
a) Unit Test
(pytorch) ... $ python test/distributed/_sharded_tensor/ops/test_init.py TestShardedTensorNNInit --v

or b) Manual run: Instruction here: https://docs.google.com/document/d/1_m1Hdo5w51-hhPlZ_F8Y6PIWrN7UgJZqiSpARYvhsaE/edit#
s/uniform_/normal_ or kaiming_uniform_

Imported from OSS

Reviewed By: pritamdamania87

Differential Revision: D31845654

fbshipit-source-id: e7aedc0972539da59f7b84bbbf617caf6b206d52
@facebook-github-bot facebook-github-bot deleted the gh/bowangbj/12/head branch November 25, 2021 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@albanD albanD albanD left review comments

@wanchaol wanchaol Awaiting requested review from wanchaol

@fduwjj fduwjj Awaiting requested review from fduwjj

+1 more reviewer

@pritamdamania87 pritamdamania87 pritamdamania87 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla signed oncall: distributed Add this issue/PR to distributed oncall triage queue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bowangbj @facebook-github-bot @albanD @pritamdamania87