Skip to content

Improve collective ops on numeric input#2959

Merged
vfdev-5 merged 3 commits intopytorch:masterfrom
sadra-barikbin:Improve-collective-ops-on-numeric-input
Jun 1, 2023
Merged

Improve collective ops on numeric input#2959
vfdev-5 merged 3 commits intopytorch:masterfrom
sadra-barikbin:Improve-collective-ops-on-numeric-input

Conversation

@sadra-barikbin
Copy link
Copy Markdown
Collaborator

Description:
When ComputationModel._collective_op_dtype is None and the input to a
collective op is a float number, the constructed intermediate tensor is of
dtype torch.float32 rather than torch.double, hence we would have a loss
in precision.

Check list:

  • New tests are added (if a new feature is added)
  • New doc strings: description and/or example code are in RST format
  • Documentation is updated (if required)

@github-actions github-actions bot added the module: distributed Distributed module label May 29, 2023
@vfdev-5
Copy link
Copy Markdown
Collaborator

vfdev-5 commented May 30, 2023

@sadra-barikbin where do you spot such precision loss ?

@sadra-barikbin
Copy link
Copy Markdown
Collaborator Author

@sadra-barikbin where do you spot such precision loss ?

When I attempted to remove the need to use pytest.approx in _test_distrib_on_output, I ended up here.
After this PR I saw that there was no need to approx and == worked even without this change, however this change is correct.

Copy link
Copy Markdown
Collaborator

@vfdev-5 vfdev-5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the PR @sadra-barikbin

@vfdev-5 vfdev-5 enabled auto-merge (squash) June 1, 2023 12:54
@vfdev-5 vfdev-5 disabled auto-merge June 1, 2023 13:24
@vfdev-5 vfdev-5 merged commit 4c83da1 into pytorch:master Jun 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: distributed Distributed module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants