Enabled bfloat16 for cuda #27259

izdeby · 2019-10-02T23:33:15Z

Enabled basic support for bfloat16 on cuda
Tested via unit tests

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ezyang

This diff looks very familiar to me. Is it a respin of a diff you did previously?

izdeby · 2019-10-03T14:53:50Z

aten/src/THC/THCNumerics.cuh

+  static inline __host__ __device__ bool isnan(at::BFloat16 a) {
+    #ifdef _MSC_VER
+      // Windows requires this explicit conversion. The reason is unclear
+      // related issue with clang: https://reviews.llvm.org/D37906
+      return ::isnan((float) a);
+    #else
+      return ::isnan(a);
+    #endif
+  }
+
+  static inline __host__ __device__ bool isinf(at::BFloat16 a) {
+    #ifdef _MSC_VER
+      // Windows requires this explicit conversion. The reason is unclear
+      // related issue with clang: https://reviews.llvm.org/D37906
+      return ::isinf((float) a);
+    #else
+      return ::isinf(a);
+    #endif
+  }
+};
+


@ezyang, this part

This diff looks very familiar to me. Is it a respin of a diff you did previously?
The diff is identical to the one that have been reverted except this part which fixes win build.

ezyang · 2019-10-03T15:01:32Z

aten/src/THC/THCNumerics.cuh

+    #ifdef _MSC_VER
+      // Windows requires this explicit conversion. The reason is unclear
+      // related issue with clang: https://reviews.llvm.org/D37906
+      return ::isnan((float) a);


nit: (you don't have to do anything about this, just fyi for later) generally people prefer static_cast<float> in C++ as it prevents the C style cast from, e.g., being treated like a reinterpret_cast

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

izdeby · 2019-10-21T18:56:24Z

@pytorchbot rebase this please

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

zasdfgbnm

@izdeby I see your fix merge conflict commit b34c8dd
you simply delete the BinaryOpsKernel.cu. Is this correct? This file has been split due to its long compilation time. I guess you might need to modify the split files separately. See: #29428

izdeby · 2019-11-13T22:54:18Z

@zasdfgbnm, thanks! good catch

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Summary: Enabled basic support for bfloat16 on cuda Tested via unit tests Pull Request resolved: pytorch/pytorch#27259 Differential Revision: D17728661 Pulled By: izdeby fbshipit-source-id: 99efb6bc4aec029fe6bbc8a68963dca9c9dc5810

facebook-github-bot · 2019-11-20T18:50:48Z

@izdeby merged this pull request in 36a47d7.

pytorchbot added module: build Build system issues module: cuda Related to torch.cuda, and CUDA support in general module: internals Related to internal abstractions in c10 and ATen module: operators module: tests Issues related to tests (not the torch.testing module) labels Oct 2, 2019

izdeby requested a review from ezyang October 2, 2019 23:33

facebook-github-bot reviewed Oct 2, 2019

View reviewed changes

ezyang approved these changes Oct 3, 2019

View reviewed changes

izdeby commented Oct 3, 2019

View reviewed changes

ezyang reviewed Oct 3, 2019

View reviewed changes

izdeby force-pushed the bf16cuda_2 branch from 6166c58 to 26a3db8 Compare October 3, 2019 15:25

facebook-github-bot reviewed Oct 3, 2019

View reviewed changes

izdeby force-pushed the bf16cuda_2 branch from b3a8ffa to 7f8cce8 Compare October 3, 2019 23:54

facebook-github-bot reviewed Oct 21, 2019

View reviewed changes

izdeby force-pushed the bf16cuda_2 branch from 3ba3420 to ab21766 Compare October 22, 2019 15:54

facebook-github-bot reviewed Oct 22, 2019

View reviewed changes

izdeby force-pushed the bf16cuda_2 branch from ab21766 to 3fb29a4 Compare October 29, 2019 15:45

facebook-github-bot reviewed Oct 29, 2019

View reviewed changes

izdeby force-pushed the bf16cuda_2 branch from 3fb29a4 to 60d4913 Compare November 7, 2019 16:39

facebook-github-bot reviewed Nov 7, 2019

View reviewed changes

facebook-github-bot reviewed Nov 8, 2019

View reviewed changes

facebook-github-bot reviewed Nov 11, 2019

View reviewed changes

izdeby force-pushed the bf16cuda_2 branch from b7a1ce1 to ff6d87f Compare November 13, 2019 19:20

facebook-github-bot reviewed Nov 13, 2019

View reviewed changes

zasdfgbnm reviewed Nov 13, 2019

View reviewed changes

facebook-github-bot reviewed Nov 13, 2019

View reviewed changes

Iurii Zdebskyi and others added 6 commits November 19, 2019 07:46

bfloat16 for cuda

47fa963

Fix

84c9a5b

Fix

73c1be7

Fixing lint

2c13d84

fixed merge

f6ac745

fix

0b06555

izdeby force-pushed the bf16cuda_2 branch from 5c1eb4f to 0b06555 Compare November 19, 2019 15:46

facebook-github-bot reviewed Nov 19, 2019

View reviewed changes

facebook-github-bot closed this in 36a47d7 Nov 20, 2019

facebook-github-bot added the merged label Nov 20, 2019

izdeby mentioned this pull request Nov 24, 2019

[ROCm] Add a first version of bf16 atomicAdd #30189

Closed

facebook-github-bot deleted the bf16cuda_2 branch July 13, 2020 17:53

mruberry added the Merged label Oct 28, 2020

Enabled bfloat16 for cuda #27259

Enabled bfloat16 for cuda #27259

Uh oh!

Conversation

izdeby commented Oct 2, 2019

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

izdeby Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

izdeby Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

ezyang Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

izdeby commented Oct 21, 2019

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

zasdfgbnm left a comment

Choose a reason for hiding this comment

Uh oh!

izdeby commented Nov 13, 2019

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Nov 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants