Skip to content

Speed up fill for half and bfloat16 on CPU. #28397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
xuhdev wants to merge 1 commit into from
Closed

Speed up fill for half and bfloat16 on CPU. #28397

xuhdev wants to merge 1 commit into from

Conversation

@xuhdev
Copy link
Collaborator

@xuhdev xuhdev commented Oct 22, 2019

This is done by replacing Vec<uint16_t> with Vec<int16_t>, which has all
sorts of AVX optimization available.

Benchmark (Debian 10, Release build, gcc 8.3, no turbo, Intel(R) Xeon(R) E-2136):

import timeit
for dtype in ('torch.bfloat16', 'torch.half'):
    for n, t in [(40_000, 600000),
                (400_000, 60000)]:
        print(f'a.fill_(10) for {t} times, a=torch.empty({n}, dtype={dtype})')
        print(timeit.timeit(f'a.fill_(10)', setup=f'import torch; a=torch.empty({n}, dtype={dtype})', number=t))

Before:

a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
11.064065577999827
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
10.618151295000189
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
10.989039544000207
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
10.602233665999847

After:

a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
1.530125006000162
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
1.4807136570002513
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
1.3946152990001792
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
1.457788402999995

@xuhdev xuhdev requested review from ifedan, izdeby and nairbv and removed request for ifedan October 22, 2019 02:09
nairbv
nairbv reviewed Oct 22, 2019
View reviewed changes
@xuhdev xuhdev requested a review from nairbv October 22, 2019 18:17
@xuhdev xuhdev force-pushed the speedup-fill branch 3 times, most recently from 7e383b3 to 87f31ea Compare October 22, 2019 20:20
@xuhdev xuhdev added module: cpu CPU specific problem (e.g., perf, algorithm) module: operators labels Oct 22, 2019
@xuhdev
Speed up fill for half and bfloat16 on CPU.
c447f61 
This is done by replacing Vec<uint16_t> with Vec<int16_t>, which has all
sorts of AVX optimization available.

Benchmark (Debian 10, Release build, gcc 8.3, no turbo, Intel(R) Xeon(R) E-2136):

```python
import timeit
for dtype in ('torch.bfloat16', 'torch.half'):
    for n, t in [(40_000, 600000),
                (400_000, 60000)]:
        print(f'a.fill_(10) for {t} times, a=torch.empty({n}, dtype={dtype})')
        print(timeit.timeit(f'a.fill_(10)', setup=f'import torch; a=torch.empty({n}, dtype={dtype})', number=t))
```

Before:

```
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
11.064065577999827
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
10.618151295000189
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
10.989039544000207
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
10.602233665999847
```

After:

```
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
1.530125006000162
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
1.4807136570002513
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
1.3946152990001792
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
1.457788402999995
```
@xuhdev
Copy link
Collaborator Author

xuhdev commented Oct 24, 2019

@pytorchbot merge this please

@pytorchbot pytorchbot added the merge-this-please Was marked for merge with @pytorchbot merge this please label Oct 24, 2019
facebook-github-bot
facebook-github-bot reviewed Oct 24, 2019
View reviewed changes
Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ezyang is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Oct 24, 2019

@ezyang merged this pull request in 5cf6441.

zdevito pushed a commit to zdevito/ATen that referenced this pull request Oct 24, 2019
@xuhdev @facebook-github-bot
Speed up fill for half and bfloat16 on CPU. (#28397)
9d4863c 
Summary:
This is done by replacing Vec<uint16_t> with Vec<int16_t>, which has all
sorts of AVX optimization available.

Benchmark (Debian 10, Release build, gcc 8.3, no turbo, Intel(R) Xeon(R) E-2136):

```python
import timeit
for dtype in ('torch.bfloat16', 'torch.half'):
    for n, t in [(40_000, 600000),
                (400_000, 60000)]:
        print(f'a.fill_(10) for {t} times, a=torch.empty({n}, dtype={dtype})')
        print(timeit.timeit(f'a.fill_(10)', setup=f'import torch; a=torch.empty({n}, dtype={dtype})', number=t))
```

Before:

```
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
11.064065577999827
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
10.618151295000189
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
10.989039544000207
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
10.602233665999847
```

After:

```
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
1.530125006000162
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
1.4807136570002513
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
1.3946152990001792
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
1.457788402999995
```
Pull Request resolved: pytorch/pytorch#28397

Differential Revision: D18125171

Pulled By: ezyang

fbshipit-source-id: bfb2da13f10bc582e9848073e428af9e36656b13
thiagocrepaldi pushed a commit to thiagocrepaldi/pytorch that referenced this pull request Feb 4, 2020
@xuhdev
Speed up fill for half and bfloat16 on CPU. (pytorch#28397)
ef666c0 
Summary:
This is done by replacing Vec<uint16_t> with Vec<int16_t>, which has all
sorts of AVX optimization available.

Benchmark (Debian 10, Release build, gcc 8.3, no turbo, Intel(R) Xeon(R) E-2136):

```python
import timeit
for dtype in ('torch.bfloat16', 'torch.half'):
    for n, t in [(40_000, 600000),
                (400_000, 60000)]:
        print(f'a.fill_(10) for {t} times, a=torch.empty({n}, dtype={dtype})')
        print(timeit.timeit(f'a.fill_(10)', setup=f'import torch; a=torch.empty({n}, dtype={dtype})', number=t))
```

Before:

```
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
11.064065577999827
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
10.618151295000189
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
10.989039544000207
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
10.602233665999847
```

After:

```
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.bfloat16)
1.530125006000162
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.bfloat16)
1.4807136570002513
a.fill_(10) for 600000 times, a=torch.empty(40000, dtype=torch.half)
1.3946152990001792
a.fill_(10) for 60000 times, a=torch.empty(400000, dtype=torch.half)
1.457788402999995
```
Pull Request resolved: pytorch#28397

Differential Revision: D18125171

Pulled By: ezyang

fbshipit-source-id: bfb2da13f10bc582e9848073e428af9e36656b13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@facebook-github-bot facebook-github-bot facebook-github-bot left review comments

@nairbv nairbv nairbv approved these changes

@izdeby izdeby Awaiting requested review from izdeby

Assignees

No one assigned

Labels

merge-this-please Was marked for merge with @pytorchbot merge this please Merged module: cpu CPU specific problem (e.g., perf, algorithm)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xuhdev @facebook-github-bot @nairbv @pytorchbot @mruberry