Skip to content

Sparse half embeddings on cuda #19695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nairbv wants to merge 21 commits into from
Closed

Sparse half embeddings on cuda #19695

nairbv wants to merge 21 commits into from

Conversation

@nairbv
Copy link
Collaborator

@nairbv nairbv commented Apr 24, 2019 

import torch
a = torch.nn.Embedding(3, 4, sparse=True).half().cuda()
a(torch.LongTensor([1, 0]).cuda()).sum().backward()

gave: RuntimeError: torch.cuda.sparse.HalfTensor is not enabled

This PR enables sparse.HalfTensor on cuda. Still won't work for CPU.

@pytorchbot pytorchbot added module: internals Related to internal abstractions in c10 and ATen module: nn Related to torch.nn labels Apr 24, 2019
@nairbv
Copy link
Collaborator Author

nairbv commented Apr 29, 2019

@pytorchbot retest this please

@nairbv nairbv requested review from gchanan and zou3519 April 30, 2019 19:29
zou3519
zou3519 reviewed Apr 30, 2019
View reviewed changes
zou3519
zou3519 reviewed Apr 30, 2019
View reviewed changes
zou3519
zou3519 reviewed Apr 30, 2019
View reviewed changes
zou3519
zou3519 reviewed Apr 30, 2019
View reviewed changes
gchanan
gchanan requested changes May 1, 2019
View reviewed changes
gchanan
gchanan requested changes May 1, 2019
View reviewed changes
@nairbv nairbv mentioned this pull request May 1, 2019
Open
@nairbv nairbv mentioned this pull request May 2, 2019
Closed
gchanan
gchanan requested changes May 2, 2019
View reviewed changes
@nairbv
Copy link
Collaborator Author

nairbv commented May 6, 2019

@pytorchbot rebase this please

@pytorchbot
Copy link
Collaborator

pytorchbot commented May 6, 2019

Sorry, I can't merge this because there are conflicts. To merge this yourself, run the commands below:

git fetch origin master
git fetch [email protected]:nairbv/pytorch.git sparse_half_tensors
git checkout FETCH_HEAD
git merge origin/master
git push [email protected]:nairbv/pytorch.git HEAD:sparse_half_tensors

(To learn more about this bot, see Bot commands.)

@gchanan
Copy link
Contributor

gchanan commented May 7, 2019 •
edited
Loading

summary from in-person discussion:
for now, let's punt on providing math ops on float16 on CPU, which means we won't be able to do:

  1. coalesce() on a CPU,Half,Sparse tensor, because the semantics of coalesce are to add the elements together that share indices.
  2. sparse_to_dense on CPU,Half,Sparse because it is currently implemented as zeros(dense) + sparse.

But this will allow us to train sparse half embeddings on cuda, which is the high priority ask.

@nairbv
Reverting to_dense and add support on half/cpu
22babf1 
to_dense requires add_. add is much slower than float for half types on CPU.
gchanan
gchanan requested changes May 7, 2019
View reviewed changes
@nairbv
Copy link
Collaborator Author

nairbv commented May 8, 2019

@pytorchbot restest this please

gchanan
gchanan requested changes May 8, 2019
View reviewed changes
@nairbv
Copy link
Collaborator Author

nairbv commented May 9, 2019

@pytorchbot retest this please

facebook-github-bot
facebook-github-bot reviewed May 9, 2019
View reviewed changes
Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nairbv is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

zdevito pushed a commit to zdevito/ATen that referenced this pull request May 10, 2019
@nairbv @facebook-github-bot
Sparse half embeddings on cuda (#19695)
00dbc5a 
Summary:
```
import torch
a = torch.nn.Embedding(3, 4, sparse=True).half().cuda()
a(torch.LongTensor([1, 0]).cuda()).sum().backward()

```
gave: `RuntimeError: torch.cuda.sparse.HalfTensor is not enabled`

This PR enables sparse.HalfTensor on cuda. Still won't work for CPU.
Pull Request resolved: pytorch/pytorch#19695

Differential Revision: D15281162

Pulled By: nairbv

fbshipit-source-id: 0d83d946a059393bd53d8b8102e2daa9b4c02588
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented May 10, 2019

@nairbv merged this pull request in d68802b.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@facebook-github-bot facebook-github-bot facebook-github-bot left review comments

@gchanan gchanan gchanan approved these changes

@zou3519 zou3519 zou3519 approved these changes

Assignees

No one assigned

Labels

module: internals Related to internal abstractions in c10 and ATen module: nn Related to torch.nn

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nairbv @pytorchbot @gchanan @facebook-github-bot @zou3519