skip_input for RNN #894

SeanNaren · 2017-03-01T22:59:06Z

Should be ready to go! Added skip_input for RNNs (look here for information). Let me know of any feedback etc!

…_input_2

test/test_nn.py

            for x_layer, y_layer in zip(rnn.all_weights, weights_val):
                for x, y in zip(x_layer, y_layer):
-                    x.data.copy_(y.data)
+                    if x is not None and y is not None:


test/test_nn.py

-                    grad_output = torch.randn(batch, seq_length, hidden_size * num_directions)
+                grad_output = torch.randn(seq_length, batch, hidden_size * num_directions)
+                if skip_input:
+                    input_val = torch.randn(seq_length, batch, hidden_size)


torch/backends/cudnn/rnn.py

        for param_from, param_to in zip(layer_params_from, layer_params_to):
-            assert param_from.type() == param_to.type()
-            param_to.copy_(param_from)
+            if param_from is not None and param_to is not None:


torch/nn/_functions/rnn.py

    gh = F.linear(hidden, w_hh, b_hh)
-    i_r, i_i, i_n = gi.chunk(3, 1)
+    i_r, i_i, i_n = [x.squeeze(1) for x in gi.chunk(3, 1)]
    h_r, h_i, h_n = gh.chunk(3, 1)


torch/nn/_functions/rnn.py

                grad_weight)
+            if self.skip_input:
+                grad_weight = [tuple(w for w in layer_grad_weight if w is not None)
+                               for layer_grad_weight in grad_weight]


torch/nn/_functions/rnn.py

    hx, cx = hidden
-    gates = F.linear(input, w_ih, b_ih) + F.linear(hx, w_hh, b_hh)
-    ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)
+    x_h = input.unsqueeze(1).expand(input.size(0), 4, input.size(1)) if w_ih is None else F.linear(input, w_ih, b_ih)


torch/nn/_functions/rnn.py

-    ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)
+    x_h = input.unsqueeze(1).expand(input.size(0), 4, input.size(1)) if w_ih is None else F.linear(input, w_ih, b_ih)
+    gates = x_h + F.linear(hx, w_hh, b_hh)
+    ingate, forgetgate, cellgate, outgate = [x.squeeze(1) for x in gates.chunk(4, 1)]


torch/nn/_functions/rnn.py

+    gi = input.unsqueeze(1).expand(input.size(0), 3, input.size(1)) if w_ih is None else F.linear(input, w_ih, b_ih)
    gh = F.linear(hidden, w_hh, b_hh)
-    i_r, i_i, i_n = gi.chunk(3, 1)
+    i_r, i_i, i_n = [x.squeeze(1) for x in gi.chunk(3, 1)]


# Conflicts: # torch/nn/_functions/rnn.py # torch/tensor.py

SeanNaren · 2017-05-05T16:50:17Z

Back on trying to fix this, making some progress! I'm uncertain how to deal with the current issue; the bias is still being added in cudnn v6 on the input layer, when skip input is set to true which isn't the correct behaviour. @apaszke @ngimel what do you think is the best solution for this? (refer here for more info on this issue)!

* Fixes for skip rnn * Fixes for RNN cells, patch cuDNN for true skip input behaviour

SeanNaren · 2017-05-12T15:55:42Z

@apaszke, tests are passing but needs a review! Let me know of any feedback :)

apaszke

Looks good for the most part!

test/test_nn.py

                    grad_output = make_noncontig(grad_output)
                    grad_hy = make_noncontig(grad_hy)
-                    input_var = make_noncontig(input_val)
+                    input_val = make_noncontig(input_val)


torch/backends/cudnn/rnn.py

-            assert param_from.type() == param_to.type()
-            param_to.copy_(param_from)
+            assert not ((param_from is None or param_from.dim() == 0) ^ (param_to is None or param_to.dim() == 0))
+            if not ((param_from is None or param_from.dim() == 0) and (param_to is None or param_to.dim() == 0)):


torch/backends/cudnn/rnn.py

+        if fn.skip_input:
+            params = get_parameters(fn, handle, w)
+            for layer_index in range(fn.num_directions):
+                params[layer_index][2].fill_(0)


torch/backends/cudnn/rnn.py

+        if fn.skip_input:
+            for layer_index in range(fn.num_directions):
+                params[layer_index][0] = None
+                params[layer_index][2] = None


torch/nn/_functions/rnn.py

 def LSTMCell(input, hidden, w_ih, w_hh, b_ih=None, b_hh=None):
    if input.is_cuda:
-        igates = F.linear(input, w_ih)
+        igates = input.expand(4, input.size(0), input.size(1)).transpose(0, 1) if w_ih is None else F.linear(input,


torch/nn/_functions/rnn.py

        return state(gi, gh, hidden) if b_ih is None else state(gi, gh, hidden, b_ih, b_hh)

-    gi = F.linear(input, w_ih, b_ih)
+    gi = input.expand(3, input.size(0), input.size(1)).transpose(0, 1).contiguous() if w_ih is None else \


torch/nn/_functions/rnn.py

-    i_r, i_i, i_n = gi.chunk(3, 1)
-    h_r, h_i, h_n = gh.chunk(3, 1)
+    i_r, i_i, i_n = torch.unbind(gi.view(input.size(0), 3, -1), 1)
+    h_r, h_i, h_n = torch.unbind(gh.view(input.size(0), 3, -1), 1)


SeanNaren · 2017-05-15T17:34:17Z

Made some changes as requested, but still have to figure out the unbind and fused rnn stuff!

SeanNaren · 2017-05-17T10:19:59Z

Hey @apaszke any thoughts/feedback :)

EDIT: Removed unbind commands with chunk

SeanNaren · 2017-05-21T16:28:17Z

I've modified the line to now work with chunks rather than unbind as previously implemented!

SeanNaren · 2017-05-30T12:11:18Z

Can I get a status on the PR (blocking some deep speech stuff) :)

SeanNaren · 2017-06-05T10:03:14Z

After speaking to peeps I'm going to close this PR in favour of a correct implementation of skip input until cuDNN addresses this! thanks @justinchiu :)

* Fix placement of block sync with halo loop * hdiff test

* Cherrypicked the changes from pytorch#71146

SeanNaren added 6 commits March 1, 2017 10:21

Added skip_input functionality

209f9b4

Removed unsqueeze operations

a65a03b

Merge branch 'master' of https://github.com/pytorch/pytorch into skip…

c26f604

…_input_2

Fixed repeat for boolean vars

0301bec

Fixed variable sets

6c3e8e9

Added unsqueeze operations to rnn before expansion

e8c63da

SeanNaren mentioned this pull request Mar 1, 2017

Added skip_input param to RNNs #658

Closed

Fix formatting

cccdab6

apaszke reviewed Mar 3, 2017

View reviewed changes

justinchiu reviewed Mar 15, 2017

View reviewed changes

adamlerer reviewed Mar 15, 2017

View reviewed changes

ryanleary mentioned this pull request Apr 26, 2017

Support Batch RNNs SeanNaren/deepspeech.pytorch#4

Closed

SeanNaren added 3 commits May 3, 2017 09:57

Feedback changes

31e38c9

Feedback changes

0dacccb

Merge remote-tracking branch 'upstream/master' into skip_input_2

7ec06ea

# Conflicts: # torch/nn/_functions/rnn.py # torch/tensor.py

Sean Naren added 2 commits May 12, 2017 15:15

Skip input 2 changes rebase (#5)

53e92c1

* Fixes for skip rnn * Fixes for RNN cells, patch cuDNN for true skip input behaviour

Fix formatting, remove expand method (#6)

01f0df3

apaszke reviewed May 14, 2017

View reviewed changes

Feedback changes

f50de2f

Fixed lint check

2577d27

Use chunk instead of unbind in cells

c167fba

Merge remote-tracking branch 'upstream/master' into skip_input_2

74b97a4

Merge remote-tracking branch 'upstream/master' into skip_input_2

3cffe76

SeanNaren closed this Jun 5, 2017

ezyang added the open source label Jun 24, 2019

jjsjann123 pushed a commit to jjsjann123/pytorch that referenced this pull request Aug 5, 2021

Fix placement of block sync with halo loop (pytorch#894)

9c97206

* Fix placement of block sync with halo loop * hdiff test

jaglinux pushed a commit to jaglinux/pytorch that referenced this pull request Feb 16, 2022

Add hasPrimaryContext() commits (pytorch#894)

97a8a89

* Cherrypicked the changes from pytorch#71146

zhuhong61 pushed a commit to zhuhong61/pytorch that referenced this pull request Jun 8, 2022

Upgrade CUDA_UPGRADE_GUIDE (pytorch#894)

fafd101

skip_input for RNN #894

skip_input for RNN #894

Uh oh!

Conversation

SeanNaren commented Mar 1, 2017

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

SeanNaren commented May 5, 2017

Uh oh!

SeanNaren commented May 12, 2017

Uh oh!

apaszke left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

SeanNaren commented May 17, 2017 •

edited

Loading