added initialization schemes in torch.nn.init #833

alykhantejani · 2017-02-23T10:22:39Z

Ported initialization schemed from https://github.com/alykhantejani/nninit to torch.nn.init
Added the TestNNInit TestCase to test_nn.py
small PEP8 fixes (removed whitespace etc.)

Once merged we can probably close #101

torch/nn/init.py

+        return tensor
+    else:
+        fan_in, _ = _calculate_fan_in_and_fan_out(tensor)
+        std = gain * np.sqrt(1.0 / fan_in)


alykhantejani · 2017-02-23T16:21:50Z

Darn, looks like the build fails because scipy isn't installed. I use these to do goodness of fit tests on the returned weights to a particular distribution. Do you know of alternative package I can use for this, if we don't want to have scipy as a requirement?

soumith · 2017-02-23T17:22:44Z

@alykhantejani you can keep scipy parts in the tests. Do the following:

at the top of test_nn.py add these lines:

TEST_SCIPY = True
try:
    import scipy
except ImportError:
    TEST_SCIPY = False

Then, in the tests where you have scipy references, to the tests, add the annotation:

@unittest.skipIf(not TEST_SCIPY, "SCIPY unavailable")
Here's an example: https://github.com/pytorch/pytorch/blob/master/test/test_nn.py#L673

Then, here: https://github.com/pytorch/pytorch/blob/master/.travis.yml#L21
Add the line: - travis_retry pip install scipy

torch/nn/init.py

+        return tensor.normal_(0, std)
+
+
+def kaiming_uniform(tensor, gain=1):


torch/nn/init.py

+        return tensor
+    else:
+        fan_in, _ = _calculate_fan_in_and_fan_out(tensor)
+        std = gain * np.sqrt(1.0 / fan_in)


test/test_nn.py

+                    expected_std = gain * np.sqrt(2.0 / ((tensor_shape[1] + tensor_shape[0]) * receptive_field))
+                    assert self._is_normal(input_tensor, 0, expected_std)
+
+    def test_kaiming_unifrom_errors_on_inputs_smaller_than_2d(self):


…nn_init

alykhantejani · 2017-02-24T12:12:01Z

@soumith The pip install scipy in .travis.yaml seems to be failing for the 2.x builds. Any idea why?

soumith · 2017-02-24T12:15:35Z

hmmm, googling around seems to be a known issue.
Can you try:

https://github.com/pfnet/chainerrl/blob/5a39a487e84962d84876603b7892dd5be0320078/.travis.yml#L8-L9

or:
https://github.com/DistrictDataLabs/yellowbrick/blob/master/.travis.yml#L7-L8

colesbury

This looks pretty good. I think the most important thing is to remove the numpy dependency, since PyTorch does not currently require numpy.

torch/nn/init.py

+        return tensor
+    else:
+        fan_in, _ = _calculate_fan_in_and_fan_out(tensor)
+        std = gain * np.sqrt(1.0 / fan_in)


torch/nn/init.py

@@ -1,0 +1,240 @@
+import numpy as np


torch/nn/init.py

+    """Fills the input Tensor or Variable with values according to the method described in "Understanding the difficulty of training
+       deep feedforward neural networks" - Glorot, X. and Bengio, Y., using a uniform distribution.
+
+       The resulting tensor will have values sampled from U(-a, a) where a = gain * sqrt(2/(fan_in + fan_out))


…slope of the rectifier

torch/nn/init.py

        num_input_fmaps = tensor.size(1)
        num_output_fmaps = tensor.size(0)
-        receptive_field_size = np.prod(tensor.numpy().shape[2:])
+        receptive_field_size = reduce(mul, (tensor.numpy().shape[2:]))


torch/nn/init.py

            raise ValueError("Only tensors with 2 or more dimensions are supported.")

-        flattened_shape = (tensor.size(0), int(np.prod(tensor.numpy().shape[1:])))
+        flattened_shape = (tensor.size(0), int(reduce(mul, tensor.numpy().shape[1:])))


test/test_nn.py

                    if rows > cols:
-                        assert np.allclose(np.dot(flattened_tensor.T, flattened_tensor), np.eye(cols) * gain ** 2,
-                                           atol=1e-6)
+                        assert torch.dist(torch.mm(flattened_tensor.t(), flattened_tensor),


torch/nn/init.py

+    else:
+        num_input_fmaps = tensor.size(1)
+        num_output_fmaps = tensor.size(0)
+        receptive_field_size = reduce(mul, (tensor.numpy().shape[2:]))


torch/nn/init.py

+    if isinstance(tensor, Variable):
+        uniform(tensor.data, a=a, b=b)
+        return tensor
+    else:


…IPY to test/common

alykhantejani · 2017-02-26T10:41:53Z

I've answered most reviews, but I still think there are the following open questions:

As @szagoruyko pointed out, in Kaiming initialization, there are two variants where one divides by nInputPlane*kw*kh, another by nOutputPlane*kw*kh (see here). The method in this PR divides by nInputPlane*kw*kh (fan_in). Do we want to support both? If so, what's the best way to modify the interface i.e. should we take an additional param use_fan_out=False?
Should we remove the gain parameter from the xavier_* functions? I initially added it as this is what the Lasagne implementation has (used as reference). However, now not sure it's needed or useful?
Is it still the preferred method to rebase and squash all commits? If so, I will do this once we are all happy (so no more commits go in)

apaszke · 2017-02-26T11:10:11Z

I think adding a use_fan_out flag makes sense. But if Kaiming says that he used nOutputPlane, I think we should use that as the default.
No strong opinions on that. I can't see the value, but I don't have a problem either. It's already implemented so I think we can leave it.
No need to squash yourself, we can do that on GitHub while merging the PR.

test/test_nn.py

+
+                    fan_in = input_tensor.size(1)
+                    if input_tensor.dim() > 2:
+                        fan_in *= input_tensor[0][0].numel()


szagoruyko · 2017-02-26T11:26:15Z

regarding fan_out, fan_in stabilizes forward activations, fan_out stabilizes gradients on backward, so it depends on architecture which one works better.
I think we should have a string argument so that we can extend it in future, eg. tf also has fan_avg = (fan_in + fan_out) / 2

apaszke · 2017-02-26T12:34:12Z

String arguments sound good to me.

alykhantejani · 2017-02-28T16:10:21Z

@szagoruyko @apaszke I've now added a mode kwarg to the kaiming_* initializers.

apaszke · 2017-03-01T17:43:59Z

@pytorchbot add to whitelist

apaszke · 2017-03-01T18:34:18Z

Thank you!

Co-authored-by: Ryan Spring <[email protected]>

added initialization schemes in torch.nn.init

6c88e7d

szagoruyko reviewed Feb 23, 2017

View reviewed changes

colesbury reviewed Feb 23, 2017

View reviewed changes

alykhantejani commented Feb 23, 2017

View reviewed changes

test/test_nn.py Outdated

expected_std = gain * np.sqrt(2.0 / ((tensor_shape[1] + tensor_shape[0]) * receptive_field))

assert self._is_normal(input_tensor, 0, expected_std)

def test_kaiming_unifrom_errors_on_inputs_smaller_than_2d(self):

This comment was marked as off-topic.

Sign in to view

alykhantejani added 4 commits February 24, 2017 09:24

fix typo in test name

dd191b5

Merge branch 'master' of https://github.com/pytorch/pytorch into add_…

5b3a216

…nn_init

use default gain of sqrt(2) for kaiming initialization

6c03d3b

added skipIf annotations on tests that depend on scipy

b490c27

add install of binary scipy to .travis.yaml

44b7f9c

alykhantejani force-pushed the add_nn_init branch from 6be9020 to 44b7f9c Compare February 24, 2017 12:40

colesbury reviewed Feb 24, 2017

View reviewed changes

alykhantejani added 2 commits February 25, 2017 11:00

remove usage of numpy in torch.nn.init

b191412

changed gain param in kaiming intializaers to the coefficient of the …

1c4f7ef

…slope of the rectifier

fmassa reviewed Feb 25, 2017

View reviewed changes

apaszke reviewed Feb 25, 2017

View reviewed changes

alykhantejani added 3 commits February 26, 2017 09:13

remove calls to .numpy()

a03fb42

remove numpy dependency from TestNNInit + reformat code in torch.nn.init

d519eaa

use numel() instead of view and size. move skipIfNoLapack and TEST_SC…

c0de19b

…IPY to test/common

pep8 fixes to torch.nn.init

850137b

apaszke approved these changes Feb 26, 2017

View reviewed changes

test/test_nn.py Outdated

fan_in = input_tensor.size(1)

if input_tensor.dim() > 2:

fan_in *= input_tensor[0][0].numel()

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

added mode param to kaiming_* initializers

4a2a088

apaszke closed this Mar 1, 2017

apaszke reopened this Mar 1, 2017

soumith and others added 2 commits March 1, 2017 12:32

Merge branch 'master' into add_nn_init

18e7ec5

fix flake8 issues in docstrings

7d97907

apaszke merged commit 37e0548 into pytorch:master Mar 1, 2017

alykhantejani deleted the add_nn_init branch March 2, 2017 08:53

ezyang added the open source label Jun 24, 2019

jjsjann123 pushed a commit to jjsjann123/pytorch that referenced this pull request May 19, 2021

Fix the return type for where function (pytorch#833)

06a99f2

Co-authored-by: Ryan Spring <[email protected]>

		return tensor.normal_(0, std)


		def kaiming_uniform(tensor, gain=1):

added initialization schemes in torch.nn.init #833

added initialization schemes in torch.nn.init #833

Uh oh!

Conversation

alykhantejani commented Feb 23, 2017

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

alykhantejani commented Feb 23, 2017

Uh oh!

soumith commented Feb 23, 2017

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

alykhantejani commented Feb 24, 2017

Uh oh!

soumith commented Feb 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colesbury left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

soumith commented Feb 24, 2017 •

edited

Loading

alykhantejani commented Feb 26, 2017 •

edited

Loading