Model params change with 0 learning rate #2293
Description
While training a convnet with SGD, the train/test loss and error change when using a learning rate of 0.
Log:
[TEST] Loss: 2.3440, Error: 8999/10000 (90%)
[TRAIN Epoch 1] Loss: 2.33110598225, Error: 44996/50000
[TEST] Loss: 2.3364, Error: 9027/10000 (90%)
[TRAIN Epoch 2] Loss: 2.33058553279, Error: 45001/50000
[TEST] Loss: 2.3342, Error: 9025/10000 (90%)
[TRAIN Epoch 3] Loss: 2.33120793699, Error: 44981/50000
[TEST] Loss: 2.3358, Error: 9030/10000 (90%)
[TRAIN Epoch 4] Loss: 2.33072890223, Error: 44987/50000
[TEST] Loss: 2.3350, Error: 9024/10000 (90%)
[TRAIN Epoch 5] Loss: 2.33064097578, Error: 45025/50000
[TEST] Loss: 2.3367, Error: 9029/10000 (90%)
[TRAIN Epoch 6] Loss: 2.33016999603, Error: 44991/50000
[TEST] Loss: 2.3359, Error: 9026/10000 (90%)
[TRAIN Epoch 7] Loss: 2.33080320681, Error: 44999/50000
[TEST] Loss: 2.3352, Error: 9035/10000 (90%)
[TRAIN Epoch 8] Loss: 2.33087820165, Error: 44996/50000
[TEST] Loss: 2.3365, Error: 9018/10000 (90%)
[TRAIN Epoch 9] Loss: 2.33066928387, Error: 45002/50000
[TEST] Loss: 2.3356, Error: 9025/10000 (90%)
This happens with DenseNet and ResNet.
Training script: https://gist.github.com/vabh/50c12ca28619836e32a869aa0e52ea38
The architecture can be chosen in lines 52-65
Links to implementation:
DenseNet: https://github.com/bamos/densenet.pytorch
DenseNet: https://github.com/andreasveit/densenet-pytorch
ResNeXt: https://github.com/prlz77/ResNeXt.pytorch
PyTorch version: 0.1.12_2