Bug

When using weight_decay in libtorch, CPU Memory usage is slowly increasing.

To Reproduce

I used docker container "nvidia/cuda:9.2-cudnn7-devel-ubuntu18.04", stable libtorch(1.1) for cuda9.0. This phenomenon can be reproduced using the mnist example in the pytorch/example repository.

rewrite examples/cpp/mnist/mnist.cpp l.148

- model.parameters(), torch::optim::SGDOptions(0.01).momentum(0.5));
+ model.parameters(), torch::optim::SGDOptions(0.01).momentum(0.5).weight_decay(1e-4));

Because the speed of increase is very slow, it may be better to increase the number of epochs. This happens with both CPU learning and GPU learning.

Environment

• OS: Ubuntu 18.04.2 LTS
• GCC version: (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
• CMake version: version 3.10.2
• CUDA runtime version: 9.2.148
• GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
• Nvidia driver version: 410.104
• cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1

Additional context

This phenomenon happens also in cuda10.0 and libtorch nightly build for cuda10.0.