As part of my on going training of GoogleNet in Caffe (the wining entry of ImageNet-2014) I was doing some timings, and these are my findings:

[Comparison with Caffe_reference]

• GoogleNet with cuDNN is (2.8x forward, 3.6x backward) slower than caffe_reference with cuDNN.
• VGGNet_16Layers without cuDNN is (11.5x forward, 18.7x backward) slower than caffe_reference with cuDNN.
• VGGNet_19Layers without cuDNN is (13.8x forward, 22x backward) slower than caffe_reference with cuDNN.

These experiments are run in one K40c using batch_size:128 in a server with 8-GPUs running other tasks.

[Comparison with cuDNN vs without cuDNN]

• caffe_reference with cuDNN is (1.4x forward, 1.28x backward) faster than without cuDNN
  • Average Forward pass: 200.792 ms.
  • Average Backward pass: 310.973 ms.
  • Average Forward-Backward: 511.953 ms.
• caffe_reference without cuDNN
  • Average Forward pass: 281.24 ms.
  • Average Backward pass: 398.719 ms.
  • Average Forward-Backward: 680.189 ms.
• GoogleNet with cuDNN is (1.6x forward, 1.4x backward) faster than without cuDNN
  • Average Forward pass: 562.841 ms.
  • Average Backward pass: 1123.84 ms.
  • Average Forward-Backward: 1688.8 ms.
• GoogleNet without cuDNN
  • Average Forward pass: 922.007 ms.
  • Average Backward pass: 1533.55 ms.
  • Average Forward-Backward: 2455.89 ms.

For the VGG networks I need to use batch_size: 64 to be able to fit them in memory, so I multiplied the times by 2.

• VGG_16Layers with cuDNN is (1.2x, 1.12x) slower than without cuDNN
  • Average Forward pass: 2772 ms.
  • Average Backward pass: 6546.86 ms.
  • Average Forward-Backward: 9324.94 ms.
• VGG_16Layers without cuDNN
  • Average Forward pass: 2298.68 ms.
  • Average Backward pass: 5825.2 ms.
  • Average Forward-Backward: 8124.48 ms.
• VGG_19Layers with cuDNN is (1.22x, 1.36x) slower than without cuDNN
  • Average Forward pass: 3387.08 ms.
  • Average Backward pass: 7928.3 ms.
  • Average Forward-Backward: 11316.92 ms.
• VGG_19Layers without cuDNN
  • Average Forward pass: 2769.9 ms.
  • Average Backward pass: 6850.64 ms.
  • Average Forward-Backward: 9623.26 ms.