LeNet Overview
Kaleab Tadesse
FTP0848/14
Addis Ababa Science and Technology University
Email: kaleabtadesse49@[Link]
I NTRODUCTION 7) F6 Layer (Fully Connected) Connects 120 units
from C5 to 84 units. F6 has 10,164 trainable
LeNet is a seminal Convolutional Neural Network parameters.
(CNN) architecture specifically designed for handwritten 8) Output Layer Consists of 10 output units rep-
character recognition introduced in 1998. resenting digits 0-9. Original design used Radial
Basis Function (RBF) units instead of softmax.
L E N ET-5 A RCHITECTURE B REAKDOWN RBF units compute Euclidean distance from a
parameter vector, suitable for character rejection
LeNet-5, used in experiments like the MNIST digit and large class sets.
recognition task, consists of seven layers with trainable
parameters, not counting the input layer. The architecture ACTIVATION F UNCTIONS AND D ESIGN C HOICES
progressively decreases spatial resolution due to absence LeNet-5 used tanh activation functions instead of
of padding while increasing the number of feature maps. ReLU, with a specific form: f (a) = A tanh(Sa) where
LeNet-5 is relatively small by modern standards, having A = 1.7159 and S = 2/3. This ensured f (1) = 1
about 60,000 trainable parameters. This is significantly and f (−1) = −1. Non-linearities were also sometimes
smaller than networks like AlexNet (˜60 million param- applied after pooling layers, unlike modern practices. It
eters) or VGG16 (˜138 million parameters). also didn’t use sigmoid function for the output layer,
1) Input Layer The network takes a grayscale image which is common in modern CNNs.
as input. For handwritten digit recognition experi- S UMMARY
ments, the input size was typically 32x32x1. Input
LeNet-5’s layered architecture with convolutional,
pixel values were normalized (background -0.1,
sub-sampling, and fully connected layers implemented
foreground 1.0) to have roughly zero mean and
key principles such as local receptive fields, weight shar-
unit variance, which helps accelerate learning.
ing, and spatial downsampling. Its design and gradient-
2) C1 Layer (Convolutional) This is the first convo-
based training allowed for highly effective handwritten
lutional layer with six 5x5 filters, stride 1, and no
character recognition than was possible at the time.
padding. The input size 32x32 results in 28x28x6
Despite differences from modern CNNs in activation
output. C1 has 156 trainable parameters.
and output layers, LeNet-5 remains foundational in deep
3) S2 Layer (Sub-sampling/Pooling) Applies aver-
[Link] which is developer much later gives
age pooling with a 2x2 filter and stride 2, reducing
credit to LeNet by naming their model Goog’LeNet’.
output to 14x14x6. S2 has 12 trainable parameters.
4) C3 Layer (Convolutional) Uses 16 filters of R EFERENCES
size 5x5. With valid convolutions, 14x14 input [1] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based
becomes 10x10x16. Connections to S2 are not learning applied to document recognition,” Proceedings of the
fully connected, used to reduce computation and IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
encourage feature diversity. C3 has 1,516 trainable
parameters.
5) S4 Layer (Sub-sampling/Pooling) Similar to S2,
reducing spatial size to 5x5x16. S4 has 32 trainable
parameters.
6) C5 Layer (Convolutional/Fully Connected)
Though described as convolutional, it behaves like
a fully connected layer since S4 is 5x5. Each of
the 120 units connects to all 400 inputs from S4.
C5 has 48,120 trainable parameters.
