Academia.eduAcademia.edu

Deep Neural Networks—A Brief History

Profile image of Krzysztof CiosKrzysztof Cios

2017, Studies in Computational Intelligence

Abstract
sparkles

AI

This article provides an overview of Deep Neural Networks (DNN), detailing their historical evolution and operational principles. It describes how DNN hierarchically processes information to extract features, focusing on applications like face recognition while contrasting DNN approaches with traditional methods. The paper also discusses different learning modes of DNN, their inspiration from biological neural networks, and compares the storage capacities of human brains and artificial neural networks.

Figures (16)
Figure 1. Hodgkin and Huxley’s model circuit representation.
Figure 1. Hodgkin and Huxley’s model circuit representation.
It is described by these equations:
It is described by these equations:
Figure 6. Two types of excitatory (the first two) and two types of inhibitory neurons firing patterns (taken from http://www. izhikevich.org/publications/spikes.htm). yis a uniform random variable between 0 and 1 Izhikevich model became very popular because its simplicity allows for building networks consisting of thousands of suct neurons. While using it, however, we found that increasing the strength of the stimulus caused it to fire with higher and higher frequency (no upper bound). This is not biologically plausible as neurons cannot fire during the absolute refractory period, needed for restoration of their membrane potentials, no matter the strength of the input. We thus corrected the condition for the neuron firing (Strack, Jacobs and Cios, 2013) by changing it Paes
Figure 6. Two types of excitatory (the first two) and two types of inhibitory neurons firing patterns (taken from http://www. izhikevich.org/publications/spikes.htm). yis a uniform random variable between 0 and 1 Izhikevich model became very popular because its simplicity allows for building networks consisting of thousands of suct neurons. While using it, however, we found that increasing the strength of the stimulus caused it to fire with higher and higher frequency (no upper bound). This is not biologically plausible as neurons cannot fire during the absolute refractory period, needed for restoration of their membrane potentials, no matter the strength of the input. We thus corrected the condition for the neuron firing (Strack, Jacobs and Cios, 2013) by changing it Paes
Figure 7. A) unbounded firing of the original Izhikevich model neurons; B) firing of the neurons after Strack at el. (2014) modification accounting for absolute refractory periods.
Figure 7. A) unbounded firing of the original Izhikevich model neurons; B) firing of the neurons after Strack at el. (2014) modification accounting for absolute refractory periods.
Figure 8. Comparison of the SAPR and STDP: the latter is fixed while the former changes depending on the shape of excitatory and inhibitory post synaptic functions of neurons. The difference between the two rules is that SAPR uses a function that is continuous and differentiable (important in several applications); it is also dynamic because it uses actual post-synaptic potential functions to modify the connection strengths between the neurons. In other words, the adjustments depend on the shape of SAPR, which in turn depends on the shape of the chosen postsynaptic functions in a given neural circuit. The left part of the SAPR function in Figure 8 (to the left of the y axis) is the chosen inhibitory PSP while the right part is the chosen excitatory PSP; see again the two function shapes in Figure 4. In contrast, the STDP rule uses a static function meaning that the adjustments are always the same; they do not depend on the shape of inhibitory/excitatory PSPs for a given At.
Figure 8. Comparison of the SAPR and STDP: the latter is fixed while the former changes depending on the shape of excitatory and inhibitory post synaptic functions of neurons. The difference between the two rules is that SAPR uses a function that is continuous and differentiable (important in several applications); it is also dynamic because it uses actual post-synaptic potential functions to modify the connection strengths between the neurons. In other words, the adjustments depend on the shape of SAPR, which in turn depends on the shape of the chosen postsynaptic functions in a given neural circuit. The left part of the SAPR function in Figure 8 (to the left of the y axis) is the chosen inhibitory PSP while the right part is the chosen excitatory PSP; see again the two function shapes in Figure 4. In contrast, the STDP rule uses a static function meaning that the adjustments are always the same; they do not depend on the shape of inhibitory/excitatory PSPs for a given At.
Figure 9. A) illustration of how the simple and complex cells extract specific features from input images; B) implementation of how the features are extracted and aggregated (using three hidden layers) in Neocognitron to recognize digit 2 (both pictures are taken from Kandel et al. Principles of Neural Science, 5"" edition, 2013).
Figure 9. A) illustration of how the simple and complex cells extract specific features from input images; B) implementation of how the features are extracted and aggregated (using three hidden layers) in Neocognitron to recognize digit 2 (both pictures are taken from Kandel et al. Principles of Neural Science, 5"" edition, 2013).
The first researcher to design a direct precursor of DNN, using Hubel and Wiesel’s discoveries, was Fukushima (1980) who called his network Neocognitron. Figure 10 A) illustrates how key features of an image of letter A are first picked t by simple cells (S) and then aggregated by complex cells (C), in order to recognize letter A at the output. S-layer of simple cells extracts features from the previous stage in the hierarchy, while the C-layer of complex cells ensures tolerance for shifts of features extracted by the S-layer.
The first researcher to design a direct precursor of DNN, using Hubel and Wiesel’s discoveries, was Fukushima (1980) who called his network Neocognitron. Figure 10 A) illustrates how key features of an image of letter A are first picked t by simple cells (S) and then aggregated by complex cells (C), in order to recognize letter A at the output. S-layer of simple cells extracts features from the previous stage in the hierarchy, while the C-layer of complex cells ensures tolerance for shifts of features extracted by the S-layer.
Figure 10. A) Fukushima’s Neocognitron architecture, and B) LeCun’s convolutional neural network architecture.
Figure 10. A) Fukushima’s Neocognitron architecture, and B) LeCun’s convolutional neural network architecture.
Figure 11. IRNN’s architecture: unsupervised part consists of the sensory and feature aggregating layers while the associative part is supervised. Figure 11 shows stacks of neurons represented by small balls. How they are generated and what they represent is explained in Figure 12. We see there three (hashed) subimages/windows of the three input images, which are clusterec using a novel image similarity measure (Cios and Shin, 1995). If the first two subimages are similar (as shown) they are clustered together in neuron n;. Since subimage 3 was found quite different from the subimages 1 and 2, it creates its own cluster, so the second neuron, np, is generated. The weights w, and wy, are initially set to the first subimage pixel values vector but are later updated to represent cluster center (thus representing an “average” subimage). At the end of scanning of entire images the result might be as the one shown in Figure 13. Notice, that at the center more neurons clusters) were created to represent image details, such as nose, eyes and mouth, while at the periphery where packground was about the same in all images only single neurons/clusters were needed.
Figure 11. IRNN’s architecture: unsupervised part consists of the sensory and feature aggregating layers while the associative part is supervised. Figure 11 shows stacks of neurons represented by small balls. How they are generated and what they represent is explained in Figure 12. We see there three (hashed) subimages/windows of the three input images, which are clusterec using a novel image similarity measure (Cios and Shin, 1995). If the first two subimages are similar (as shown) they are clustered together in neuron n;. Since subimage 3 was found quite different from the subimages 1 and 2, it creates its own cluster, so the second neuron, np, is generated. The weights w, and wy, are initially set to the first subimage pixel values vector but are later updated to represent cluster center (thus representing an “average” subimage). At the end of scanning of entire images the result might be as the one shown in Figure 13. Notice, that at the center more neurons clusters) were created to represent image details, such as nose, eyes and mouth, while at the periphery where packground was about the same in all images only single neurons/clusters were needed.
Figure 12. Explanation of clustering of subimages into a number of clusters/neurons.
Figure 12. Explanation of clustering of subimages into a number of clusters/neurons.
mm Figure 14. Architecture of the network of spiking neurons; (a) High-level block diagram. (b) Recurrent synaptic connections between the excitatory neurons in the feature extraction layer. (c) Synaptic connections between the excitatory neurons in the sensory/feature extraction layer and the inhibitory neurons in the feature extraction layer (taken from Shin et al. 2010 paper). The same process is repeated on the outputs of the sensory layer to aggregate the local features into more complex
mm Figure 14. Architecture of the network of spiking neurons; (a) High-level block diagram. (b) Recurrent synaptic connections between the excitatory neurons in the feature extraction layer. (c) Synaptic connections between the excitatory neurons in the sensory/feature extraction layer and the inhibitory neurons in the feature extraction layer (taken from Shin et al. 2010 paper). The same process is repeated on the outputs of the sensory layer to aggregate the local features into more complex

Related papers

Deep network in network
Hmidi Alaeddine

The different CNN models use many layers that typically include a stack of linear convolution layers combined with pooling and normalization layers to extract the characteristics of the images. Unlike these models, and instead of using a linear filter for convolution, the network in network (NiN) model uses a multilayer perception (MLP), a nonlinear function, to replace the linear filter. This article presents a new deep network in network (DNIN) model based on the NiN structure, NiN drag a universal approximator, (MLP) with rectified linear unit (ReLU) to improve classification performance. The use of MLP leads to an increase in the density of the connection. This makes learning more difficult and time learning slower. In this article, instead of ReLU, we use the linear exponential unit (eLU) to solve the vanishing gradient problem that can occur when using ReLU and to speed up the learning process. In addition, a reduction in the convolution filters size by increasing the depth is used in order to reduce the number of parameters. Finally, a batch normalization layer is applied to reduce the saturation of the eLUs and the dropout layer is applied to avoid overfitting. The experimental results on the CIFAR-10 database show that the DNIN can reduce the complexity of implementation due to the reduction in the adjustable parameters. Also the reduction in the filters size shows an improvement in the recognition accuracy of the model. Keywords Exponential linear unit (ELU) Á Convolutional neural networks (CNNs) Á Deep MLPconv Á Image recognition Á Network in Network (NiN)

View PDFchevron_right
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
Abdul Awwal

Cornell University - arXiv, 2018

In recent years, deep learning has garnered tremendous success in a variety of application domains. This new field of machine learning has been growing rapidly, and has been applied to most traditional application domains, as well as some new areas that present more opportunities. Different methods have been proposed based on different categories of learning, including supervised, semi-supervised, and un-supervised learning. Experimental results show state-of-the-art performance using deep learning when compared to traditional machine learning approaches in the fields of image processing, computer vision, speech recognition, machine translation, art, medical imaging, medical information processing, robotics and control, bio-informatics, natural language processing (NLP), cybersecurity, and many others. This report presents a brief survey on the advances that have occurred in the area of DL, starting with the Deep Neural Network (DNN). The survey goes on to cover the Convolutional Neural Network (CNN), the Recurrent Neural Network (RNN) including Long Short Term Memory (LSTM) and Gated Recurrent Units (GRU), the Auto-Encoder (AE), the Deep Belief Network (DBN), the Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL). Additionally, we have included recent developments such as advanced variant DL techniques based on these DL approaches. This work considers most of the papers published after 2012 from when the history of deep learning began. Furthermore, DL approaches that have been explored and evaluated in different application domains are also included in this survey. We also included recently developed frameworks, SDKs, and benchmark datasets that are used for implementing and evaluating deep learning approaches. There are some surveys that have been published on Deep Learning using Neural Networks [1, 38] and a survey on RL [234]. However, those papers have not discussed the individual advanced techniques for training large scale deep learning models and the recently developed method of generative models [1].

View PDFchevron_right
Deep Learning : Techniques and Applications
Debasmita Bhoumik

Learning is a process by which a system improves its performance from experience . Since 2006, deep learning is emerged as a new area of machine learning, impacting a wide range of signal and information processing work in both the traditional and the new scopes. Many traditional machine learning and signal processing techniques exploit shallow architectures, which contain a single layer of nonlinear feature transformation. Examples of shallow architectures are conventional hidden Markov models (HMMs), maximum entropy (MaxEnt) models, support vector machines (SVMs) , kernel regression, and multilayer perceptron (MLP) with a single hidden layer. A standard neural network (NN) consists of many simple, connected processors called neurons, each producing a sequence of real-valued activations. Input neurons get activated through sensors perceiving the environment, other neurons get activated through weighted connections from previously active neurons. Human information processing mechanisms (e.g., vision and speech), however, recommend the need of deep architectures for extracting complex structure and building internal representation from rich sensory inputs (e.g., natural image and its motion, speech, and music). It is natural to believe that the state of the art can be advanced in processing these types of media signals if efficient and effective deep learning algorithms are developed. Signal processing systems with deep architectures are composed of many layers of nonlinear processing stages, where each lower layer’s outputs are fed to its immediate higher layer as the input. Deep learning is a type of machine learning in which a model learns to perform classification tasks directly from images, text, or sound. Deep learning is usually implemented using a neural network architecture. The term “deep” refers to the number of layers in the network—the more layers, the deeper the network. Traditional neural networks contain only 2 or 3 layers, while deep networks can have hundreds. A few examples of deep learning at work: A self-driving vehicle slows down as it approaches a pedestrian crosswalk, An ATM rejects a counterfeit bank note, A smartphone app gives an instant translation of a foreign street sign. Deep learning is especially well-suited to identification applications such as face recognition, text translation, voice recognition, and advanced driver assistance systems, including, lane classification and traffic sign recognition In this paper we will see some of those techniques to achieve deep learning and their corresponding applications.

View PDFchevron_right
Contributions to deep learning methodologies
Shabab Bazrafkan

2018

In recent years the Deep Neural Networks (DNN) has been using widely in a big range of machine learning and data-mining purposes. This pattern recognition approach can handle highly nonlinear problems. In this work, three main contributions to DNN are presented. 1A method called Semi Parallel Deep Neural Networks (SPDNN) is introduced wherein several deep architectures are mixed and merged using graph contraction technique to take advantage of all the parent networks. 2The importance of data is investigated in several attempts and an augmentation technique know as Smart Augmentation is presented. 3To extract more information from a database, multiple works on Generative Adversarial Networks (GAN) are given wherein the joint distribution of data and its ground truth is approximated and in other projects conditional generators for classification and regression problems are trained and tested.

View PDFchevron_right
Deep Learning Techniques for Image Recognition (Machine Learning)
Olumide Olope, Obanewa Kolapo

Path of Science, 2024

Deep learning (DL), a sophisticated subset of machine learning (ML), has emerged as a transformative force within the broader realm of artificial intelligence (AI). By leveraging architectures such as convolutional neural networks (CNNs), DL has significantly advanced image recognition capabilities, enabling systems to identify and classify visual data with remarkable precision accurately. This technology is not only applicable to image recognition. Still, it has also made strides in diverse areas, such as speech recognition, language translation, automated gameplay, healthcare diagnostics, and the development of self-driving vehicles. The success of DL in this domain can be attributed to its ability to learn hierarchical representations of data, allowing for improved feature extraction and pattern recognition. Despite its impressive performance, deep learning is not without its limitations. Key challenges include its reliance on vast amounts of labelled data, which can be difficult and expensive to obtain, its lack of common sense reasoning and difficulties in addressing complex, multifaceted problems.

View PDFchevron_right
Loading...

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.