Feedforward neural networks for compound signals

IEEE Transactions on Neural Networks, 1996

In this paper a general class of fast learning algorithms for feedforward neural networks is introduced and described. The approach exploits the separability of each layer into linear and nonlinear blocks and consists of two steps. The first step is the descent of the error functional in the space of the outputs of the linear blocks (descent in the neuron space), which can be performed using any preferred optimization strategy. In the second step, each linear block is optimized separately by using a Least Squares (LS) criterion.

Neural Networks, 1989

This paper rigorously establishes thut standard rnultiluyer feedforward networks with as f&v us one hidden layer using arbitrary squashing functions ure capable of upproximating uny Bore1 measurable function from one finite dimensional space to another to any desired degree of uccuracy, provided sujficirntly muny hidden units are available. In this sense, multilayer feedforward networks are u class of universul rlpproximators.

Neurocomputing, 2009

In this paper, we propose a new neural network architecture based on a family of referential multilayer perceptrons (RMLPs) that play a role of generalized receptive fields. In contrast to ''standard'' radial basis function (RBF) neural networks, the proposed topology of the network offers a considerable level of flexibility as the resulting receptive fields are highly diversified and capable of adjusting themselves to the characteristics of the locally available experimental data. We discuss in detail a design strategy of the novel architecture that fully exploits the modeling capabilities of the contributing RMLPs. The strategy comprises three phases. In the first phase, we form a ''blueprint'' of the network by employing a specialized version of the commonly encountered fuzzy C-means (FCM) clustering algorithm, namely the conditional (context-based) FCM. In this phase our intent is to generate a collection of information granules (fuzzy sets) in the space of input and output variables, narrowed down to some certain contexts. In the second phase, based upon a global view at the structure, we refine the input-output relationships by engaging a collection of RMLPs where each RMLP is trained by using the subset of data associated with the corresponding context fuzzy set. During training each receptive field focuses on the characteristics of these locally available data and builds a nonlinear mapping in a referential mode. Finally, the connections of the receptive fields are optimized through global minimization of the linear aggregation unit located at the output layer of the overall architecture. We also include a series of numeric experiments involving synthetic and real-world data sets which provide a thorough comparative analysis with standard RBF neural networks.

2011

The`Neural Pipeline' is introduced as an articial neural network architecture that controls information ow using its own connection structure. The architecture is multi-layered with`external' connections between the layers to control the data. Excitatory connections transfer data from each layer to the next and inhibitory feedback connections run from each layer to the previous layer. Using these connections a layer can temporarily silence the previous layer and stop further inputs until it nishes processing. When excitation and inhibition are balanced, waves of activity propagate sequentially through the layers after each input; this is`correct' behaviour. When the system is`over' inhibited, the inhibitory feedback outweighs the excitation from the input. At least one layer remains inhibited for too long so further inputs cannot stimulate the layer. Over inhibition can be corrected by increasing the delay between inputs. When the system is`under' inhibited the excitation in the layer is larger than the inhibition. The layer is therefore not silenced and continues to spike. In the layers, excitatory and inhibitory spiking neurons are randomly interconnected. Changing layer parameters inuences the system behaviour. Recommendations for correct behaviour include: low neuron connectivity and balancing the external inhibition and layer activity. With variations of only the internal topology and weights, all three behaviours can be exhibited. Each layer is trained as a separate Liquid State Machine, with readout neurons trained to respond to a particular input. A set of six shapes can be learnt by all layers of a three layer Neural Pipeline. The layers are trained to recognise dierent features; layer 1 recognising the position while layer 2 identies the shape. The system can cope when the same noisy signal is applied to all inputs, but begins to make mistakes when dierent noise is applied to each input neuron. The thesis introduces and develops the Neural Pipeline architecture to provide a platform for further work.

1998

The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made.

1996

List of Figures 1.1 Detailed (top) and symbolic (bottom) representations of the artificial neuron. 1.2 Examples of activation functions, a(.): logistic and hyperbolic tangent. 1.3 The conventional 2-layer d: q: l feedforward network with d inputs, q hidden neurons, and a single output. 1.4 Example of a good fit (a), an under-fit (b), and an over-fit (c) of training data. The first case will results in good generalisation, whereas the latter two in poor. 2.1 Universal approximation in one dimension. 15 26 3.1 Two choices for the discretising function Q(w) [841.47 3.2 Frequency distribution of numbers generated by tan(RND). 3.3 Qp,, cc '(w). 53 3.4 Comparison of the actual error term and its approximation in the range [-3,31.53 3.5 The black-hole function. 54 4.1 The set of decision boundaries of an integer [-3,31 weight 2-input perceptron with offset. Some of the possible 73 decision boundaries lie outside the {(-l,-1), (1,1)) square, and therefore are not shown. 63 4.2 Linearly separable data sets with decision boundaries at gradually varying angles. 64 iv V LIST OF FIGURES 4.3 IWN minimum E,,,.. as a function of the number of hidden neurons for the data sets shown in Figure 4.2. The 1 and 2 hidden neuron E

Neurocomputing, 2003

In this paper a new strategy for adaptively and autonomously constructing a multi-hidden-layer feedforward neural network (FNN) is introduced. The proposed scheme belongs to a class of structure level adaptation algorithms that adds both new hidden units and new hidden layers one at a time when it is determined to be needed. Using this strategy, a FNN may be constructed having as many hidden layers and hidden units as required by the complexity of the problem being considered. Simulation results applied to regression problems are included to demonstrate the performance capabilities of the proposed scheme.

Journal of Vlsi Signal Processing Systems for Signal Image and Video Technology, 2002

Designing a neural network (NN) to process complex-valued signals is a challenging task since a complex nonlinear activation function (AF) cannot be both analytic and bounded everywhere in the complex plane . To avoid this difficulty, 'splitting', i.e., using a pair of real sigmoidal functions for the real and imaginary components has been the traditional approach. However, this 'ad hoc'

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization schemes, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs, and other machine learning methods such as random forests and support vector machines. For FNNs we considered (i) ReLU networks without normalization, (ii) batch normalization, (iii) layer normalization, (iv) weight normalization, (v) highway networks, and (vi) residual networks. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.

1992

Acoustics, Speech, and Signal Processing, 1988. ICASSP-88., 1988 International Conference on

ScienceAsia, 2014

In this paper, a framework based on algebraic structures to formalize various types of neural networks is presented. The working strategy is to break down neural networks into building blocks, relationships between each building block, and their operations. Building blocks are collections of primary components or neurons. In turn, neurons are collections of properties functioning as single entities, transforming an input into an output. We perceive a neuron as a function. Thus the flow of information in a neural network is a composition between functions. Moreover, we also define an abstract data structure called a layer which is a collection of entities which exist in the same time step. This layer concept allows the parallel computation of our model. There are two types of operation in our model; recalling operators and training operators. The recalling operators are operators that challenge the neural network with data. The training operators are operators that change parameters of neurons to fit with the data. This point of view means that all neural networks can be constructed or modelled using the same structures with different parameters.

Statistica Neerlandica, 2000

We study the relation between the asymptotic behaviour of synchronous Boltzmann machines and synchronous Hop®eld networks. More speci®cally, we consider the relation between the pseudo consensus function that is used in analyzing Boltzmann machines and the energy function that is used in the study of Hop®eld networks. We show that for small values of the control parameter, synchronous Boltzmann machines and synchronous Hop®eld networks compute global respectively local maxima of the same function.

Chapman & Hall/CRC Computer & Information Science Series, 2007