Harnessing neural networks: A random matrix approach

2021

Machine learning applications employ FFNN (Feed Forward Neural Network) in their discipline enormously. But, it has been observed that the FFNN requisite speed is not up the mark. The fundamental causes of this problem are: 1) for training neural networks, slow gradient descent methods are broadly used and 2) for such methods, there is a need for iteratively tuning hidden layer parameters including biases and weights. To resolve these problems, a new emanant machine learning algorithm, which is a substitution of the feed-forward neural network, entitled as Extreme Learning Machine (ELM) introduced in this paper. ELM also come up with a general learning scheme for the immense diversity of different networks (SLFNs and multilayer networks). According to ELM originators, the learning capacity of networks trained using backpropagation is a thousand times slower than the networks trained using ELM, along with this, ELM models exhibit good generalization performance. ELM is more efficient...

This paper develops multi-layer classifiers and auto-encoders based on the Random Neural Network. Our motivation is to build robust classifiers that can be used in systems applications such as Cloud management for the accurate detection of states that can lead to failures. Using an idea concerning some to soma interactions between natural neuronal cells, we discuss a basic building block constructed of clusters of densely packet cells whose mathematical properties are based on G-Networks and the Random Neural Network. These mathematical properties lead to a transfer function that can be exploited for large arrays of cells. Based on this mathematical structure we build multi-layer networks. In order to evaluate the level of classification accuracy that can be achieved, we test these auto-encoders and classifiers on a widely used standard database of handwritten characters. Abstract. This paper develops multi-layer classifiers and auto-encoders based on the Random Neural Network. Our motivation is to build robust classifiers that can be used in systems applications such as Cloud management for the accurate detection of states that can lead to failures. Using an idea concerning some to soma interactions between natural neuronal cells, we discuss a basic building block constructed of clusters of densely packet cells whose mathematical properties are based on G-Networks and the Random Neural Network. These mathematical properties lead to a transfer function that can be exploited for large arrays of cells. Based on this mathematical structure we build multi-layer networks. In order to evaluate the level of classification accuracy that can be achieved, we test these auto-encoders and classifiers on a widely used standard database of handwritten characters. AQ1

FeedForward Neural Networks (FFNNs) have been successfully applied in many scientific areas such as computer science, engineering, biology, medicine and defense, among others. Their main advantages are (1) to approximate complex nonlinear mappings directly from the input samples; and (2) to provide models for a large class of natural and artificial phenomena that are difficult to handle using classical parametric techniques. Nevertheless, one of its most important drawbacks is that the traditional training procedures do not provide an efficient and fast implementation. This is due to the many parameters to be properly tuned by slow (often gradient based) algorithms, in order to obtain a good enough model. Furthermore, the training phase has to be repeated in order to perform model structure selection, for example the * Page (PS/TeX): 2 / 2, COMPOSITE 2 Pedro J. García-Laencina et al.

2015

Extreme learning machine is a new scheme for learning the feedforward neural network, where the input weights and biases determining the nonlinear feature mapping are initiated randomly and are not learned. In this work, we analyze approximation ability of the extreme learning machine depending on the activation function type and ranges from which input weights and biases are randomly generated. The studies are performed on the example of approximation of one variable function with varying complexity. The ranges of input weights and biases are determined for ensuring the sufficient flexibility of the set of activation functions to approximate the target function in the input interval.

Springer eBooks, 2017

The Random Neural Network (RNN) is a recurrent spiking neuronal model that has been used for learning and dynamic optimisation of large scale network systems. Here we use the RNN to construct dense block of spiking neuronal cells in conjunction with Deep Learning to mimic the stochastic behaviour of biological neurons in mammalian brains. Together with prior work on extreme learning machines (ELM), we construct multilayer architectures (MLA) that exploit dense clusters of RNNs for Deep Learning and evaluate their performance on large visual recognition datasets. The results obtained indicate that this approach can reach and exceed the levels of performance that have been previously reported. Finally, we develop an incremental learning algorithm to train such RNN-ELM multilayer architectures for purpose of handing big data.

2021

Machine learning applications employ FFNN (Feed Forward Neural Network) in their discipline enormously. But, it has been observed that the FFNN requisite speed is not up the mark. The fundamental causes of this problem are: 1) for training neural networks, slow gradient descent methods are broadly used and 2) for such methods, there is a need for iteratively tuning hidden layer parameters including biases and weights. To resolve these problems, a new emanant machine learning algorithm, which is a substitution of the feed-forward neural network, entitled as Extreme Learning Machine (ELM) introduced in this paper. ELM also come up with a general learning scheme for the immense diversity of different networks (SLFNs and multilayer networks). According to ELM originators, the learning capacity of networks trained using backpropagation is a thousand times slower than the networks trained using ELM, along with this, ELM models exhibit good generalization performance. ELM is more efficient...

2009

In order to provide a guideline about the number of hidden neurons N(h) and learning rate eta for large-scale neural networks from the viewpoint of stable learning, the authors try to formulate the boundary of stable learning roughly, and to adjust it to the actual learning results of random number mapping problems. It is confirmed in the simulation that the hidden-output connection weights become small as the number of hidden neurons becomes large, and also that the trade-off in the learning stability between input-hidden and hidden-output connections exists. Finally, two equations N(h) = radic(N(i) N(o)) and eta = 32 /radic(N(i)N(o)) are roughly introduced where N(i) and N(o) are the number of input and output neurons respectively even though further adjustment is necessary for other problems or conditions.

2018

This article provides a theoretical analysis of the asymptotic performance of a regression or classification task performed by a simple random neural network. This result is obtained by leveraging a new framework at the crossroads between random matrix theory and the concentration of measure theory. This approach is of utmost interest for neural network analysis at large in that it naturally dismisses the difficulty induced by the non-linear activation functions, so long that these are Lipschitz functions. As an application, we provide formulas for the limiting law of the random neural network output and compare them conclusively to those obtained practically on handwritten digits databases.

arXiv (Cornell University), 2022

We argue that many properties of fully-connected feedforward neural networks (FCNNs), also called multi-layer perceptrons (MLPs), are explainable from the analysis of a single pair of operations, namely a random projection into a higher-dimensional space than the input, followed by a sparsification operation. For convenience, we call this pair of successive operations expand-and-sparsify following the terminology of Dasgupta. We show how expand-andsparsify can explain the observed phenomena that have been discussed in the literature, such as the so-called Lottery Ticket Hypothesis, the surprisingly good performance of randomlyinitialized untrained neural networks, the efficacy of Dropout in training and most importantly, the mysterious generalization ability of overparameterized models, first highlighted by Zhang et al. and subsequently identified even in non-neural network models by Belkin et al.

Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (INNS), 2015

Extreme learning machines (ELM) represent a new fast learning algorithm for single layer feedforward networks. In this paper, we investigate several practical properties of training ELM in the context of a simulated function approximation task. ELM with different hidden layer activation functions and varying number of hidden nodes are applied in the learning task and the function approximation accuracy is examined with respect to the range scaling of input variables. The results demonstrate that ELM models are very sensitive with respect to proper input scaling. The approximate range of optimal ELM performance covers only input range scaling within an order of magnitude. Comparison with classical feedforward neural networks with sigmoidal activation functions shows that these are not affected by input range scaling. Results encourage further studies of practical aspect of efficient training and applying ELM.

Neurocomputing, 2014

This paper proposes a learning framework for single-hidden layer feedforward neural networks (SLFN) called optimized extreme learning machine (O-ELM). In O-ELM, the structure and the parameters of the SLFN are determined using an optimization method. The output weights, like in the batch ELM, are obtained by a least squares algorithm, but using Tikhonov's regularization in order to improve the SLFN performance in the presence of noisy data. The optimization method is used to select the set of input variables, the hidden-layer configuration and bias, the input weights and the Tikhonov's regularization factor. The proposed framework has been tested with three optimization methods (genetic algorithms, simulated annealing, and differential evolution) over sixteen benchmark problems available in public repositories.

Neural Network World, 2013

Extreme learning machine (ELM) is an emergent method for training single hidden layer feedforward neural networks (SLFNs) with extremely fast training speed, easy implementation and good generalization performance. This work presents effective ensemble procedures for combining ELMs by exploiting diversity. A large number of ELMs are initially trained in three different scenarios: the original feature input space, the obtained feature subset by forward selection and different random subsets of features. The best combination of ELMs is constructed according to an exact ranking of the trained models and the useless networks are discarded. The experimental results on several regression problems show that robust ensemble approaches that exploit diversity can effectively improve the performance compared with the standard ELM algorithm and other recent ELM extensions.

2018

Especially in the last decade, Artificial Intelligence (AI) has gained increasing popularity as the neural networks represent incredibly exciting and powerful machine learning-based techniques that can solve many real-time problems. The learning capability of such systems is directly related with the evaluation methods used. In this study, the effectiveness of the calculation parameters in a Single-Hidden Layer Feedforward Neural Networks (SLFNs) will be examined. We will present how important the selection of an activation function is in the learning stage. A lot of work is developed and presented for SLFNs up to now. Our study uses one of the most commonly known learning algorithms, which is Extreme Learning Machine (ELM). Main task of an activation function is to map the input value of a neural network to the output node with a high learning or achievement rate. However, determining the correct activation function is not as simple as thought. First we try to show the effect of the activation functions on different datasets and then we propose a method for selection process of it due to the characteristic of any dataset. The results show that this process is providing a remarkably better performance and learning rate in a sample neural network.

Neural Networks, 2013

Selection of the optimal neural architecture to solve a pattern classification problem entails to choose the relevant input units, the number of hidden neurons and its corresponding interconnection weights. This problem has been widely studied in many research works but their solutions usually involve excessive computational cost in most of the problems and they do not provide a unique solution. This paper proposes a new technique to efficiently design the MultiLayer Perceptron (MLP) architecture for classification using the Extreme Learning Machine (ELM) algorithm. The proposed method provides a high generalization capability and a unique solution for the architecture design. Moreover, the selected final network only retains those input connections that are relevant for the classification task. Experimental results show these advantages.

Neural Networks, 2012

Error minimized extreme learning machines Support vector sequential feed-forward neural networks Sequential approximations a b s t r a c t Recently, error minimized extreme learning machines (EM-ELMs) have been proposed as a simple and efficient approach to build single-hidden-layer feed-forward networks (SLFNs) sequentially. They add random hidden nodes one by one (or group by group) and update the output weights incrementally to minimize the sum-of-squares error in the training set. Other very similar methods that also construct SLFNs sequentially had been reported earlier with the main difference that their hidden-layer weights are a subset of the data instead of being random. These approaches are referred to as support vector sequential feed-forward neural networks (SV-SFNNs), and they are a particular case of the sequential approximation with optimal coefficients and interacting frequencies (SAOCIF) method. In this paper, it is firstly shown that EM-ELMs can also be cast as a particular case of SAOCIF. In particular, EM-ELMs can easily be extended to test some number of random candidates at each step and select the best of them, as SAOCIF does. Moreover, it is demonstrated that the cost of the computation of the optimal outputlayer weights in the originally proposed EM-ELMs can be improved if it is replaced by the one included in SAOCIF. Secondly, we present the results of an experimental study on 10 benchmark classification and 10 benchmark regression data sets, comparing EM-ELMs and SV-SFNNs, that was carried out under the same conditions for the two models. Although both models have the same (efficient) computational cost, a statistically significant improvement in generalization performance of SV-SFNNs vs. EM-ELMs was found in 12 out of the 20 benchmark problems.