Analysis of Unscaled Contributions in Cross Connected Networks

In this paper, we present a new learning mechanism called mixed-mode learning. This learning mechanism transforms an existing supervising learning algorithm from one that finds a unique one-to-one mapping into an algorithm that finds one of the many one-to-many mappings. Since the objective of learning is relaxed by this transformation, the number of learning epochs can be reduced significantly. We show in this paper that mixedmode learning can be applied in the well-known cascade correlation learning algorithm to reduce the number of hidden units required for convergence when applied to classification problems whose desired outputs are binary. Our experimental results confirm this reduction, although they show that more learning time is required than that of cascade correlation. In general, reducing the number of hidden units at the expense of additional learning time is usually desirable.

IEEE Transactions on Signal Processing, 1994

In this paper we provide theoretical foundations for a new neural model for singular value decomposition based on an extension of the Hebbian learning rule called the crosscoupled Hebhian rule. The model is extracting the SVD of the cross-correlation matrix of two stochastic signals and is an extension on previous work on neural-network-related principal component analysis (PCA). We prove the asymptotic convergence of the network to the principal (normalized) singular vectors of the cross-correlation and we provide simulation results which suggest that the convergence is exponential. The new model may have useful applications in the problems of filtering for signal processing and signal detection.

Journal of Physics A: Mathematical and General, 1997

On-line learning in layered perceptrons is often hampered by plateaus in the time dependence of the performance. Studies on backpropagation in networks with a small number of input units have revealed that correlations between subsequently presented patterns shorten the length of such plateaus. We show how to extend the statistical mechanics framework to quantitatively check the e ect of correlations on learning in networks with a large number of input units. The surprisingly compact description we obtain makes it possible to derive properties

Connection Science, 2007

Cascade-correlation (cascor) networks grow by recruiting hidden units to adjust their computational power to the task being learned. The standard cascor algorithm recruits each hidden unit on a new layer, creating deep networks. In contrast, the flat cascor variant adds all recruited hidden units on a single hidden layer. Student-teacher network approximation tasks were used to investigate the ability of flat and standard cascor networks to learn the input-output mapping of other, randomly initialized flat and standard cascor networks. For lowcomplexity approximation tasks, there was no significant performance difference between flat and standard student networks. Contrary to the common belief that standard cascor does not generalize well due to cascading weights creating deep networks, we found that both standard and flat cascor generalized well on problems of varying complexity. On high-complexity tasks, flat cascor networks had fewer connection weights and learned with less computational cost than standard networks did.

The effect of conelations in neural networks is investigated by wnsider@g biased input and output palm'" Statistical mechanics is applied @ study training times and intend potentials of the MINOVER and ADALME leaming algorithms. For the latter, a direet extension to generalization abilify is obtained. Comparison with computer simulations shows good apemen1 with theoretical predictions. With biased pattems, we find

2000

We extend previous research on parameter dynamics of digital filters to examine weight sensitivity and interdependence in feedforward networks. Weight sensitivity refers to the effect of small weight perturbations on the network's output, and weight interdependence refers to the degree of co-linearity between weights. A combined measure of the weight space (τ), defined as the ratio of weight interdependence to sensitivity, is explored in networks with hidden-unit activation functions of different complexity in the contexts of learning (1) a nonlinearly separable bivariate normal classification task, (2) the XOR problem, (3) sigmoidal functions, and (4) sine functions. Simulations show that networks with more complex activation functions give rise to a smaller τ and more rapid learning, suggesting that weight sensitivity and interdependence together are indicative of network complexity and are predictive of learning efficiency. Weight Sensitivity and Interdependence 3 Characterizing Network Complexity and Learning Efficiency by the Ratio of Weight Interdependence to Sensitivity 1 Introduction Mathematical models of human cognition must explain complex phenomena but, at the same time, need to retain parsimony of description. Similarly, in practical applications, neural networks should learn rapidly and generalize extensively without excessive computational complexity (e.g., Hinton and van Camp, 1993; Hochreiter and Schmidhuber, 1997). The learning capabilities of a network can be readily ascertained; however, the measurement of model complexity continues to be subject to theoretical debate and research (e.g., proposed three primary indicators of model complexity: the number of parameters, the model's functional form, and the range of its parameter space. In network terms, the number of parameters corresponds to the number of units and weights, whereas functional form and range correspond to the nature of the activation function of those units. Although the number of units and adjustable weights can be readily verified, they are poor indicators of computational power because they ignore the functional form of the activation function and thus usually do not capture the intrinsic degrees of freedom of a network. A network's computational capacity derives instead from the feature space defined

Connection Science, 2001

Journal of Chemical Information and Computer Sciences, 1998

Cascade Learning (CL) [20] is a new adaptive approach to train deep neural networks. It is particularly suited to transfer learning, as learning is achieved in a layerwise fashion, enabling the transfer of selected layers to optimize the quality of transferred features. In the domain of Human Activity Recognition (HAR), where the consideration of resource consumption is critical, CL is of particular interest as it has demonstrated the ability to achieve significant reductions in computational and memory costs with negligible performance loss. In this paper, we evaluate the use of CL and compare it to end to end (E2E) learning in various transfer learning experiments, all applied to HAR. We consider transfer learning across objectives, for example opening the door features transferred to opening the dishwasher. We additionally consider transfer across sensor locations on the body, as well as across datasets. Over all of our experiments, we find that CL achieves state of the art performance for transfer learning in comparison to previously published work, improving F 1 scores by over 15%. In comparison to E2E learning, CL performs similarly considering F 1 scores, with the additional advantage of requiring fewer parameters. Finally, the overall results considering HAR classification performance and memory requirements demonstrate that CL is a good approach for transfer learning.

International Statistical Review, 2017

PCA is a statistical method, which is directly related to EVD and SVD. Neural networks-based PCA method estimates PC online from the input data sequences, which especially suits for high-dimensional data due to the avoidance of the computation of large covariance matrix, and for the tracking of nonstationary data, where the covariance matrix changes slowly over time. Neural networks and algorithms for PCA will be described in this chapter, and algorithms given in this chapter are typically unsupervised learning methods. PCA has been widely used in engineering and scientific disciplines, such as pattern recognition, data compression and coding, image processing, high-resolution spectrum analysis, and adaptive beamforming. PCA is based on the spectral analysis of the second moment matrix that statistically characterizes a random vector. PCA is directly related to SVD, and the most common way to perform PCA is via the SVD of a data matrix. However, the capability of SVD is limited for very large data sets. It is well known that preprocessing usually maps a high-dimensional space to a low-dimensional space with the least information loss, which is known as feature extraction. PCA is a well-known feature extraction method, and it allows the removal of the second-order correlation among given random processes. By calculating the eigenvectors of the covariance matrix of the input vector, PCA linearly transforms a high-dimensional input vector into a low-dimensional one whose components are uncorrelated.

1992

In [5], a new incremental cascade network architecture has been presented. This paper discusses the properties of such cascade networks and investigates their generalization abilities under the particular constraint of small data sets. The evaluation is done for cascade networks consisting of local linear maps using the MackeyGlass time series prediction task as a benchmark. Our results indicate that to bring the potential of large networks to bear on the problem of extracting information from small data sets without running the risk of overjitting, deeply cascaded network architectures are more favorable than shallow broad architectures that contain the same number of nodes.