Second order Hebbian neural networks and blind source separation

1996

Novel on{line learning algorithms with self adaptive learning rates (parameters) for blind separation of signals are proposed. The main motivation for development of new learning rules is to improve c o n vergence speed and to reduce cross{talking, especially for non{stationary signals. Furthermore, we have discovered that under some conditions the proposed neural network models with associated learning algorithms exhibit a random switch of attention, i.e. they have ability of chaotic or random switching or cross{over of output signals in such way that a specied separated signal may a p p e a r a t v arious outputs at dierent time windows. Validity, performance and dynamic properties of the proposed learning algorithms are investigated by computer simulation experiments. 157

Neurocomputing, 1999

Blind source separation problems have recently drawn a lot of attention in unsupervised neural learning. In the current approaches, the number of sources is typically assumed to be known in advance, but this does not usually hold in practical applications. In this paper, various neural network architectures and associated adaptive learning algorithms are discussed for handling the cases where the number of sources is unknown. These techniques include estimation of the number of sources, redundancy removal among the outputs of the networks, and extraction of the sources one at a time. Validity and performance of the described approaches are demonstrated by extensive computer simulations for natural image and magnetoencephalographic (MEG) data. 0925-2312/99/$ -see front matter 1999 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 -2 3 1 2 ( 9 8 ) 0 0 0 9 1 -5

1997

IEEE Transactions on Neural Networks, 2004

Independent component analysis is to extract independent signals from their linear mixtures without assuming prior knowledge of their mixing coefficients. As we know, a number of factors are likely to affect separation results in practical applications, such as the number of active sources, the distribution of source signals, and noise.The purpose of this paper to develop a general framework of blind separation from a practical point of view with special emphasis on the activation function adaptation. First, we propose the exponential generative model for probability density functions. A method of constructing an exponential generative model from the activation functions is discussed. Then, a learning algorithm is derived to update the parameters in the exponential generative model. The learning algorithm for the activation function adaptation is consistent with the one for training the demixing model. Stability analysis of the learning algorithm for the activation function is also discussed. Both theoretical analysis and simulations show that the proposed approach is universally convergent regardless of the distributions of sources. Finally, computer simulations are given to demonstrate the effectiveness and validity of the approach.

1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258), 1999

The information-theoretic framework for source separation is highly suitable. However the choice of the nonlinearity or the estimation of the multidimensional joint probability density function are nontrivial. We propose here a generalized Gaussian model to construct a generalized blind source separation network based on the minimum entropy principle. This new separation network can suppress the interference to a significant amount compared to the traditional LMS-echo-canceler. The simulation is given to show the disparity of the performance as α varies. Finally how to choose the appropriate α in our generalized anti-Hebbian rule is discussed.

International journal of neural systems

In standard blind source separation, one tries to extract unknown source signals from their instantaneous linear mixtures by using a minimum of a priori information. We have recently shown that certain nonlinear extensions of principal component type neural algorithms can be successfully applied to this problem. In this paper, we show that a nonlinear PCA criterion can be minimized using least-squares approaches, leading to computationally efficient and fast converging algorithms. Several versions of this approach are developed and studied, some of which can be regarded as neural learning algorithms. A connection to the nonlinear PCA subspace rule is also shown. Experimental results are given, showing that the least-squares methods usually converge clearly faster than stochastic gradient algorithms in blind separation problems.

2001

Abstract This paper presents a novel method for blindly separating unobservable independent source signals from their nonlinear mixtures. The demixing system is modeled using a parameterized neural network whose parameters can be determined under the criterion of independence of its outputs. Two cost functions based on higher order statistics are established to measure the statistical dependence of the outputs of the demixing system.

Neurocomputing, 2009

Principal Component Analysis is often thought of as a preprocessing step for Blind Source Separation (BSS). Although second order methods have been proposed for BSS in the past, these approaches cannot be easily implemented by neural models. In this paper we demonstrate that PCA is more than a preprocessing step and, in fact, it can be used directly for solving the BSS problem in combination with very simple temporal filtering process. We also demonstrate that a PCA extension called Oriented PCA (OPCA) can be also used for the same purpose without prewhitening the observed data. Both approaches can be implemented using efficient neural models that are shown to successfully extract the hidden sources.

Proceedings of the International Joint Conference on Neural Networks, 2003.

In this paper, we present an algorithm that minimizes the mutual information between the outputs of a perceptron with two hidden layers. The neural network is then used as separating system in the NonLinear Blind Source Separation problem.

Signal Processing, 1991

The separation of independent sources from an array of sensors is a classical but difficult problem in signal processing. Based on some biological observations, an adaptive algorithm is proposed to separate simultaneously all the unknown independent sources. The adaptive rule, which constitutes an independence test using non-linear functions, is the main original point of this blind identification procedure. Moreover, a new concept, that of INdependent Components Analysis (INCA), more powerful than the classical Principal Components Analysis (in decision tasks) emerges from this work.

1996

In this paper an adaptive approach to cancellation of additive, convolutional noise from many{source mixtures with a simultaneous blind source separation is proposed. Associated neural network learning algorithms are developed on the basis of decorrelation principle and energy minimization of output signals. The reference noise is transformed into a convolutional one by employing an adaptive FIR lter in each channel. Several models of NN learning processes are considered. In the basic approach the noisy signals are separated simultaneously with the additive noise cancellation. The simplied model employs separate learning steps for noise cancellation and source separation. Multi{layer neural networks improve the quality of results. Results of comparative tests of proposed methods are provided.

Artificial Neural Networks—ICANN …, 2001

International Symposium on Neural Networks, 1995

Studies the application of some nonlinear neural pricipal component analysis (PCA) type approaches to the separation of independent source signals from their linear mixture. This problem is important in signal processing and communications, and it cannot be solved using standard PCA. Using prewhitening and appropriate choice of nonlinearities, several algorithms proposed by the authors yield good separation results for sub-Gaussian

Proceedings of the International Joint Conference on Neural Networks, 2003., 2003

A comparative study of neural implementations running principal component analysis (PCA) and independent component analysis (ICA) was carried out. Both artificially generated data and real biomedical time series were employed in order to critically evaluate and assess the performance of various algorithms under study. The assumption of independence, even if weak, was proved reach in relevant interferences on brain activity.

1997 IEEE International Conference on Acoustics, Speech, and Signal Processing

This paper provides a detailed and rigorous analysis of the two commonly used methods for redundancy reduction: Linear Independent Component Analysis (ICA) and Information Maximization (InfoMax). The paper shows analytically that ICA based on the Kullback-Leibler information as a mutual information measure and InfoMax lead to the same solution if the parameterization of the output nonlinear functions in the latter method is sufficiently rich. Furthermore, this work briefly discusses the alternative redundancy measures not based on the Kullback-Leibler information distance and Nonlinear ICA. The practical issues of applying ICA and InfoMax are also discussed.