A new class of APEX-like PCA algorithms

IEEE Transactions on Neural Networks, 2000

This paper presents a class of algorithms for principal component analysis obtained by modification of a class of algorithms for principal subspace analysis (PSA) known as Plumbley's General Stochastic Approximation. Modification of the algorithms is based on Time-Oriented Hierarchical Method. The method uses two distinct time scales. On a faster time scale PSA algorithm is responsible for the "behaviour" of all output neurons. On a slower time scale, output neurons will compete for fulfilment of their "own interests". On this scale, basis vectors in the principal subspace are rotated toward the principal eigenvectors.

Proceedings of 1995 American Control Conference - ACC'95, 1995

Proceedings ESANN, 2002

Traditionally, nonlinear principal component analysis (NLPCA) is seen as nonlinear generalization of the standard (linear) principal component analysis (PCA). So far, most of these generalizations rely on a symmetric type of learning. Here we propose an algorithm that extends PCA into NLPCA through a hierarchical type of learning. The hierarchical algorithm (h-NLPCA), like many versions of the symmetric one (s-NLPCA), is based on a multi-layer perceptron with an auto-associative topology, the learning rule of which has been upgraded to accommodate the desired discrimination between components. With h-NLPCA we seek not only the nonlinear subspace spanned by the optimal set of components, ideal for data compression, but we give particular interest to the order in which these components appear. Due to its hierarchical nature, our algorithm is shown to be very efficient in detecting meaningful nonlinear features from real world data, as well as in providing a nonlinear whitening. Furthermore, in a quantitative type of analysis, the h-NLPCA achieves better classification accuracies, with a smaller number of components than most traditional approaches.

2002 11th European Signal Processing Conference, 2002

Principal Components Analysis (PCA) is a very important statistical tool in signal processing, which has found successful applications in numerous engineering problems as well as other fields. In general, an on-line algorithm to adapt the PCA network to determine the principal projections of the input data is desired. The authors have recently introduced a fast, robust, and efficient PCA algorithm called SIPEX-G without detailed comparisons and analysis of performance. In this paper, we investigate the performance of SIPEX-G through Monte Carlo runs on synthetic data and on realistic problems where PCA is applied. These problems include direction of arrival estimation and subspace Wiener filtering.

2009 International Joint Conference on Neural Networks, 2009

Principal component analysis (PCA) is a commonly applied technique for data analysis and processing, e.g. compression or clustering. In this paper we propose a probabilistic PCA model based on the Born rule. In off-line realization it can be seen as a successive optimization problem. In the on-line realization it will be solved by introduction of two different time scales. It will be shown that recently proposed time oriented hierarchical method, used for realization of biologically plausible PCA neural networks, represents a special case of the proposed model. The proposed model gives a general framework for creating different PCA realizations/algorithms. A particular realization can optimize locality of calculation, convergence speed, preciseness or some other parameter of interest. We will present some experimental results to illustrate effectiveness of the proposed model.

Kramer's nonlinear principal components analysis (NLPCA) neural networks are feedforward autoassociative networks with five layers. The third layer has fewer nodes than the input or output layers. This paper proposes a geometric interpretation for Kramer's method by showing that NLPCA fits a lower-dimensional curve or surface through the training data. The first three layers project observations onto the curve or surface giving scores. The last three layers define the curve or surface. The first three layers are a continuous function, which we show has several implications: NLPCA "projections" are suboptimal producing larger approximation error, NLPCA is unable to model curves and surfaces that intersect themselves, and NLPCA cannot parameterize curves with parameterizations having discontinuous jumps. We establish results on the identification of score values and discuss their implications on interpreting score values. We discuss the relationship between NLPCA and principal curves and surfaces, another nonlinear feature extraction method.

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 11, NO. 2, MARCH 2000, 2000

We derive and discuss new adaptive algorithms for principal component analysis (PCA) that are shown to converge faster than the traditional PCA algorithms due to Oja, Sanger, and Xu. It is well known that traditional PCA algorithms that are derived by using gradient descent on an objective function are slow to converge. Furthermore, the convergence of these algorithms depends on appropriate choices of the gain sequences. Since online applications demand faster convergence and an automatic selection of gains, we present new adaptive algorithms to solve these problems. We first present an unconstrained objective function, which can be minimized to obtain the principal components. We derive adaptive algorithms from this objective function by using: 1) gradient descent; 2) steepest descent; 3) conjugate direction; and 4) Newton-Raphson methods. Although gradient descent produces Xu's LMSER algorithm, the steepest descent, conjugate direction, and Newton-Raphson methods produce new adaptive algorithms for PCA. We also provide a discussion on the landscape of the objective function, and present a global convergence proof of the adaptive gradient descent PCA algorithm using stochastic approximation theory. Extensive experiments with stationary and nonsta-tionary multidimensional Gaussian sequences show faster convergence of the new algorithms over the traditional gradient descent methods. We also compare the steepest descent adaptive algorithm with state-of-the-art methods on stationary and nonstationary sequences. Index Terms-Adaptive principal component analysis, eigende-composition, principal subspace analysis.

1996

In this paper a fast and ecient adaptive learning algorithm for estimation of the principal components is developed. It seems to be especially useful in applications with changing environment, where the learning process has to be repeated in on{line manner. The approach can be called the cascade recursive least square (CRLS) method, as it combines a cascade (hierarchical) neural network scheme for input signal reduction with the RLS (recursive least square) lter for adaptation of learning rates. Successful application of the CRLS method for 2{D image compression{reconstruction and its performance in comparison to other known PCA adaptive algorithms are also documented.

Pattern Recognition Letters, 2003

In sequential principal component (PC) extraction, when increasing numbers of PCs are extracted the accumulated extraction error becomes dominant and makes a reliable extraction of the remaining PCs difficult. This paper presents an improved cascade recursive least squares method for PCsÕ extraction. The good features of the proposed approach are illustrated through simulation results, and include improved convergence speed and higher extraction accuracy.

We present a modification of Oja's single unit P C learning rule that behaves optimally in a certain sense, if the unit output value -the representation value of the input signal-is corrupted with noise.

Lecture Notes in Computer Science, 1997

In this paper we present new developments of a previous work dealing with the problem of the strongly-constrained orthonormal analysis of random signals. In the former work a neural learning rule arising from the study of the dynamics of a massive system in an abstract space was introduced, and the set of equations describing the motion of such a system was directly interpreted as a learning rule for neural layers. This adaptation rule can be used to solve several problems where orthonormal matrices are involved. Here we show t wo applications of such an approach: one dealing with PCA and one dealing with ICA.

2006

This paper presents a class of algorithms for principal component analysis obtained by modification of a class of algorithms for principal subspace analysis (PSA) known as Plumbley's General Stochastic Approximation. Modification of the algorithms is based on Time-Oriented Hierarchical Method. The method uses two distinct time scales. On a faster time scale PSA algorithm is responsible for the "behaviour" of all output neurons. On a slower time scale, output neurons will compete for fulfilment of their "own interests". On this scale, basis vectors in the principal subspace are rotated toward the principal eigenvectors.

Encyclopedia of Statistics in Behavioral Science, 2002

Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205), 2001

In this paper, we propose a neural network called Time Adaptive Principal Components Analysis (TAPCA) which is composed of a number of Time Adaptive Self-Organizing Map (TASOM) networks. Each TASOM in TAPCA network estimates one eigenvector of tlie correlation matrix of input vectors entered so far, without having to calculate the correlation matrix. This estimation is done in an online fashion. The input distribution can be nonstationary, too. The eigenvectors appear in order of importance: the first TASOM calculates tlie eigenvector corresponding to the largest eigenvalue of the correlation matrix, and so on. Tlie TAPCA network is tested in stationary environments, and is compared with tlie eigendecomposition (ED) method and GeneraliLed Hebbian Algorithm (GHA) network. It performs better tlian both methods and needs fewer samples to converge.

Lecture Notes in Computational Science and Enginee, 2008

Although linear principal component analysis (PCA) originates from the work of Sylvester [67] and Pearson [51], the development of nonlinear counterparts has only received attention from the 1980s. Work on nonlinear PCA, or NLPCA, can be divided into the utilization of autoassociative neural networks, principal curves and manifolds, kernel approaches or the combination of these approaches. This article reviews existing algorithmic work, shows how a given data set can be examined to determine whether a conceptually more demanding NLPCA model is required and lists developments of NLPCA algorithms. Finally, the paper outlines problem areas and challenges that require future work to mature the NLPCA research field.