Academia.eduAcademia.edu

Adaptive Principal component EXtraction (APEX) and applications

Profile image of Konstantinos DiamantarasKonstantinos Diamantaras

1994, IEEE Transactions on Signal Processing

Abstract

In this paper we describe a neural network model (APEX) for multiple principal component extraction. All the synaptic weights of the model are trained with the normalized Hebbian learning rule. The network structure features a hierarchical set of lateral connections among the output units which serve the purpose of weight orthogonalization. This structure also allows the size of the model to grow or shrink without need for retraining the old units. The exponential convergence of the network is formally proved while there is significant performance improvement over previous methods. By establishing an important connection with the recursive least squares algorithm we have been able to provide the optimal size for the learning step-size parameter which leads to a significant improvement in the convergence speed. This is in contrast with previous neural PCA models which lack such numerical advantages. The APEX algorithm is also parallelizable allowing the concurrent extraction of multiple principal components. Furthermore, APEX is shown to be applicable to the constrained PCA problem where the signal variance is maximized under external orthogonality constraints. We then study various principal component analysis (PCA) applications that might benefit from the adaptive solution offered by APEX. In particular we discuss applications in spectral estimation, signal detection and image compression and filtering, while other application domains are also briefly outlined.

Figures (17)
Fig. 1. The APEX model. The solid lines denote the weights w;, cj, which are trained at the sth stage.The dashed lines correspond to the weights of the already trained neurons. Note that the lateral weights asymptotically converge to zero so they don’t appear between the already trained units. weights in the network will span an orthonormal basis of the m-dimensional principal component subspace, while the lateral weights tend to zero as orthogonalization is achieved. Thus, we associate the term “anti-Hebbian connections” or “orthogonalization connections” with c;, in contrast to the term “Hebbian connections” associated with the feed-forward weights. The mathematical details of the anti-Hebbian rule (also called Lateral Orthogonalization Rule) used to train c; are to follow.
Fig. 1. The APEX model. The solid lines denote the weights w;, cj, which are trained at the sth stage.The dashed lines correspond to the weights of the already trained neurons. Note that the lateral weights asymptotically converge to zero so they don’t appear between the already trained units. weights in the network will span an orthonormal basis of the m-dimensional principal component subspace, while the lateral weights tend to zero as orthogonalization is achieved. Thus, we associate the term “anti-Hebbian connections” or “orthogonalization connections” with c;, in contrast to the term “Hebbian connections” associated with the feed-forward weights. The mathematical details of the anti-Hebbian rule (also called Lateral Orthogonalization Rule) used to train c; are to follow.
which is the same as APEX equation 3. The only difference is that now we have a specific choice for the value of 2; which is optimal in the sense of the criterion Jy (w, N). Moreover, the optimal choice of the step-size parameter has a profound impact in the convergence speed of the algorithm as will be discussed in Section IV. Clearly @ can also be calculated iteratively by Jin (w. N) can be minimized iteratively using the RLS algo- rithm [30] which yields the following updating equations
which is the same as APEX equation 3. The only difference is that now we have a specific choice for the value of 2; which is optimal in the sense of the criterion Jy (w, N). Moreover, the optimal choice of the step-size parameter has a profound impact in the convergence speed of the algorithm as will be discussed in Section IV. Clearly @ can also be calculated iteratively by Jin (w. N) can be minimized iteratively using the RLS algo- rithm [30] which yields the following updating equations
Fig. 2. Comparison of the theoretical versus the actual decay rates via simulation.
Fig. 2. Comparison of the theoretical versus the actual decay rates via simulation.
Fig. 3. The parallel APEX network model.
Fig. 3. The parallel APEX network model.
Fig. 4. (a) Plot of the square distance jle,,, ~ w,,,4||7, between the actual components and the ones estimated using sequential APEX. The data are repeated in sweeps (1 sweep = 200 iterations). (b) Average y?_, over each sweep for cach neuron ji,
Fig. 4. (a) Plot of the square distance jle,,, ~ w,,,4||7, between the actual components and the ones estimated using sequential APEX. The data are repeated in sweeps (1 sweep = 200 iterations). (b) Average y?_, over each sweep for cach neuron ji,
yx is basically the projection of x; on the subspace spanned by the columns of W which is orthogonal to £. In [21] it was shown that the optimal solution to the CPC problem is
yx is basically the projection of x; on the subspace spanned by the columns of W which is orthogonal to £. In [21] it was shown that the optimal solution to the CPC problem is
Fig. 6. (a) The single-output model. The connections denote the weights w,.cy, Which are trained. (b) The multiple-output model. A. APEX Solves CPC The following facts can be easily shown regarding the eigenvalues/vectors of R,:
Fig. 6. (a) The single-output model. The connections denote the weights w,.cy, Which are trained. (b) The multiple-output model. A. APEX Solves CPC The following facts can be easily shown regarding the eigenvalues/vectors of R,:
Fig. 7. APEX model performance for CPC. We use 200 64-dimensional data vectors repeated in sweeps. (a) The convergence of the error ||qg¢ — ey ||? for each neuron m. (b) The output variance: average {y”} over each sweep for each neuron, The final variance equals approximately to the corresponding eigenvalue Amn.
Fig. 7. APEX model performance for CPC. We use 200 64-dimensional data vectors repeated in sweeps. (a) The convergence of the error ||qg¢ — ey ||? for each neuron m. (b) The output variance: average {y”} over each sweep for each neuron, The final variance equals approximately to the corresponding eigenvalue Amn.
SIMULATION RESULTS FOR HARMONIC RETRIEVAL; ‘*’ FOLLOWING A NUMBER DENOTES THAT THE FREQUENCY CANNOT BE RESOLVED TABLE I B. Signal Detection: Drifting Phase Sinusoids
SIMULATION RESULTS FOR HARMONIC RETRIEVAL; ‘*’ FOLLOWING A NUMBER DENOTES THAT THE FREQUENCY CANNOT BE RESOLVED TABLE I B. Signal Detection: Drifting Phase Sinusoids
In the non-noisy case we have: Let K be the total number of samples. Let We will also assume that the signal is only known in a finite interval, therefore only a finite dimensional estimate R,. of the correlation matrix can be computed.
In the non-noisy case we have: Let K be the total number of samples. Let We will also assume that the signal is only known in a finite interval, therefore only a finite dimensional estimate R,. of the correlation matrix can be computed.
In this section we propose to use principal component analysis for the detection problem discussed above. The idea stems from the fact that in the coherent case, all eigenvalues of R,, except two, are equal to 0 while the principal component subspace corresponding to the nonzero ones is spanned by the quadrature signals cos(wk) and sin(wk). The optimal detector then projects the observation signal onto the optimal subspace following with a projection-energy threshold test. In the drifting phase case the signal spectrum has a nonzero bandwidth proportional to € and thus the eigenvalues of R.,, are all nonzero in general. However, if € is not too large then the first two eigenvalues clearly dominate, so in the extreme case where € = 0 (coherent case) only the first two eigenvalues are nonzero as discussed above. When € is nonzero however the space spanned by cos and sin is only as a rough approximation to the 2-D principal component subspace. This motivates us to compare the performance of the mth order noncoherent detector with the m-order detector produced after substituting cos and sin with the first two eigenvectors e; and e2 In our comparison experiments we used the same value of m as proposed in [46] for both A,, and A‘,. The eigenvectors are estimated using a 2-neuron parallel APEX network trained on the incoming data. We window the observation signal using a window of size N/m in order to produce vector data x; = [2j,---.2;~w/m4i] which are consequently used
In this section we propose to use principal component analysis for the detection problem discussed above. The idea stems from the fact that in the coherent case, all eigenvalues of R,, except two, are equal to 0 while the principal component subspace corresponding to the nonzero ones is spanned by the quadrature signals cos(wk) and sin(wk). The optimal detector then projects the observation signal onto the optimal subspace following with a projection-energy threshold test. In the drifting phase case the signal spectrum has a nonzero bandwidth proportional to € and thus the eigenvalues of R.,, are all nonzero in general. However, if € is not too large then the first two eigenvalues clearly dominate, so in the extreme case where € = 0 (coherent case) only the first two eigenvalues are nonzero as discussed above. When € is nonzero however the space spanned by cos and sin is only as a rough approximation to the 2-D principal component subspace. This motivates us to compare the performance of the mth order noncoherent detector with the m-order detector produced after substituting cos and sin with the first two eigenvectors e; and e2 In our comparison experiments we used the same value of m as proposed in [46] for both A,, and A‘,. The eigenvectors are estimated using a 2-neuron parallel APEX network trained on the incoming data. We window the observation signal using a window of size N/m in order to produce vector data x; = [2j,---.2;~w/m4i] which are consequently used
The difficulty of calculating the likelihood ratio in (4Q) makes the optimal detector in (47) impractical to use. Even the approximation (48) is very computationally costly and has the additional disadvantage that it is difficult to implement in an analog circuit which could alleviate the computational cost problem. Various authors have investigated alternatives to the optimal solution. In [14] the approach of estimating 6; via an Extended Kalman Filter is considered. Once 4; is approximated, a coherent detector is used for testing the two hypotheses. In [46] the authors compare the performance of the mth order noncoherent detector (proposed in [13]) which uses the test statistic with the standard noncoherent detector (i.e., the quadrature Jetector) and the optimal quadratic detector (48). The authors Jerived the optimal value of m for the mth order detector and found that for this value, it has comparable performance io the optimal quadratic detector and is quite a lot simpler (o implement, while it is significantly better than the standard noncoherent detector.
The difficulty of calculating the likelihood ratio in (4Q) makes the optimal detector in (47) impractical to use. Even the approximation (48) is very computationally costly and has the additional disadvantage that it is difficult to implement in an analog circuit which could alleviate the computational cost problem. Various authors have investigated alternatives to the optimal solution. In [14] the approach of estimating 6; via an Extended Kalman Filter is considered. Once 4; is approximated, a coherent detector is used for testing the two hypotheses. In [46] the authors compare the performance of the mth order noncoherent detector (proposed in [13]) which uses the test statistic with the standard noncoherent detector (i.e., the quadrature Jetector) and the optimal quadratic detector (48). The authors Jerived the optimal value of m for the mth order detector and found that for this value, it has comparable performance io the optimal quadratic detector and is quite a lot simpler (o implement, while it is significantly better than the standard noncoherent detector.
Fig. 8. Probability of miss comparison between (a) the mth order nonco- herent detector and (b) eigenvalue-based detector.
Fig. 8. Probability of miss comparison between (a) the mth order nonco- herent detector and (b) eigenvalue-based detector.
Fig. 9. Deflection ratio comparison between (a) the mth-order noncoherent detector and (b) the eigenvalue-based detector.
Fig. 9. Deflection ratio comparison between (a) the mth-order noncoherent detector and (b) the eigenvalue-based detector.
Fig. 10. The CPC technique may be applied to compute hybrid DCT and KL transform codes for image compression, using & DCT components and m Constrained Principal Components. Shown here are (a) the original image, (b) the compressed/decoded image using only & = 8 DCT components (SNR = 11.6), (c) using only & = 16 DCT components (SNR = 12.8), (d) combined DCT/CPC, with & = 4, m = 4, (SNR = 17.8), (e) conventional PC approach, using 2 = & PC’s, (SNR = 18.8), and (f} combined DCT/CPC, with & = 8, mr = §, (SNR = 22.5).
Fig. 10. The CPC technique may be applied to compute hybrid DCT and KL transform codes for image compression, using & DCT components and m Constrained Principal Components. Shown here are (a) the original image, (b) the compressed/decoded image using only & = 8 DCT components (SNR = 11.6), (c) using only & = 16 DCT components (SNR = 12.8), (d) combined DCT/CPC, with & = 4, m = 4, (SNR = 17.8), (e) conventional PC approach, using 2 = & PC’s, (SNR = 18.8), and (f} combined DCT/CPC, with & = 8, mr = §, (SNR = 22.5).
Fig. 11. The CPC analysis is applied to remove interference (rain) fram a picture. The constraint matrix V is set to be equal to the “rain” component. Shown here are images of (a). picture with rain (ie, a = 0); (b) using a “hard” factor a = 1, note that the “white rain” is converted to the “dark rain”; (c) using a softer a = 0.5, the “white rain” becomes lighter—but is still on the “white” side—hinting a harder factor needed; (d) using a “harder” factor a = 0.6, the rain almost disappears.
Fig. 11. The CPC analysis is applied to remove interference (rain) fram a picture. The constraint matrix V is set to be equal to the “rain” component. Shown here are images of (a). picture with rain (ie, a = 0); (b) using a “hard” factor a = 1, note that the “white rain” is converted to the “dark rain”; (c) using a softer a = 0.5, the “white rain” becomes lighter—but is still on the “white” side—hinting a harder factor needed; (d) using a “harder” factor a = 0.6, the rain almost disappears.

Related papers

Principal Component Analysis Networks and Algorithms Xiangyu Kong, Changhua Hu and Zhansheng Duan Springer, 2017, XXII + 323 pages, £112.00, hardcover ISBN: 978-9-811-02913-4
Ravindra Khattree

International Statistical Review, 2017

PCA is a statistical method, which is directly related to EVD and SVD. Neural networks-based PCA method estimates PC online from the input data sequences, which especially suits for high-dimensional data due to the avoidance of the computation of large covariance matrix, and for the tracking of nonstationary data, where the covariance matrix changes slowly over time. Neural networks and algorithms for PCA will be described in this chapter, and algorithms given in this chapter are typically unsupervised learning methods. PCA has been widely used in engineering and scientific disciplines, such as pattern recognition, data compression and coding, image processing, high-resolution spectrum analysis, and adaptive beamforming. PCA is based on the spectral analysis of the second moment matrix that statistically characterizes a random vector. PCA is directly related to SVD, and the most common way to perform PCA is via the SVD of a data matrix. However, the capability of SVD is limited for very large data sets. It is well known that preprocessing usually maps a high-dimensional space to a low-dimensional space with the least information loss, which is known as feature extraction. PCA is a well-known feature extraction method, and it allows the removal of the second-order correlation among given random processes. By calculating the eigenvectors of the covariance matrix of the input vector, PCA linearly transforms a high-dimensional input vector into a low-dimensional one whose components are uncorrelated.

View PDFchevron_right
Adaptive learning algorithm for principal component analysis with partial data
Andrzej Cichocki

1996

In this paper a fast and ecient adaptive learning algorithm for estimation of the principal components is developed. It seems to be especially useful in applications with changing environment, where the learning process has to be repeated in on{line manner. The approach can be called the cascade recursive least square (CRLS) method, as it combines a cascade (hierarchical) neural network scheme for input signal reduction with the RLS (recursive least square) lter for adaptation of learning rates. Successful application of the CRLS method for 2{D image compression{reconstruction and its performance in comparison to other known PCA adaptive algorithms are also documented.

View PDFchevron_right
Neural Network for Principal Component Analysis with Applications in Image Compression
Vlamos Panayiotis, Panayiotis Vlamos, Catalina Cocianu

2007

Classical feature extraction and data projection methods have been extensively investigated in the pattern recognition and exploratory data analysis literature. Feature extraction and multivariate data projection allow avoiding the "curse of dimensionality", improve the generalization ability of classifiers and significantly reduce the computational requirements of pattern classifiers. During the past decade a large number of artificial neural networks and learning algorithms have been proposed for solving feature extraction problems, most of them being adaptive in nature and well-suited for many real environments where adaptive approach is required. Principal Component Analysis, also called Karhunen-Loeve transform is a well-known statistical method for feature extraction, data compression and multivariate data projection and so far it has been broadly used in a large series of signal and image processing, pattern recognition and data analysis applications.

View PDFchevron_right
Principal Component Analysis Networks and Algorithms
Ravindra Khattree

Principal Component Analysis Networks and Algorithms, 2017

PCA is a statistical method, which is directly related to EVD and SVD. Neural networks-based PCA method estimates PC online from the input data sequences, which especially suits for high-dimensional data due to the avoidance of the computation of large covariance matrix, and for the tracking of nonstationary data, where the covariance matrix changes slowly over time. Neural networks and algorithms for PCA will be described in this chapter, and algorithms given in this chapter are typically unsupervised learning methods. PCA has been widely used in engineering and scientific disciplines, such as pattern recognition, data compression and coding, image processing, high-resolution spectrum analysis, and adaptive beamforming. PCA is based on the spectral analysis of the second moment matrix that statistically characterizes a random vector. PCA is directly related to SVD, and the most common way to perform PCA is via the SVD of a data matrix. However, the capability of SVD is limited for very large data sets. It is well known that preprocessing usually maps a high-dimensional space to a low-dimensional space with the least information loss, which is known as feature extraction. PCA is a well-known feature extraction method, and it allows the removal of the second-order correlation among given random processes. By calculating the eigenvectors of the covariance matrix of the input vector, PCA linearly transforms a high-dimensional input vector into a low-dimensional one whose components are uncorrelated.

View PDFchevron_right
Neural Implementation of a Class of PCA Learning Algorithms
Catalina Cocianu

Economic computation and economic cybernetics studies and research / Academy of Economic Studies

Principal component analysis allows the identification of a linear transformation such that the axes of the resulted coordinate system correspond to the largest variability of the investigated signal. The advantages of using principal components reside from the fact that bands are uncorrelated and no information contained in one band can be predicted by the knowledge of the other bands, therefore the information contained by each band is maximum for the whole set of bits. The paper reports a series of conclusions concerning the performance and efficiency of some of the most frequently used PCA algorithms implemented on neural architectures.

View PDFchevron_right

Related topics

  • Image Processing
  • Data Compression
  • Image compression
  • Spectral Estimation

    • Related papers

    A Parallel PCA Neural Network Approach for Feature Extraction
    Luciano A Digiampietri

    Anais do 10. Congresso Brasileiro de Inteligência Computacional, 2016

    Principal Component Analysis (PCA) is a well known statistical method that has successfully been applied for reducing data dimensionality. Focusing on a neural network which approximates the results obtained by classical PCA, the main contribution of this work consists in introducing a parallel modeling for such network. A comparative study shows that the proposal presents promising results when a multi-core computer is available.

    View PDFchevron_right
    Some theoretical results on nonlinear principal components analysis
    Edward Malthouse

    Proceedings of 1995 American Control Conference - ACC'95, 1995

    View PDFchevron_right
    TAPCA: time adaptive self-organizing maps for adaptive principal components analysis
    Reza Safabakhsh

    Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205), 2001

    In this paper, we propose a neural network called Time Adaptive Principal Components Analysis (TAPCA) which is composed of a number of Time Adaptive Self-Organizing Map (TASOM) networks. Each TASOM in TAPCA network estimates one eigenvector of tlie correlation matrix of input vectors entered so far, without having to calculate the correlation matrix. This estimation is done in an online fashion. The input distribution can be nonstationary, too. The eigenvectors appear in order of importance: the first TASOM calculates tlie eigenvector corresponding to the largest eigenvalue of the correlation matrix, and so on. Tlie TAPCA network is tested in stationary environments, and is compared with tlie eigendecomposition (ED) method and GeneraliLed Hebbian Algorithm (GHA) network. It performs better tlian both methods and needs fewer samples to converge.

    View PDFchevron_right
    Kernel Online System for Fast Principal Component Analysis and its Adaptive Learning
    Antonina Bondarchuk

    International Journal of Computing, 2021

    An artificial neural system for data compression that sequentially processes linearly nonseparable classes is proposed. The main elements of this system include adjustable radial-basis functions (Epanechnikov’s kernels), an adaptive linear associator learned by a multistep optimal algorithm, and Hebb-Sanger neural network whose nodes are formed by Oja’s neurons. For tuning the modified Oja’s algorithm, additional filtering (in case of noisy data) and tracking (in case of nonstationary data) properties were introduced. The main feature of the proposed system is the ability to work in conditions of significant nonlinearity of the initial data that are sequentially fed to the system and have a non-stationary nature. The effectiveness of the developed approach was confirmed by the experimental results. The proposed kernel online neural system is designed to solve compression and visualization tasks when initial data form linearly nonseparable classes in general problem of Data Stream Mi...

    View PDFchevron_right
    A new class of APEX-like PCA algorithms
    Aurelio Uncini

    1998

    One of the most commonly known algorithm to perform neural Principal Component Analysis of real-valued random signals is the Kung-Diamantaras' Adaptive Principal component EXtractor (APEX) for a laterally-connected neural architecture.

    View PDFchevron_right
    An improved sequential method for principal component analysis
    Z. Wang, S. Fiori

    Pattern Recognition Letters, 2003

    In sequential principal component (PC) extraction, when increasing numbers of PCs are extracted the accumulated extraction error becomes dominant and makes a reliable extraction of the remaining PCs difficult. This paper presents an improved cascade recursive least squares method for PCsÕ extraction. The good features of the proposed approach are illustrated through simulation results, and include improved convergence speed and higher extraction accuracy.

    View PDFchevron_right
    An efficient, robust, and fast converging principal components extraction algorithm: SIPEX-G
    Jose Principe

    2002 11th European Signal Processing Conference, 2002

    Principal Components Analysis (PCA) is a very important statistical tool in signal processing, which has found successful applications in numerous engineering problems as well as other fields. In general, an on-line algorithm to adapt the PCA network to determine the principal projections of the input data is desired. The authors have recently introduced a fast, robust, and efficient PCA algorithm called SIPEX-G without detailed comparisons and analysis of performance. In this paper, we investigate the performance of SIPEX-G through Monte Carlo runs on synthetic data and on realistic problems where PCA is applied. These problems include direction of arrival estimation and subspace Wiener filtering.

    View PDFchevron_right
    Neural networks and principal component analysis: Learning from examples without local minima
    Pierre Baldi

    Neural networks, 1989

    View PDFchevron_right
    Nonlinear PCA: a new hierarchical approach
    Matthias Scholz

    Proceedings ESANN, 2002

    Traditionally, nonlinear principal component analysis (NLPCA) is seen as nonlinear generalization of the standard (linear) principal component analysis (PCA). So far, most of these generalizations rely on a symmetric type of learning. Here we propose an algorithm that extends PCA into NLPCA through a hierarchical type of learning. The hierarchical algorithm (h-NLPCA), like many versions of the symmetric one (s-NLPCA), is based on a multi-layer perceptron with an auto-associative topology, the learning rule of which has been upgraded to accommodate the desired discrimination between components. With h-NLPCA we seek not only the nonlinear subspace spanned by the optimal set of components, ideal for data compression, but we give particular interest to the order in which these components appear. Due to its hierarchical nature, our algorithm is shown to be very efficient in detecting meaningful nonlinear features from real world data, as well as in providing a nonlinear whitening. Furthermore, in a quantitative type of analysis, the h-NLPCA achieves better classification accuracies, with a smaller number of components than most traditional approaches.

    View PDFchevron_right
    A Multipurpose Linear Component Analysis Method Based on Modulated Hebb-Oja Learning Rule
    Marko Jankovic

    IEEE Signal Processing Letters, 2000

    This letter presents a Hebb-type learning algorithm for on-line linear calculation of principal components. The proposed method is based on a recently proposed cooperative-competitive concept, named the time-oriented hierarchical method. The algorithm performs deflation on the signal power rather than on the signal itself. It will be also shown when, or how, this algorithm can be used as a blind signal separation algorithm. The proposed synaptic efficacy learning rule does not need the explicit information about the value of the other efficacies to make individual efficacy modification. The number of necessary global calculation circuits is one. Index terms -adaptive algorithm, principal/independent component analysis EDICS: SAS-STAT Detection, estimation and classification theory and methods, statistical signal processing SAS-ADAP Adaptive systems and adaptive filtering

    View PDFchevron_right