Deep Subspace Clustering Networks
Srinivasan Nallathambi
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
We present a novel deep neural network architecture for unsupervised subspace clustering. This architecture is built upon deep auto-encoders, which non-linearly map the input data into a latent space. Our key idea is to introduce a novel self-expressive layer between the encoder and the decoder to mimic the "self-expressiveness" property that has proven effective in traditional subspace clustering. Being differentiable, our new self-expressive layer provides a simple but effective way to learn pairwise affinities between all data points through a standard back-propagation procedure. Being nonlinear, our neural-network based method is able to cluster data points having complex (often nonlinear) structures. We further propose pre-training and fine-tuning strategies that let us effectively learn the parameters of our subspace clustering networks. Our experiments show that our method significantly outperforms the state-of-the-art unsupervised subspace clustering techniques.
Related papers
Deep subspace clustering to achieve jointly latent feature extraction and discriminative learning
Neurocomputing, 2020
Deep subspace clustering based on autoencoder and self-expression layer has become popular for clustering method. However, these models only focus on feature extraction by minimizing reconstruction error rather than specific clustering task, leading to unsatisfactory performance. To overcome the above shortcoming, we propose a deep subspace clustering framework which jointly extracts features via the embedding neural network and performs subspace learning. The model contains three modules, an autoencoder, a self-expression layer and a supervised competitive feature learning module. The proposed model is highly capable of capturing characteristic features to guide the clustering task by using relative entropy to minimize probabilistic cluster assignments and the target variables. The three modules are consolidated to be jointly trained and optimized competitively. Experimental results on five benchmark datasets demonstrate the effectiveness of the proposed deep subspace clustering by comparing with eleven baseline methods.
Multi-view Deep Subspace Clustering Networks
2019
Multi-view subspace clustering aims to discover the inherent structure by fusing multi-view complementary information. Most existing methods first extract multiple types of hand-crafted features and then learn a joint affinity matrix for clustering. The disadvantage lies in two aspects: 1) Multi-view relations are not embedded into feature learning. 2) The end-to-end learning manner of deep learning is not well used in multi-view clustering. To address the above issues, we propose a novel multi-view deep subspace clustering network (MvDSCN) by learning a multi-view self-representation matrix in an end-to-end manner. MvDSCN consists of two sub-networks, i.e., diversity network (Dnet) and universality network (Unet). A latent space is built upon deep convolutional auto-encoders and a self-representation matrix is learned in the latent space using a fully connected layer. Dnet learns view-specific self-representation matrices while Unet learns a common self-representation matrix for al...
Pseudo-Supervised Deep Subspace Clustering
IEEE Transactions on Image Processing, 2021
Auto-Encoder (AE)-based deep subspace clustering (DSC) methods have achieved impressive performance due to the powerful representation extracted using deep neural networks while prioritizing categorical separability. However, self-reconstruction loss of an AE ignores rich useful relation information and might lead to indiscriminative representation, which inevitably degrades the clustering performance. It is also challenging to learn high-level similarity without feeding semantic labels. Another unsolved problem facing DSC is the huge memory cost due to n × n similarity matrix, which is incurred by the selfexpression layer between an encoder and decoder. To tackle these problems, we use pairwise similarity to weigh the reconstruction loss to capture local structure information, while a similarity is learned by the self-expression layer. Pseudo-graphs and pseudolabels, which allow benefiting from uncertain knowledge acquired during network training, are further employed to supervise similarity learning. Joint learning and iterative training facilitate to obtain an overall optimal solution. Extensive experiments on benchmark datasets demonstrate the superiority of our approach. By combining with the k-nearest neighbors algorithm, we further show that our method can address the large-scale and out-ofsample problems.
Hypergraph-Supervised Deep Subspace Clustering
Mathematics, 2021
Auto-encoder (AE)-based deep subspace clustering (DSC) methods aim to partition high-dimensional data into underlying clusters, where each cluster corresponds to a subspace. As a standard module in current AE-based DSC, the self-reconstruction cost plays an essential role in regularizing the feature learning. However, the self-reconstruction adversely affects the discriminative feature learning of AE, thereby hampering the downstream subspace clustering. To address this issue, we propose a hypergraph-supervised reconstruction to replace the self-reconstruction. Specifically, instead of enforcing the decoder in the AE to merely reconstruct samples themselves, the hypergraph-supervised reconstruction encourages reconstructing samples according to their high-order neighborhood relations. By the back-propagation training, the hypergraph-supervised reconstruction cost enables the deep AE to capture the high-order structure information among samples, facilitating the discriminative featur...
Multilayer Graph Approach to Deep Subspace Clustering
arXiv (Cornell University), 2024
Deep subspace clustering (DSC) networks based on self-expressive model learn representation matrix, often implemented in terms of fully connected network, in the embedded space. After the learning is finished, representation matrix is used by spectral clustering module to assign labels to clusters. However, such approach ignores complementary information that exist in other layers of the encoder (including the input data themselves). Herein, we apply selected linear subspace clustering algorithm to learn representation matrices from representations learned by all layers of encoder network including the input data. Afterward, we learn a multilayer graph that in a multi-view like manner integrates information from graph Laplacians of all used layers. That improves further performance of selected DSC network. Furthermore, we also provide formulation of our approach to cluster out-ofsample/test data points. We validate proposed approach on four well-known datasets with two DSC networks as baseline models. In almost all the cases, proposed approach achieved statistically significant improvement in three performance metrics. MATLAB code of proposed algorithm is posted on .
Unsupervised Clustering for Deep Learning: A tutorial survey
2019
Unsupervised learning methods play an essential role in many deep learning approaches because the training of complex models with several parameters is an extremely datahungry process. The execution of such a training process in a fully supervised manner requires numerous labeled examples. Since the labeling of the training samples is very time-consuming, learning approaches that require less or no labeled examples are sought. Unsupervised learning can be used to extract meaningful information on the structure and hierarchies in the data, relying only on the data samples without any ground truth provided. The extracted knowledge representation can be used as a basis for a deep model that requires less labeled examples, as it already has a good understanding of the hidden nature of the data and should be only fine-tuned for the specific task. The trend for deep learning applications most likely leads to substituting as much portion of supervised learning methods with unsupervised lea...
Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Clustering is among the most fundamental tasks in machine learning and artificial intelligence. In this paper, we propose Variational Deep Embedding (VaDE), a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE). Specifically, VaDE models the data generative procedure with a Gaussian Mixture Model (GMM) and a deep neural network (DNN): 1) the GMM picks a cluster; 2) from which a latent embedding is generated; 3) then the DNN decodes the latent embedding into an observable. Inference in VaDE is done in a variational way: a different DNN is used to encode observables to latent embeddings, so that the evidence lower bound (ELBO) can be optimized using the Stochastic Gradient Variational Bayes (SGVB) estimator and the reparameterization trick. Quantitative comparisons with strong baselines are included in this paper, and experimental results show that VaDE significantly outperforms the state-of-the-art clustering methods on 5 benchmark...
Non-Parametric Clustering Using Deep Neural Networks
IEEE Access, 2020
In this paper, a novel algorithm for non-parametric image clustering, is proposed. Non-parametric clustering methods operate by considering the number of clusters unknown as opposed to parametric clustering, where the number of clusters is known a priori. In the present work, a deep neural network is trained, in order to decide whether an arbitrary sized group of elements can be considered as a unique cluster or it consists of more than one clusters. Using this trained neural network as clustering criterion, an iterative algorithm is built, able to cluster any given dataset. Evaluation of the proposed method on several public datasets shows that the proposed method is either on par or outperforms state-of-the-art methods even when compared to parametric image clustering methods. The proposed method is additionally able to correctly cluster input samples from a completely different dataset than the one it has been trained on, as well as data coming from different modalities. Results on cross-dataset clustering show evidence of the generalization potential of the proposed method. INDEX TERMS Cross-dataset, high dimensional clustering, machine learning, non-parametric.
Deep Temporal Clustering : Fully Unsupervised Learning of Time-Domain Features
ArXiv, 2018
Unsupervised learning of time series data, also known as temporal clustering, is a challenging problem in machine learning. Here we propose a novel algorithm, Deep Temporal Clustering (DTC), to naturally integrate dimensionality reduction and temporal clustering into a single end-to-end learning framework, fully unsupervised. The algorithm utilizes an autoencoder for temporal dimensionality reduction and a novel temporal clustering layer for cluster assignment. Then it jointly optimizes the clustering objective and the dimensionality reduction objec tive. Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric. Several similarity metrics and state-of-the-art algorithms are considered and compared. To gain insight into temporal features that the network has learned for its clustering, we apply a visualization method that generates a region of interest heatmap for the time series. The viability of the algorithm is demon...
CAEclust: A Consensus of Autoencoders Representations for Clustering
Image Processing On Line
The CAEclust Python package implements an original deep spectral clustering in an ensemble framework. Recently, strategies combining classical clustering approaches and deep autoencoders have been proposed, but their effectiveness is impeded by deep network hyperparameters settings. We alleviate this issue with a consensus solution that hinges on the fusion of multiple deep autoencoder representations and spectral clustering. CAEclust offers an efficient merging of encodings by using the landmarks strategy and demonstrates its effectiveness on benchmark data. CAEclust enables to reproduce our experiments and explore novel datasets.
Related papers
Robust Self-Supervised Convolutional Neural Network for Subspace Clustering and Classification
2020
Insufficient capability of existing subspace clustering methods to handle data coming from nonlinear manifolds, data corruptions, and out-of-sample data hinders their applicability to address real-world clustering and classification problems. This paper proposes the robust formulation of the self-supervised convolutional subspace clustering network ($S^2$ConvSCN) that incorporates the fully connected (FC) layer and, thus, it is capable for handling out-of-sample data by classifying them using a softmax classifier. $S^2$ConvSCN clusters data coming from nonlinear manifolds by learning the linear self-representation model in the feature space. Robustness to data corruptions is achieved by using the correntropy induced metric (CIM) of the error. Furthermore, the block-diagonal (BD) structure of the representation matrix is enforced explicitly through BD regularization. In a truly unsupervised training environment, Robust $S^2$ConvSCN outperforms its baseline version by a significant am...
Deep Subspace Clustering with Data Augmentation
2020
The idea behind data augmentation techniques is based on the fact that slight changes in the percept do not change the brain cognition. In classification, neural networks use this fact by applying transformations to the inputs to learn to predict the same label. However, in deep subspace clustering (DSC), the ground-truth labels are not available, and as a result, one cannot easily use data augmentation techniques. We propose a technique to exploit the benefits of data augmentation in DSC algorithms. We learn representations that have consistent subspaces for slightly transformed inputs. In particular, we introduce a temporal ensembling component to the objective function of DSC algorithms to enable the DSC networks to maintain consistent subspaces for random transformations in the input data. In addition, we provide a simple yet effective unsupervised procedure to find efficient data augmentation policies. An augmentation policy is defined as an image processing transformation with...
DAC: Deep Autoencoder-based Clustering, a General Deep Learning Framework of Representation Learning
ArXiv, 2021
Clustering performs an essential role in many real world applications, such as market research, pattern recognition, data analysis, and image processing. However, due to the high dimensionality of the input feature values, the data being fed to clustering algorithms usually contains noise and thus could lead to in-accurate clustering results. While traditional dimension reduction and feature selection algorithms could be used to address this problem, the simple heuristic rules used in those algorithms are based on some particular assumptions. When those assumptions does not hold, these algorithms then might not work. In this paper, we propose DAC, Deep Autoencoder-based Clustering, a generalized datadriven framework to learn clustering representations using deep neuron networks. Experiment results show that our approach could effectively boost performance of the K-Means clustering algorithm on a variety types of datasets.
Adaptive Attribute and Structure Subspace Clustering Network
IEEE Transactions on Image Processing
Deep self-expressiveness-based subspace clustering methods have demonstrated effectiveness. However, existing works only consider the attribute information to conduct the selfexpressiveness, which may limit the clustering performance. In this paper, we propose a novel adaptive attribute and structure subspace clustering network (AASSC-Net) to simultaneously consider the attribute and structure information in an adaptive graph fusion manner. Specifically, we first exploit an autoencoder to represent input data samples with latent features for the construction of an attribute matrix. We also construct a mixed signed and symmetric structure matrix to capture the local geometric structure underlying data samples. Then, we perform self-expressiveness on the constructed attribute and structure matrices to learn their affinity graphs separately. Finally, we design a novel attention-based fusion module to adaptively leverage these two affinity graphs to construct a more discriminative affinity graph. Extensive experimental results on commonly used benchmark datasets demonstrate that our AASSC-Net significantly outperforms state-of-the-art methods. In addition, we conduct comprehensive ablation studies to discuss the effectiveness of the designed modules. The code will be publicly available at https://github.com/ZhihaoPENG-CityU.
Multi-View Feature Boosting Network for Deep Subspace Clustering
2022 IEEE International Conference on Image Processing (ICIP)
Subspace clustering is widely used to find clusters in different subspaces within a dataset. Autoencoders are popular deep subspace clustering methods using feature extraction and dimensional reduction. However, neural networks are vulnerable to overfitting, and therefore have limited potential for unsupervised subspace clustering. This paper proposes a deep multi-view subspace clustering network with feature boosting module to successfully extract meaningful features in different views and to fuse multi-view representations in a complementary manner for enhanced clustering results. The multi-view boosting provides the robust features for unsupervised clustering by emphasizing the features and removing the redundant noise. Quantitative and qualitative analysis on various benchmark datasets verifies that the proposed method outperforms state-of-the-art subspace clustering methods.
Deep Clustering: A Comprehensive Survey
arXiv (Cornell University), 2022
Cluster analysis plays an indispensable role in machine learning and data mining. Learning a good data representation is crucial for clustering algorithms. Recently, deep clustering, which can learn clustering-friendly representations using deep neural networks, has been broadly applied in a wide range of clustering tasks. Existing surveys for deep clustering mainly focus on the single-view fields and the network architectures, ignoring the complex application scenarios of clustering. To address this issue, in this paper we provide a comprehensive survey for deep clustering in views of data sources. With different data sources and initial conditions, we systematically distinguish the clustering methods in terms of methodology, prior knowledge, and architecture. Concretely, deep clustering methods are introduced according to four categories, i.e., traditional single-view deep clustering, semi-supervised deep clustering, deep multi-view clustering, and deep transfer clustering. Finally, we discuss the open challenges and potential future opportunities in different fields of deep clustering.
Deep Clustering Based On A Mixture Of Autoencoders
2019
In this paper we propose a Deep Autoencoder Mixture Clustering (DAMIC) algorithm based on a mixture of deep autoencoders where each cluster is represented by an autoencoder. A clustering network transforms the data into another space and then selects one of the clusters. Next, the autoencoder associated with this cluster is used to reconstruct the data-point. The clustering algorithm jointly learns the nonlinear data representation and the set of autoencoders. The optimal clustering is found by minimizing the reconstruction loss of the mixture of autoencoder network. Unlike other deep clustering algorithms, no regularization term is needed to avoid data collapsing to a single point. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.
Deeply transformed subspace clustering
Signal Processing, 2020
Subspace clustering assumes that the data is separable into separate subspaces; this assumption may not always hold. For such cases, we assume that, even if the raw data is not separable into subspaces, one can learn a deep representation such that the learnt representation is separable into subspaces. To achieve the intended goal, we embed subspace clustering techniques (locally linear manifold clustering, sparse subspace clustering and low rank representation) into deep transform learning. The entire formulation is jointly learnt; giving rise to a new class of methods called deeply transformed subspace clustering (DTSC). To test the performance of the proposed techniques, benchmarking is performed on image clustering problems. Comparison with state-of-the-art clustering techniques shows that our formulation improves upon them.
Image clustering by autoencoders
Proceedings of the V International conference Information Technology and Nanotechnology 2019
This paper describes an approach to solving the problem of finding similar images by visual similarity using neural networks on previously unmarked data. We propose to build special architecture of the neural network - autoencoder, through which high-level features are extracted from images. The search for the nearest elements is realized by the Euclidean metric in the generated feature space, after a preliminary decomposition into two-dimensional space. Proposed approach of generate feature space can be applied to the classification task using pre-clustering.
Self-supervised Dimensionality Reduction with Neural Networks and Pseudo-labeling
Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
Dimensionality reduction (DR) is used to explore high-dimensional data in many applications. Deep learning techniques such as autoencoders have been used to provide fast, simple to use, and high-quality DR. However, such methods yield worse visual cluster separation than popular methods such as t-SNE and UMAP. We propose a deep learning DR method called Self-Supervised Network Projection (SSNP) which does DR based on pseudo-labels obtained from clustering. We show that SSNP produces better cluster separation than autoencoders, has out-of-sample, inverse mapping, and clustering capabilities, and is very fast and easy to use.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.