Bayesian Mixture Models For Semi-Supervised Clustering

2019

In most real-world applications of clustering, data is partially labeled by an expert. Classical clustering approaches have been extensively studied in the presence of partial labels, however little work has been done to treat the general case of Bayesian mixture models. In this paper, we propose a new approach to perform semisupervised clustering using parametric and non parametric mixture models. We show how our approach generalizes mixture models with different types of emission distributions and priors under the same theoretical framework for semi-supervised clustering. The partial labels intervene in the clustering in the form of a Hidden Markov Random Field (HMRF) that introduces a penalty if the partial labels are not respected. We demonstrate how to perform inference in both the finite and infinite case with priors on the mixture components and the parameters using variational inference. Our experimental evaluations on synthetic data show how the method can leverage the partial labels to choose the correct clustering and the correct number of clusters. We also show that by introducing a small fraction of partial labels our method improves the clustering accuracy and outperforms a strong baseline in the literature on benchmark datasets.

2009

We present a Bayesian variational inference scheme for semisupervised clustering in which data is supplemented with side information in the form of common labels. There is no mutual exclusion of classes assumption and samples are represented as a combinatorial mixture over multiple clusters. We illustrate performance on six datasets and find a positive comparison against constrained K-means clustering.

Bayesian Analysis, 2011

Proceedings of the Second International Conference on Signal Processing and Multimedia Applications, 2007

In this communication, we propose a novel approach to perform the unsupervised and non parametric clustering of n-D data upon a Bayesian framework. The iterative approach developed is derived from the Classification Expectation-Maximization (CEM) algorithm, in which the parametric modelling of the mixture density is replaced by a non parametric modelling using local kernels, and the posterior probabilities account for the coherence of current clusters through the measure of class-conditional entropies. Applications of this method to synthetic and real data including multispectral images are presented. The classification issues are compared with other recent unsupervised approaches, and we show that our method reaches a more reliable estimation of the number of clusters while providing slightly better rates of correct classification in average.

2014 22nd International Conference on Pattern Recognition, 2014

Clustering is one of the essential tasks in machine learning and statistical pattern recognition. One of the most popular approaches in cluster analysis is the one based on the parametric finite mixture model. However, often, parametric models are not well adapted to represent complex and realistic data sets. Another issue in the finite mixture model-based clustering approach is the one of selecting the number of mixture components. The Bayesian non-parametric statistical methods for clustering provide a principled way to overcome these issues. This paper proposes a new Bayesian non-parametric approach for clustering. It relies on an Infinite Gaussian mixture model with an eigenvalue decomposition of the covariance matrix of each cluster, and a Chinese Restaurant Process (CRP) prior over the hidden partition. The CRP prior allows to control the model complexity in a principled way, and to automatically learn the number of clusters from the data. The covariance matrix decomposition allows to fit various flexible models going from simplest spherical ones to the more complex general one. We develop a Gibbs sampler to learn the various models and apply it to simulated data and benchmarks, and a real-world data issued from a challenging problem of whale song decomposition. The obtained results highlight the interest of the proposed nonparametric parsimonious mixture model for clustering.

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017

We propose a general framework for discriminative Bayesian nonparametric clustering to promote the inter-discrimination among the learned clusters in a fully Bayesian nonparametric (BNP) manner. Our method combines existing BNP clustering and discriminative models by enforcing latent cluster indices to be consistent with the predicted labels resulted from probabilistic discriminative model. This formulation results in a well-defined generative process wherein we can use either logistic regression or SVM for discrimination. Using the proposed framework, we develop two novel discriminative BNP variants: the discriminative Dirichlet process mixtures, and the discriminative-state infinite HMMs for sequential data. We develop efficient data-augmentation Gibbs samplers for posterior inference. Extensive experiments in image clustering and dynamic location clustering demonstrate that by encouraging discrimination between induced clusters, our model enhances the quality of clustering in com...

2010

Most clustering algorithms produce a single clustering solution. Similarly, feature selection for clustering tries to find one feature subset where one interesting clustering solution resides. However, a single data set may be multi-faceted and can be grouped and interpreted in many different ways, especially for high dimensional data, where feature selection is typically needed. Moreover, different clustering solutions are interesting for different purposes. Instead of committing to one clustering solution, in this paper we introduce a probabilistic nonparametric Bayesian model that can discover several possible clustering solutions and the feature subset views that generated each cluster partitioning simultaneously. We provide a variational inference approach to learn the features and clustering partitions in each view. Our model allows us not only to learn the multiple clusterings and views but also allows us to automatically learn the number of views and the number of clusters in each view.

The Annals of Statistics, 2020

Motivated by problems in data clustering, we establish general conditions under which families of nonparametric mixture models are identifiable, by introducing a novel framework involving clustering overfitted parametric (i.e. misspecified) mixture models. These identifiability conditions generalize existing conditions in the literature, and are flexible enough to include for example mixtures of Gaussian mixtures. In contrast to the recent literature on estimating nonparametric mixtures, we allow for general nonparametric mixture components, and instead impose regularity assumptions on the underlying mixing measure. As our primary application, we apply these results to partition-based clustering, generalizing the notion of a Bayes optimal partition from classical parametric model-based clustering to nonparametric settings. Furthermore, this framework is constructive so that it yields a practical algorithm for learning identified mixtures, which is illustrated through several examples on real data. The key conceptual device in the analysis is the convex, metric geometry of probability measures on metric spaces and its connection to the Wasserstein convergence of mixing measures. The result is a flexible framework for nonparametric clustering with formal consistency guarantees.

2007

This paper proposes a new approach to model-based clustering under prior knowledge. The proposed formulation can be interpreted from two different angles: as penalized logistic regression, where the class labels are only indirectly observed (via the probability density of each class); as finite mixture learning under a grouping prior. To estimate the parameters of the proposed model, we derive a (generalized) EM algorithm with a closed-form E-step, in contrast with other recent approaches to semi-supervised probabilistic clustering which require Gibbs sampling or suboptimal shortcuts. We show that our approach is ideally suited for image segmentation: it avoids the combinatorial nature Markov random field priors, and opens the door to more sophisticated spatial priors (e.g., wavelet-based) in a simple and computationally efficient way. Finally, we extend our formulation to work in unsupervised, semi-supervised, or discriminative modes.