Bootstrap technique in cluster analysis
Anil Jain1987
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
AImtract--We define a method to estimate the number of clusters in a data set E, using the bootstrap technique. This approach involves the generation of several "fake" data sets by sampling patterns with replacement in E (bootstrapping). For each number, K, of clusters, a measure of stability of the K-cluster partitions over the bootstrap samples is used to characterize the significance of the K-cluster partition for the original data set. The value of K which provides the most stable partitions is the estimate of the number ot clusters m 6. I ne perlormance ot tam new techmque Is demonstrated on both synthetic and real data, and is applied to the segmentation of range images.
Related papers
Evaluation of structure and reproducibility of cluster solutions using the bootstrap
Marketing Letters, 2010
Segmentation results derived using cluster analysis depend on (1) the structure of the data and (2) algorithm parameters. Typically neither the data structure is assessed in advance of clustering nor is the sensitivity of the analysis to changes in algorithm parameters. We propose a benchmarking framework based on bootstrapping techniques that accounts for sample and algorithm randomness. This provides much needed guidance both to data analysts and users of clustering solutions regarding the choice of the final clusters from computations which are exploratory in nature.
On the Added Value of Bootstrap Analysis for K-Means Clustering
Journal of Classification, 2015
Because of its deterministic nature, K-means does not yield confidence information about centroids and estimated cluster memberships, although this could be useful for inferential purposes. In this paper we propose to arrive at such information by means of a non-parametric bootstrap procedure, the performance of which is tested in an extensive simulation study. Results show that the coverage of hyper-ellipsoid bootstrap confidence regions for the centroids is in general close to the nominal coverage probability. For the cluster memberships, we found that probabilistic membership information derived from the bootstrap analysis can be used to improve the cluster assignment of individual objects, albeit only in the case of a very large number of clusters. However, in the case of smaller numbers of clusters, the probabilistic membership information still appeared to be useful as it indicates for which objects the cluster assignment resulting from the analysis of the original data is likely to be correct; hence, this information can be used to construct a partial clustering in which the latter objects only are assigned to clusters.
MCS: a method for finding the number of clusters
This paper proposes a maximum clustering similarity (MCS) method for determining the number of clusters in a data set by studying the behavior of similarity indices comparing two (of several) clustering methods. The similarity between the two clusterings is calculated at the same number of clusters, using the indices of Rand (R), Fowlkes and Mallows (FM), and Kulczynski (K) each corrected for chance agreement. The number of clusters at which the index attains its maximum is a candidate for the optimal number of clusters. The proposed method is applied to simulated bivariate normal data, and further extended for use in circular data. Its performance is compared to the criteria discussed in Tibshirani, Walther, and Hastie (2001). The proposed method is not based on any distributional or data assumption which makes it widely applicable to any type of data that can be clustered using at least two clustering algorithms.
An examination of procedures for determining the number of clusters in a data set
Psychometrika, 1985
A Monte Carlo evaluation of 30 procedures for determining the number of clusters was conducted on artificial data sets which contained either 2, 3, 4, or 5 distinct nonoverlapping clusters. To provide a variety of clustering solutions, the data sets were analyzed by four hierarchical clustering methods. External criterion measures indicated excellent recovery of the true cluster structure by the methods at the correct hierarchy level. Thus, the clustering present in the data was quite strong. The simulation results for the stopping rules revealed a wide range in their ability to determine the correct number of clusters in the data. Several procedures worked fairly well, whereas others performed rather poorly. Thus, the latter group of rules would appear to have little validity, particularly for data sets containing distinct clusters. Applied researchers are urged to select one or more of the better criteria. However, users are cautioned that the performance of some of the criteria may be data dependent.
A Novel Approach for Determination of Clusters from Unlabeled Data Sets
2012
We introduce the Novel approach for etermination of clusters from unlabeled data sets. we nvestigate a new method called Extended Support vector achine(ESVM)along with existing Dark Block xtraction (DBE) which is based on an existing algorithm or Visual Assessment of Cluster Tendency (VAT) of a ata set, using several common image and signal rocessing techniques. Its basic steps include )Generating a VAT image of an input dissimilarity atrix, 2)Performing image segmentation on the VAT mage to obtaina binary image, followed by directional orphologicalfiltering, 3)Applying a distance transform o the filtered binary image and projecting the pixel alues onto the maindiagonal axis of the image to form a rojection signal, 4)Smoothing the projection signal, omputing its First-order derivative and then detecting ajor peaks and valleys in the resulting signal to decide he number of clusters, and 5)TheK-Means algorithm is pplied to the major peaks. We alsoimplement the luster Count Extraction ...
Enhanced Visual Analysis for Cluster Tendency Assessment and Data Partitioning
IEEE Transactions on Knowledge and Data Engineering, 2010
Visual methods have been widely studied and used in data cluster analysis. Given a pairwise dissimilarity matrix D D of a set of n objects, visual methods such as the VAT algorithm generally represent D D as an n  n image IðD DÞ where the objects are reordered to reveal hidden cluster structure as dark blocks along the diagonal of the image. A major limitation of such methods is their inability to highlight cluster structure when D D contains highly complex clusters. This paper addresses this limitation by proposing a Spectral VAT algorithm, where D D is mapped to D D 0 in a graph embedding space and then reordered toD D 0 using the VAT algorithm. A strategy for automatic determination of the number of clusters in IðD D 0 Þ is then proposed, as well as a visual method for cluster formation from IðD D 0 Þ based on the difference between diagonal blocks and off-diagonal blocks. A sampling-based extended scheme is also proposed to enable visual cluster analysis for large data sets. Extensive experimental results on several synthetic and real-world data sets validate our algorithms. Index Terms-Clustering, VAT, cluster tendency, spectral embedding, out-of-sample extension. Ç 1 INTRODUCTION A general question in the data mining community is how to organize observed data into meaningful structures (or taxonomies). As a tool of exploratory data analysis [36], cluster analysis aims at grouping objects of a similar kind into their respective categories. Given a data set O comprising n objects fo 1 ; o 2 ;. .. ; o n g (e.g., fish, flowers, beers, etc.), (crisp) clustering partitions the data into c groups C 1 ; C 2 ;. .. ; C c , so that C i \ C j ¼ (; if i 6 ¼ j and C 1 [ C 2 [ Á Á Á [ C c ¼ O. There have been a large number of data clustering algorithms in the recent literature [24]. In general, clustering of unlabeled data poses three major problems: 1) assessing cluster tendency, i.e., how many clusters to seek or what is the value of c?, 2) partitioning the data into c groups, and 3) validating the c clusters discovered. Given "only" a pairwise dissimilarity matrix D D 2 R nÂn representing a data set of n objects (i.e., the original object data is not necessarily available), this paper addresses the first two problems, i.e., determining the number of clusters c prior to clustering and partitioning the data into c clusters. Most clustering algorithms require the number of clusters c as an input parameter, so the quality of the resulting clusters is largely dependent on the estimation of
Cluster analysis: a further approach based on density estimation
Computational Statistics & Data Analysis, 2001
A cluster methodology, motivated via density estimation, is proposed. It is based on the idea of estimating the population clusters, which, following , are deÿned as the connected parts of the "substantial" support of the underlying density. The empirical clusters are deÿned by analogy in terms of the substantial support of a convolution (kernel-type) density estimator. The sample observations are grouped into data clusters, according to the empirical cluster they belong. An algorithm to implement the method, based on resampling ideas, is proposed. It allows either to automatically choose the number of clusters or to give this number as an input. Some theoretical and practical aspects are brie y discussed and a simulation study is given. The results show a good performance of our method, in terms of e ciency and robustness, when compared with two classical cluster algorithms: k-means and single linkage. Finally, a real-data example is discussed.
A Validity Index for Prototype-Based Clustering of Data Sets With Complex Cluster Structures
IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2011
Evaluation of how well the extracted clusters fit the 4 true partitions of a data set is one of the fundamental chal-5 lenges in unsupervised clustering because the data structure and 6 the number of clusters are unknown a priori. Cluster validity 7 indices are commonly used to select the best partitioning from 8 different clustering results; however, they are often inadequate 9 unless clusters are well separated or have parametrical shapes. 10 Prototype-based clustering (finding of clusters by grouping the 11 prototypes obtained by vector quantization of the data), which 12 is becoming increasingly important for its effectiveness in the 13 analysis of large high-dimensional data sets, adds another dimen-14 sion to this challenge. For validity assessment of prototype-based 15 clusterings, previously proposed indexes-mostly devised for the 16 evaluation of point-based clusterings-usually perform poorly. 17 The poor performance is made worse when the validity indexes 18 are applied to large data sets with complicated cluster structure. 19 In this paper, we propose a new index, Conn_Index, which can 20 be applied to data sets with a wide variety of clusters of different 21 shapes, sizes, densities, or overlaps. We construct Conn_Index 22 based on inter-and intra-cluster connectivities of prototypes. 23 Connectivities are defined through a "connectivity matrix", which 24 is a weighted Delaunay graph where the weights indicate the local 25 data distribution. Experiments on synthetic and real data indicate 26 that Conn_Index outperforms existing validity indices, used in 27 this paper, for the evaluation of prototype-based clustering results. 28 Index Terms-Cluster validity index, complex data structure, 29 connectivity, Conn_Index, prototype-based clustering. 30 I. INTRODUCTION 31 U NSUPERVISED clustering aims to extract the natural 32 partitions in a data set without a priori class information. 33 It groups the data samples into subsets so that samples within a 34 subset are more similar to each other than to samples in other 35 subsets. Any given clustering method can produce a different 36 partitioning depending on its parameters and criteria. This leads 37 to one of the main challenges in clustering-to determine, 38 without auxiliary information, how well the obtained clusters fit 39 the natural partitions of the data set. The common approach for 40 this evaluation is to use validity indices. A meaningful validity
Stability of Mixed-Type Cluster Partitions for Determination of the Number of Clusters
Studies in classification, data analysis, and knowledge organization, 2022
For partitioning clustering methods, the number of clusters has to be determined in advance. One approach to deal with this issue are stability indices. In this paper several stability-based validation methods are investigated with regard to the-prototypes algorithm for mixed-type data. The stability-based approaches are compared to common validation indices in a comprehensive simulation study in order to analyze preferability as a function of the underlying data generating process.
Parametric and non-parametric unsupervised cluster analysis
1997
Much work has been published on methods for assessing the probable number of clusters or structures within unknown data sets. This paper aims to look in more detail at two methods, a broad parametric method, based around the assumption of Gaussian clusters and the other a non-parametric method which utilises methods of scale-space filtering to extract robust structures within a data set.
Related topics
Related papers
Estimation of the number of clusters and influence zones
2001
Whereas estimating the number of clusters is directly involved in the ®rst steps of unsupervised classi®cation procedures, the problem still remains topical. In our attempt to propose a solution, we focalize on procedures that do not make any assumptions on the cluster shapes. Indeed the classi®cation approach we use is based on the estimation of the probability density function (PDF) using the Parzen±Rosenblatt method. The modes of the PDF lead to the construction of in¯uence zones which are intrinsically related to the number of clusters. In this paper, using dierent sizes of kernel and dierent samplings of the data set, we study the eects they imply on the relation between in¯uence zones and the number of clusters. This ends up in a proposal of a method for counting the clusters. It is illustrated in simulated conditions and then applied on experimental results chosen from the ®eld of multi-component image segmentation. Ó (M. Herbin), [email protected] (N. Bonnet). 0167-8655/01/$ -see front matter Ó 2001 Published by Elsevier Science B.V. PII: S 0 1 6 7 -8 6 5 5 ( 0 1 ) 0 0 1 0 3 -9
Resampling method for unsupervised estimation of cluster validity
Neural computation, 2001
We introduce a method for validation of results obtained by clustering analysis of data. The method is based on resampling the available data. A figure of merit that measures the stability of clustering solutions against resampling is introduced. Clusters which are stable against resampling give rise to local maxima of this figure of merit. This is presented first for a one-dimensional data set, for which an analytic approximation for the figure of merit is derived and compared with numerical measurements. Next, the applicability of the method is demonstrated for higher dimensional data, including gene microarray expression data.
Bootstrap approach for clustering method with applications
International Journal of ADVANCED AND APPLIED SCIENCES
Discovering patterns of big data is an important step to actionable insights data. The clustering method is used to identify the data pattern by splitting the data set into clusters with associated variables. Various research works proposed a bootstrap method for clustering the array data but there is a weak view of statistical or theoretical results and measures of the model consistency or stability. The purpose of this paper is to assess model stability and cluster consistency of the K-number of clusters by using bootstrap sampling patterns with replacement. In addition, we present a reasonable number of clusters via bootstrap methods and study the significance of the K-number of clusters for the original data set by looking at the value of the K-number that provides the most stable clusters. Practically, bootstrap is used to measure the accuracy of estimation and analyze the stability of the outcomes of cluster methods. We discuss the performance of suggestion clusters through ru...
The Effect of Cluster Size, Dimensionality, and the Number of Clusters on Recovery of True Cluster Structure
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
An evaluation of four clustering methods and four extemal criterion measures was conducted with respect to the effect of the number of clusters, dimensionality, and relative cluster sizes on the recovery of true cluster structure. The four methods were the single link, complete link, group average (UPGMA), and Ward's minimum variance algorithms. The results indicated that the four criterion measures were generally consistent with each other, of which two highly similar pairs were identified. The tirst pair consisted of the Rand and corrected Rand statistics, and the second pair was the Jaccard and the Fowlkes and Mallows indexes. With respect to the methods, recovery was found to improve as the number of clusters increased and as the number of dimensions increased. The relative cluster size factor produced differential perfonnance effects, with Ward's procedure providing the best recovery when the clusters were of equal size. The group average method gave equivalent or better recovery when the clusters were of unequal size.
a new method for the characterization of clusters
appliedmathgroup.org
This new method for characterizing clusters is based on the simulation of a diffusionlike process. A resolution-parameter-R-is introduced such that when assigned successive values from an increasing sequence, it is possible to detect the following:
DP-Dip: A skinny method for estimating the number and center of clusters
Procedia Computer Science, 2018
Multi-scales data containing structures at different scales of shape and density is very common in both synthetic and real world. However, it is a big problem to cluster this kind of data accurately. Choosing an appropriate clustering number is the first step, important and not easy. In this paper, we propose a skinny method, DP-Dip, to estimate the number of clusters. Different from many popular methods, DP-Dip does not make any assumptions about data distribution and only admit one assumption: each cluster is a unimodal distribution. Besides, the method never perform complicated formulas and calculations but employ recursion until the final numbers are determined. Specifically, this algorithm firstly finds the density peaks to input the initial numbers, then splits some clusters according to the modality-testing result, finally merge some clusters if they should be combined.
Estimating the number of data clusters via the contrast statistic
Journal of Biomedical Science and Engineering, 2012
A new method (the Contrast statistic) for estimating the number of clusters in a set of data is proposed. The technique uses the output of self-organising map clustering algorithm, comparing the change in dependency of "Contrast" value upon clusters number to that expected under a uniform distribution. A simulation study shows that the Contrast statistic can be used successfully either, when variables describing the object in a multi-dimensional space are independent (ideal objects) or dependent (real biological objects).
1 An Efficient Formulation of the Improved Visual Assessment of Cluster Tendency (iVAT) Algorithm
The VAT algorithm is a visual method for determining the possible number of clusters in, or the cluster tendency of, a set of objects. The improved VAT (iVAT) algorithm uses a graph-theoretic distance transform to improve the effectiveness of the VAT algorithm for "tough" cases where VAT fails to accurately show the cluster tendency. In this paper we present an efficient formulation of the iVAT algorithm which reduces the computational complexity of the iVAT algorithm from O(N 3 ) to O(N 2 ). We also prove a direct relationship between the VAT image and the iVAT image produced by our efficient formulation. We conclude with three examples displaying clustering tendencies in three of the Karypis data sets that illustrate the improvement offered by the iVAT transformation. We also provide a comparison of iVAT images to those produced by the Reverse Cuthill-Mckee (RCM) algorithm; our examples suggest that iVAT is superior to the RCM method of display.
A new algorithm for hybrid hierarchical clustering with visualization and the bootstrap
Journal of Statistical Planning and Inference, 2003
We propose a hybrid clustering method, hierarchical ordered partitioning and collapsing hybrid (HOPACH), which is a hierarchical tree of clusters. The methodology combines the strengths of both partitioning and agglomerative clustering methods. At each node, a cluster is partitioned into two or more smaller clusters with an enforced ordering of the clusters. Collapsing steps uniting the two closest clusters into one cluster can be used to correct for errors made in the partitioning steps. We implement a version of HOPACH which optimizes a measure of clustering strength, such as average silhouette, at each partitioning and collapsing step. An important beneÿt of a hierarchical tree is that one can look at clusters at increasing levels of detail. We propose to visualize the clusters at any level of the tree by plotting the distance matrix corresponding with an ordering of the clusters and an ordering of elements within the clusters. A ÿnal ordered list of elements is obtained by running down the tree completely. The bootstrap can be used to establish the reproducibility of the clusters and the overall variability of the followed procedure. The power of the methodology compared to current algorithms is illustrated with simulated and publicly available cancer gene expression data sets.
Statistical Models and Methods for Cluster Analysis and Image Segmentation
1986
INFORMATION & DECISION SCIENCES DEPARTMENT *COLLEGE OF BUSINESS ADMINISTRATION UNIVERSITY OF ILLINOIS AT CHICAGO * BOX 4348. CHICAGO, IL 60680 3/30/86
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.