Tree Index: A New Cluster Evaluation Technique

Intelligent Data Analysis, 2009

Clustering quality or validation indices allow the evaluation of the quality of clustering in order to support the selection of a specific partition or clustering structure in its natural unsupervised environment, where the real solution is unknown or not available. In this paper, we investigate the use of quality indices mostly based on the concepts of clusters' compactness and separation, for the evaluation of clustering results (partitions in particular). This work intends to offer a general perspective regarding the appropriate use of quality indices for the purpose of clustering evaluation. After presenting some commonly used indices, as well as indices recently proposed in the literature, key issues regarding the practical use of quality indices are addressed. A general methodological approach is presented which considers the identification of appropriate indices thresholds. This general approach is compared with the simple use of quality indices for evaluating a clustering solution.

Background: High-dimensional biomedical data are frequently clustered to identify subgroup structures pointing at distinct disease subtypes. It is crucial that the used cluster algorithm works correctly. However, by imposing a predefined shape on the clusters, classical algorithms occasionally suggest a cluster structure in homogenously distributed data or assign data points to incorrect clusters. We analyzed whether this can be avoided by using emergent self-organizing feature maps (ESOM). Methods: Data sets with different degrees of complexity were submitted to ESOM analysis with large numbers of neurons, using an interactive R-based bioinformatics tool. On top of the trained ESOM the distance structure in the high dimensional feature space was visualized in the form of a so-called U-matrix. Clustering results were compared with those provided by classical common cluster algorithms including single linkage, Ward and k-means. Results: Ward clustering imposed cluster structures on cluster-less ''golf ball " , ''cuboid " and ''S-shaped " data sets that contained no structure at all (random data). Ward clustering also imposed structures on permuted real world data sets. By contrast, the ESOM/U-matrix approach correctly found that these data contain no cluster structure. However, ESOM/U-matrix was correct in identifying clusters in biomedical data truly containing subgroups. It was always correct in cluster structure identification in further canon-ical artificial data. Using intentionally simple data sets, it is shown that popular clustering algorithms typically used for biomedical data sets may fail to cluster data correctly, suggesting that they are also likely to perform erroneously on high dimensional biomedical data. Conclusions: The present analyses emphasized that generally established classical hierarchical clustering algorithms carry a considerable tendency to produce erroneous results. By contrast, unsupervised machine-learned analysis of cluster structures, applied using the ESOM/U-matrix method, is a viable, unbiased method to identify true clusters in the high-dimensional space of complex data.

2016 International Joint Conference on Neural Networks (IJCNN), 2016

This paper deals with a major challenge in clustering that is optimal model selection. It presents new efficient clustering quality indexes relying on feature maximization, which is an alternative measure to usual distributional measures relying on entropy, Chi-square metric or vector-based measures such as Euclidean distance or correlation distance. First Experiments compare the behavior of these new indexes with usual cluster quality indexes based on Euclidean distance on different kinds of test datasets for which ground truth is available. This comparison clearly highlights altogether the superior accuracy and stability of the new method on these datasets, its efficiency from low to high dimensional range and its tolerance to noise. Further experiments are then conducted on "real life" textual data extracted from a multisource bibliographic database for which ground truth is unknown. These experiments show that the accuracy and stability of these new indexes allow to deal efficiently with diachronic analysis, when other indexes do not fit the requirements for this task.

2007

Mutual information has been used in many clustering algorithms for measuring general dependencies between random data variables, but its difficulties in computing for small size datasets has limited its efficiency for clustering in many applications. A novel clustering method is proposed which estimates mutual information based on information potential computed pair-wise between data points and without any prior assumptions about cluster density function. The proposed algorithm increases the mutual information in each step in an agglomerative hierarchy scheme. We have shown experimentally that maximizing mutual information between data points and their class labels will lead to an efficient clustering. Experiments done on a variety of artificial and real datasets show the superiority of this algorithm, besides its low computational complexity, in comparison to other information based clustering methods and also some ordinary clustering algorithms.

IAEME PUBLICATION, 2014

Clustering is a method of unsupervised learning, and a common technique for statistical data analysis used in many fields, including machine learning, data mining, pattern recognition, image analysis and bioinformatics. Hierarchical clustering cannot represent distinct clusters with similar expression patterns. In order to address the problem that the time complexity of the existing hierarchical K-means algorithms is high and most of algorithms are sensitive to noise, a hierarchical K-means clustering algorithm based on silhouette and entropy (HKSE) is put forward. The optimal number of clusters is determined through computing the average Improved Silhouette of the dataset, such that the time complexity can be reduced. we develop an algorithm for calculating the overlap rate. Then we show how the theory can be used to deal with the problem of cluster merging in a hierarchical approach to clustering and give an optimal number of clusters automatically. Finally, experimental results demonstrate the effectiveness of the overlap rate measuring method and the new hierarchical clustering algorithm.

2019

Clustering analysis is one of the most commonly used techniques for uncovering patterns in data mining. Most clustering methods require establishing the number of clusters beforehand. However, due to the size of the data currently used, predicting that value is at a high computational cost task in most cases. In this article, we present a clustering technique that avoids this requirement, using hierarchical clustering. There are many examples of this procedure in the literature, most of them focusing on the dissociative or descending subtype, while in this article we cover the agglomerative or ascending subtype. Being more expensive in computational and temporal cost, it nevertheless allows us to obtain very valuable information, regarding elements membership to clusters and their groupings, that is to say, their dendrogram. Finally, several sets of data have been used, varying their dimensionality. For each of them, we provide the calculations of internal validation indexes to test...

2010

Determining the number of clusters present in a data set automatically is a very important problem. Conventional clustering techniques assume a certain number of clusters, and then try to find out the possible cluster structure associated to the above number. For very large and complex data sets it is not easy to guess this number of clusters. There exists validity based clustering techniques, which measure a certain cluster validity measure of a certain clustering result by varying the number of clusters. After doing this for a broad range of possible number of clusters, this method selects the number for which the validity measure is optimum. This method is, however, awkward and may not always be applicable for very large data sets. Recently an interesting visual technique for determining clustering tendency has been developed. This new technique is called VAT in abbreviation. The original VAT and its different versions are found to determine the number of clusters, before actually applying any clustering algorithm, very satisfactorily. In this paper, we have proposed an out-of-core VAT algorithm (o-VAT) for very large data sets.

2014

Number of variables or attributes of any data set effect to a large extent clustering of that particular data. These attributes directly affect the dissimilarity or distance measures thereby effecting accuracy of data. So dimensionality reduction techniques can definitely improve clustering. As clustering is a unsupervised machine learning technique, the validation of results obtained from application of clustering algorithm to a particular data set is a big issue. This paper formulates a new model for data clustering using combination of feature extraction, data clustering algorithm and clustering validity index/indices. The data clustering algorithm used is Agglomerative Hierarchal Clustering Algorithm. The different features reduction techniques used are PCA, CMDS, ISOMAP and HLLE. The clustering validity indices used are Silhouette index, Dunn index, Davies Bouldin Index and Calinski Harbasaz index. Keywords—Agglomerative Hierarchal Clustering, PCA, CMDS, ISOMAP, HLLE, Silhouett...

IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2011

Evaluation of how well the extracted clusters fit the 4 true partitions of a data set is one of the fundamental chal-5 lenges in unsupervised clustering because the data structure and 6 the number of clusters are unknown a priori. Cluster validity 7 indices are commonly used to select the best partitioning from 8 different clustering results; however, they are often inadequate 9 unless clusters are well separated or have parametrical shapes. 10 Prototype-based clustering (finding of clusters by grouping the 11 prototypes obtained by vector quantization of the data), which 12 is becoming increasingly important for its effectiveness in the 13 analysis of large high-dimensional data sets, adds another dimen-14 sion to this challenge. For validity assessment of prototype-based 15 clusterings, previously proposed indexes-mostly devised for the 16 evaluation of point-based clusterings-usually perform poorly. 17 The poor performance is made worse when the validity indexes 18 are applied to large data sets with complicated cluster structure. 19 In this paper, we propose a new index, Conn_Index, which can 20 be applied to data sets with a wide variety of clusters of different 21 shapes, sizes, densities, or overlaps. We construct Conn_Index 22 based on inter-and intra-cluster connectivities of prototypes. 23 Connectivities are defined through a "connectivity matrix", which 24 is a weighted Delaunay graph where the weights indicate the local 25 data distribution. Experiments on synthetic and real data indicate 26 that Conn_Index outperforms existing validity indices, used in 27 this paper, for the evaluation of prototype-based clustering results. 28 Index Terms-Cluster validity index, complex data structure, 29 connectivity, Conn_Index, prototype-based clustering. 30 I. INTRODUCTION 31 U NSUPERVISED clustering aims to extract the natural 32 partitions in a data set without a priori class information. 33 It groups the data samples into subsets so that samples within a 34 subset are more similar to each other than to samples in other 35 subsets. Any given clustering method can produce a different 36 partitioning depending on its parameters and criteria. This leads 37 to one of the main challenges in clustering-to determine, 38 without auxiliary information, how well the obtained clusters fit 39 the natural partitions of the data set. The common approach for 40 this evaluation is to use validity indices. A meaningful validity

Entropy, 2011

Hierarchical clustering has been extensively used in practice, where clusters can be assigned and analyzed simultaneously, especially when estimating the number of clusters is challenging. However, due to the conventional proximity measures recruited in these algorithms, they are only capable of detecting mass-shape clusters and encounter problems in identifying complex data structures. Here, we introduce two bottom-up hierarchical approaches that exploit an information theoretic proximity measure to explore the nonlinear boundaries between clusters and extract data structures further than the second order statistics. Experimental results on both artificial and real datasets demonstrate the superiority of the proposed algorithm compared to conventional and information theoretic clustering algorithms reported in the literature, especially in detecting the true number of clusters.