Dey thesis preview

Computing Research Repository, 2010

Very large databases are required to store massive amounts of data that are continuously inserted and queried. Analyzing huge data sets and extracting valuable pattern in many applications are interesting for researchers. We can identify two main groups of techniques for huge data bases mining. One group refers to streaming data and applies mining techniques whereas second group attempts to solve this problem directly with efficient algorithms. Recently many researchers have focused on data stream as an efficient strategy against huge data base mining instead of mining on entire data base. The main problem in data stream mining means evolving data is more difficult to detect in this techniques therefore unsupervised methods should be applied. However, clustering techniques can lead us to discover hidden information. In this survey, we try to clarify: first, the different problem definitions related to data stream clustering in general; second, the specific difficulties encountered in this field of research; third, the varying assumptions, heuristics, and intuitions forming the basis of different approaches; and how several prominent solutions tackle different problems.

Big Data Analytics, 2016

Clustering is a key data mining task. This is the problem of partitioning a set of observations into clusters such that the intra-cluster observations are similar and the inter-cluster observations are dissimilar. The traditional setup where a static dataset is available in its entirety for random access is not applicable as we do not have the entire dataset at the launch of the learning, the data continue to arrive at a rapid rate, we can not access the data randomly, and we can make only one or at most a small number of passes on the data in order to generate the clustering results. These types of data are referred to as data streams. The data stream clustering problem requires a process capable of partitioning observations continuously while taking into account restrictions of memory and time. In the literature of data stream clustering methods, a large number of algorithms use a two-phase scheme which consists of an online component that processes data stream points and produces summary statistics, and an offline component that uses the summary data to generate the clusters. An alternative class is capable of generating the final clusters without the need of an offline phase. This paper presents a comprehensive survey of the data stream clustering methods and an overview of the most well-known streaming platforms which implement clustering.

The clustering problem is a difficult problem for the data stream domain. This is because the large volumes of data arriving in a stream renders most traditional algorithms too inefficient. In recent years, a few one-pass clustering algorithms have been developed for the data stream problem. Although such methods address the scalability issues of the clustering problem, they are generally blind to the evolution of the data and do not address the following issues: (1) The quality of the clusters is poor when the data evolves considerably over time. (2) A data stream clustering algorithm requires much greater functionality in discovering and exploring clusters over different portions of the stream. The widely used practice of viewing data stream clustering algorithms as a class of one-pass clustering algorithms is not very useful from an application point of view. For example, a simple one-pass clustering algorithm over an entire data stream of a few years is dominated by the outdated history of the stream. The exploration of the stream over different time windows can provide the users with a much deeper understanding of the evolving behavior of the clusters. At the same time, it is not possible to simultaneously perform dynamic clustering over all possible time horizons for a data stream of even moderately large volume. This paper discusses a fundamentally different philosophy for data stream clustering which is guided by application-centered requirements. The idea is divide the clustering process into an online component which periodically stores detailed summary statistics and an offline component which uses only this summary statistics. The offline component is utilized by the analyst who can use a wide variety of inputs (such as time horizon or number of clusters) in order to provide a quick understanding of the broad clusters in the data stream. The problems of efficient choice, storage , and use of this statistical data for a fast data stream turns out to be quite tricky. For this purpose, we use the concepts of a pyrami-dal time frame in conjunction with a micro-clustering approach. Our performance experiments over a number of real and synthetic data sets illustrate the effectiveness, efficiency, and insights provided by our approach.

2016 IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS), 2016

In this paper, we propose a new approach to fuzzy data clustering. We present a new algorithm, called TEDA-Cloud, based on the recently introduced TEDA approach to outlier detection. TEDA-Cloud is a statistical method based on the concepts of typicality and eccentricity able to group similar data observations. Instead of the traditional concept of clusters, the data is grouped in the form of granular unities called data clouds, which are structures with no pre-defined shape or set boundaries. TEDA-Cloud is a fully autonomous and self-evolving algorithm that can be used for data clustering of online data streams and applications that require real-time response. Since it is fully autonomous, TEDA-Cloud is able to "start from scratch" (from an empty knowledge basis), create, update and merge data clouds, in a fully autonomous manner, without requiring any user-defined parameters (e.g. number of clusters, size, radius) or previous training. Moreover, TEDA-Cloud, unlike most of the traditional statistical approaches, does not rely on a specific data distribution or on the assumption of independence of data samples. The results, obtained from multiple data sets that are very well known in literature, are very encouraging.

Nowadays the growth of the datasets size causes some difficulties to extract useful information and knowledge especially in specific domains. However , new methods in data mining need to be developed in both sides of supervised and unsupervised approaches. Nevertheless, data stream clustering can be taken into account as an effective strategy to apply for huge data as an unsupervised fashion. In this research we not only propose a framework for data stream clustering but also evaluate different aspects of existing obstacles in this arena. The main problem in data stream clustering is visiting data once therefore new methods should be applied. On the other hand, concept drift must be recognized in real-time. In this paper, we try to clarify: first, the different aspects of problem with regard to data stream clustering generally and how several prominent solutions tackle different problems; second, the varying assumptions, heuristics, and intuitions forming the basis of approaches and finally a new framework for data stream clustering is proposed with regard to the specific difficulties encountered in this field of research.

Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods, 2017

Data stream clustering is becoming an active research area in big data. It refers to group constantly arriving new data records in large chunks to enable dynamic analysis/updating of information patterns conveyed by the existing clusters, the outliers, and the newly arriving data chunk. Prototype-based algorithms for solving the problem have their promises for simplicity and efficiency. However, existing implementations have limitations in relation to quality of clusters, ability to discover outliers, and little consideration of possible new patterns in different chunks. In this paper, a new incremental algorithm called Enhanced Incremental K-Means (EINCKM) is developed. The algorithm is designed to detect new clusters in an incoming data chunk, merge new clusters and existing outliers to the currently existing clusters, and generate modified clusters and outliers ready for the next round. The algorithm applies a heuristic-based method to estimate the number of clusters (K), a radius-based technique to determine and merge overlapped clusters and a variance-based mechanism to discover the outliers. The algorithm was evaluated on synthetic and real-life datasets. The experimental results indicate improved clustering correctness with a comparable time complexity to existing methods dealing with the same kind of problems.

2017

In order to extract truthful knowledge out of the data present in a data warehouse, a wide range of knowledge discovery techniques have been provided that process the data in multiple passes. But nowadays, we are facing a challenge of handling massive data in a proper and timely manner so as to extract useful information (knowledge) from streaming data. Such massive streaming data cannot be stored in our limited storage and due to its continuous flow we need to process it in single pass. Various algorithms have been provided in order to perform the single pass extraction of knowledge from streaming data; however, no single data mining algorithm can be used applicably for all the problems because of the different kinds of real data sets or synthetic data sets. This paper discusses various streaming data mining clustering techniques and compares the algorithms taking into consideration for the future challenges.

In recent years, advances in both hardware and software technology has allowed us to automatically record transactions and other information everyday at a rapid rate. Huge volumes of web, sensory and transactional data are continuously generated everyday as data streams, which need to be analyzed online as they arrive. Analysis of data streams have been researched extensively because of its emerging, imminent, and broad applications. One of the important method is clustering have been widely studied in the data mining community. Many existing data mining methods cannot be applied directly on streaming data because of the fact that the data needs to be mined in single pass. Furthermore, in data stream processing temporal locality is also quite important, because the essential patterns in the data may change and therefore, the clusters in the past history may no longer remain relevant to the future. In this paper we explore various issues and challenges on clustering data streams.

recently plenty of applications generated data stream. Clustering is a challenging issues in data streams domain. This is because the large volume of data arriving in a stream and evolving over time. Some clustering algorithms have been developed for evolving data streams. Besides limited memory, the nature of evolving data stream implies some requirements for clustering. In this paper, we analyze the requirements needed for clustering evolving data streams. We review some of the latest algorithms in the literature and discuss how they meet the requirements.

PloS one, 2024

The clustering analysis approach treats multivariate data tuples as objects and groups them into clusters based on their similarities or dissimilarities within the dataset. However, in modern world, a significant volume of data is continuously generated from diverse sources over time. In these dynamic scenarios, the data is not static but continually evolves. Consequently, the interesting patterns and inherent subgroups within the datasets also change and develop over time. The researchers have paid special attention to monitoring changes in cluster solutions of evolving streams. For this matter, several algorithms have been proposed in the literature. However, to date, no study has examined the effect of variability in cluster sizes on the evolution of cluster solutions. Moreover, no guidance is available on determining the impact of cluster sizes on the type of changes they experience in the streams. In the present simulation study using artificial datasets, the evolution of clusters is examined concerning the variability in cluster sizes. The findings are substantial because tracing and monitoring the changes in clustering solutions have a wide range of applications in every field of research. This study determines the minimum sample size required in the clustering of time-stamped datasets.