DATA STREAM CLUSTERING ISSUES AND CHALLENGES-A SURVEY
IJAR Indexing
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
In recent years, advances in both hardware and software technology has allowed us to automatically record transactions and other information everyday at a rapid rate. Huge volumes of web, sensory and transactional data are continuously generated everyday as data streams, which need to be analyzed online as they arrive. Analysis of data streams have been researched extensively because of its emerging, imminent, and broad applications. One of the important method is clustering have been widely studied in the data mining community. Many existing data mining methods cannot be applied directly on streaming data because of the fact that the data needs to be mined in single pass. Furthermore, in data stream processing temporal locality is also quite important, because the essential patterns in the data may change and therefore, the clusters in the past history may no longer remain relevant to the future. In this paper we explore various issues and challenges on clustering data streams.
Related papers
A Study on Clustering in Data Stream
In recent years, advances in hardware technology have facilitated new ways of collecting data continuously. Tremendous and potentially infinite volumes of data streams are often generated by real time systems, internet traffic, financial market, communication network, remote sensors, and other environments. Analyzing huge data sets and extracting valuable pattern in many applications are interesting for researchers. We identify two techniques for huge data bases mining. One is to streaming data and apply mining techniques whereas second is to solve this problem directly with competent algorithms. The main problem in data stream mining means growing data is more difficult to detect in this techniques therefore unsupervised methods should be applied. However, clustering techniques can indication us to determine hidden information.
A Survey on Clustering Algorithms for Data Streams
International Journal of Computer Applications
Data stream mining is an emerging area for extracting useful information from continuous arriving data. Web click stream, weather monitoring, network traffic, shopping history, web log are some key resources of generating data stream. Clustering is one of the most useful technique for analsing stream data, as it does not require any predefined class labeling. Data stream mining is challanging as the data is massive and arriving continuously. The traditional clustering algorithms cannot be directly applied on the data streams. Data stream mining needs one scan algorithms to extract rich data in the form of data streams. In this paper we discuss various data stream clustering algorithms with their limitations and required data structures. This paper also provides a comparative study of these algorithms. Real world applications of data streams, data resources and publicly available softwares are also discussed.
Data Stream Clustering: Challenges and Issues
Computing Research Repository, 2010
Very large databases are required to store massive amounts of data that are continuously inserted and queried. Analyzing huge data sets and extracting valuable pattern in many applications are interesting for researchers. We can identify two main groups of techniques for huge data bases mining. One group refers to streaming data and applies mining techniques whereas second group attempts to solve this problem directly with efficient algorithms. Recently many researchers have focused on data stream as an efficient strategy against huge data base mining instead of mining on entire data base. The main problem in data stream mining means evolving data is more difficult to detect in this techniques therefore unsupervised methods should be applied. However, clustering techniques can lead us to discover hidden information. In this survey, we try to clarify: first, the different problem definitions related to data stream clustering in general; second, the specific difficulties encountered in this field of research; third, the varying assumptions, heuristics, and intuitions forming the basis of different approaches; and how several prominent solutions tackle different problems.
State-of-the-art on clustering data streams
Big Data Analytics, 2016
Clustering is a key data mining task. This is the problem of partitioning a set of observations into clusters such that the intra-cluster observations are similar and the inter-cluster observations are dissimilar. The traditional setup where a static dataset is available in its entirety for random access is not applicable as we do not have the entire dataset at the launch of the learning, the data continue to arrive at a rapid rate, we can not access the data randomly, and we can make only one or at most a small number of passes on the data in order to generate the clustering results. These types of data are referred to as data streams. The data stream clustering problem requires a process capable of partitioning observations continuously while taking into account restrictions of memory and time. In the literature of data stream clustering methods, a large number of algorithms use a two-phase scheme which consists of an online component that processes data stream points and produces summary statistics, and an offline component that uses the summary data to generate the clusters. An alternative class is capable of generating the final clusters without the need of an offline phase. This paper presents a comprehensive survey of the data stream clustering methods and an overview of the most well-known streaming platforms which implement clustering.
Data Stream Clustering Techniques, Applications, and Models: Comparative Analysis and Discussion
Big Data and Cognitive Computing, 2018
Data growth in today’s world is exponential, many applications generate huge amount of data streams at very high speed such as smart grids, sensor networks, video surveillance, financial systems, medical science data, web click streams, network data, etc. In the case of traditional data mining, the data set is generally static in nature and available many times for processing and analysis. However, data stream mining has to satisfy constraints related to real-time response, bounded and limited memory, single-pass, and concept-drift detection. The main problem is identifying the hidden pattern and knowledge for understanding the context for identifying trends from continuous data streams. In this paper, various data stream methods and algorithms are reviewed and evaluated on standard synthetic data streams and real-life data streams. Density-micro clustering and density-grid-based clustering algorithms are discussed and comparative analysis in terms of various internal and external c...
A Framework for Clustering Evolving Data Streams A Framework for Clustering Evolving Data Streams
The clustering problem is a difficult problem for the data stream domain. This is because the large volumes of data arriving in a stream renders most traditional algorithms too inefficient. In recent years, a few one-pass clustering algorithms have been developed for the data stream problem. Although such methods address the scalability issues of the clustering problem, they are generally blind to the evolution of the data and do not address the following issues: (1) The quality of the clusters is poor when the data evolves considerably over time. (2) A data stream clustering algorithm requires much greater functionality in discovering and exploring clusters over different portions of the stream. The widely used practice of viewing data stream clustering algorithms as a class of one-pass clustering algorithms is not very useful from an application point of view. For example, a simple one-pass clustering algorithm over an entire data stream of a few years is dominated by the outdated history of the stream. The exploration of the stream over different time windows can provide the users with a much deeper understanding of the evolving behavior of the clusters. At the same time, it is not possible to simultaneously perform dynamic clustering over all possible time horizons for a data stream of even moderately large volume. This paper discusses a fundamentally different philosophy for data stream clustering which is guided by application-centered requirements. The idea is divide the clustering process into an online component which periodically stores detailed summary statistics and an offline component which uses only this summary statistics. The offline component is utilized by the analyst who can use a wide variety of inputs (such as time horizon or number of clusters) in order to provide a quick understanding of the broad clusters in the data stream. The problems of efficient choice, storage , and use of this statistical data for a fast data stream turns out to be quite tricky. For this purpose, we use the concepts of a pyrami-dal time frame in conjunction with a micro-clustering approach. Our performance experiments over a number of real and synthetic data sets illustrate the effectiveness, efficiency, and insights provided by our approach.
Stream Data Mining: Techniques and issues
The continuous developments in hardware and software technology in recent years have enabled the capture of different variety of data in different format in a wide range of fields. It includes telecommunication data, transaction data, electric power grid data and other similar dynamic data. Every day, huge volumes of sensor, transactional and web data are rapidly generated as streams, which should be analyzed. A data stream is a sequence of unbounded, real time data items with a very high data rate. Traditional data mining techniques cannot be applied with such huge and fast data. It requires advanced mining methods for the analysis and mining as well as to handle the various issues associated with it.
Mining Data Streams: A Survey
dimensional stream processing and analysis methods. In this paper, we try to discuss the issues of data streams. We present the overview of basic methodologies for stream data processing. We discuss the various data streams mining algorithms.
State-of-the-art in data stream mining
2007
Data streams became ubiquitous as many sources produce data continuously and rapidly. Examples of streaming data include customer click streams, telephone records, web logs, multimedia data, and sets of retail chain transactions. Data streams have brought new challenges to the data mining research community. In consequence, new techniques are needed to process streaming data in reasonable time and space. The goal of this tutorial is to present and discuss the research problems, issues and challenges in learning from data streams. We will present the state-of-the-art techniques in change detection, clustering, classification, frequent patterns, and time series analysis from data streams. Applications of mining data streams in different domains are highlighted. Open issues and future directions will conclude this tutorial. The tutorial also points to data stream mining resources. Specific goals and objectives-Introducing the area of data stream mining-Giving a detailed explanation of the major techniques in the area-Emphasizing the research issues and challenges Expected background of the audience Basic knowledge of data mining concepts and techniques is required.
Related topics
Related papers
Comparison of Clustering Algorithms in Data stream mining: A Literature Survey
2017
In order to extract truthful knowledge out of the data present in a data warehouse, a wide range of knowledge discovery techniques have been provided that process the data in multiple passes. But nowadays, we are facing a challenge of handling massive data in a proper and timely manner so as to extract useful information (knowledge) from streaming data. Such massive streaming data cannot be stored in our limited storage and due to its continuous flow we need to process it in single pass. Various algorithms have been provided in order to perform the single pass extraction of knowledge from streaming data; however, no single data mining algorithm can be used applicably for all the problems because of the different kinds of real data sets or synthetic data sets. This paper discusses various streaming data mining clustering techniques and compares the algorithms taking into consideration for the future challenges.
Data stream clustering: a review
Artificial Intelligence Review, 2020
Number of connected devices is steadily increasing and these devices continuously generate data streams. Real-time processing of data streams is arousing interest despite many challenges. Clustering is one of the most suitable methods for real-time data stream processing, because it can be applied with less prior information about the data and it does not need labeled instances. However, data stream clustering differs from traditional clustering in many aspects and it has several challenging issues. Here, we provide information regarding the concepts and common characteristics of data streams, such as concept drift, data structures for data streams, time window models and outlier detection. We comprehensively review recent data stream clustering algorithms and analyze them in terms of the base clustering technique, computational complexity and clustering accuracy. A comparison of these algorithms is given along with still open problems. We indicate popular data stream repositories and datasets, stream processing tools and platforms. Open problems about data stream clustering are also discussed. Keywords Data streams • Data stream clustering • Real-time clustering • 1 Introduction More devices including sensors are becoming interconnected and interconnected devices continuously generate streams of data at high speed. Offline processing of
A statistical approach for clustering in streaming data
Artificial Intelligence Research, 2013
Recently data stream has been extensively explored due to its emergence in large deal of applications such as sensor networks, web click streams and network flows. Vast majority of researches in the context of data stream mining are devoted to supervise learning, whereas, in real word human practice label of data are rarely available to the learning algorithms. Hence, clustering as the most important unsupervised learning has been in the gravity of focus of quite a lot number of the researchers in data stream community. Clustering paradigms basically place the similar objects together and separate the dissimilar ones into different clusters. In this paper, we propose a Statistical framework for data Stream Clustering, which abbreviated as StatisStreamClust that makes use of two components to find clusters in data stream. The first component especially designed to detect concept change where data underlying distributions change from time to time. Upon detection of concept change by the first component, the second component is triggered to update the whole clustering model. StatisStreamClust brings great benefits to data stream clustering including no sensitivity to the number of clusters and dimensions, reasonable complexity and in the meantime desirable performance, and finally no need to determine window size a priori. To explore the advantages of our approach, quite a lot of experiments with different settings and specifications are conducted. The obtained results are very promising.
Different Aspects of Data Stream Clustering
Nowadays the growth of the datasets size causes some difficulties to extract useful information and knowledge especially in specific domains. However , new methods in data mining need to be developed in both sides of supervised and unsupervised approaches. Nevertheless, data stream clustering can be taken into account as an effective strategy to apply for huge data as an unsupervised fashion. In this research we not only propose a framework for data stream clustering but also evaluate different aspects of existing obstacles in this arena. The main problem in data stream clustering is visiting data once therefore new methods should be applied. On the other hand, concept drift must be recognized in real-time. In this paper, we try to clarify: first, the different aspects of problem with regard to data stream clustering generally and how several prominent solutions tackle different problems; second, the varying assumptions, heuristics, and intuitions forming the basis of approaches and finally a new framework for data stream clustering is proposed with regard to the specific difficulties encountered in this field of research.
Data stream mining techniques: a review
TELKOMNIKA Telecommunication Computing Electronics and Control, 2019
A plethora of infinite data is generated from the Internet and other information sources. Analyzing this massive data in real-time and extracting valuable knowledge using different mining applications platforms have been an area for research and industry as well. However, data stream mining has different challenges making it different from traditional data mining. Recently, many studies have addressed the concerns on massive data mining problems and proposed several techniques that produce impressive results. In this paper, we review real time clustering and classification mining techniques for data stream. We analyze the characteristics of data stream mining and discuss the challenges and research issues of data steam mining. Finally, we present some of the platforms for data stream mining.
Survey Paper on Various Challenges in Data Stream Mining
International journal of engineering research and technology, 2021
In recent years, advances in hardware technology have facilitated new ways of collecting data in a continuous manner. In many applications such as sensor networks, internet traffic the volume of such data is so large that it may be impossible to store the data locally. Eventually, even when the data can be stored, the volume of the incoming data may be so large that it may be impossible to process any particular record more than once. Therefore, many data mining techniques become more challenging. The progress in hardware technology has advanced and hence made it possible for organizations to store and record large streams of transactional data. Such datasets which continuously and rapidly grow over time are referred to as data streams. Data Stream Mining is the process of extracting knowledge from continuous flow of data which comes to the system in a stream. After a lot of research, data mining has become a well established field now, the data stream problem poses a number of challenges which are not easily solved by traditional data mining methods.This paper proposes various challenges in the data stream and also provides different ways to handle them efficiently.
Clustering Algorithms for Data Stream
Nowadays streaming data is delivered by more and more applications, due to this crucial method for data and knowledge engineering is considered to be clustering data streams. It is a two step process. A normal approach is to summarize the data stream in real-time with an online process into so called micro-clusters. Local density estimates are represented by micro-clusters by assembling the information of many data points which is defined in an area. A traditional clustering algorithm is used in a second offline step, in which larger final clusters are formed by reclustering the micro-clusters. For reclustering, the pseudo points which are used are actually coordinator of the micro-clusters with the weights which are density estimates. However, in the online process, information about density in the area between micro-clusters is not preserved and reclustering is based on possibly inaccurate assumptions about the distribution of data within and between micro-clusters (e.g., uniform or Gaussian). In this paper we depicted various algorithms that are used for clustering and their workings in brief.
Requirements for Clustering Evolving Data Stream
recently plenty of applications generated data stream. Clustering is a challenging issues in data streams domain. This is because the large volume of data arriving in a stream and evolving over time. Some clustering algorithms have been developed for evolving data streams. Besides limited memory, the nature of evolving data stream implies some requirements for clustering. In this paper, we analyze the requirements needed for clustering evolving data streams. We review some of the latest algorithms in the literature and discuss how they meet the requirements.
Stream Data Mining: Platforms, Algorithms, Performance Evaluators and Research Trends
International Journal of Database Theory and Application, 2016
Streaming data are potentially infinite sequence of incoming data at very high speed and may evolve over the time. This causes several challenges in mining large scale high speed data streams in real time. Hence, this field has gained a lot of attention of researchers in previous years. This paper discusses various challenges associated with mining such data streams. Several available stream data mining algorithms of classification and clustering are specified along with their key features and significance. Also, the significant performance evaluation measures relevant in streaming data classification and clustering are explained and their comparative significance is discussed. The paper illustrates various streaming data computation platforms that are developed and discusses each of them chronologically along with their major capabilities. This paper clearly specifies the potential research directions open in high speed large scale data stream mining from algorithmic, evolving nature and performance evaluation measurement point of view. Finally, Massive Online Analysis (MOA) framework is used as a use case to show the result of key streaming data classification and clustering algorithms on the sample benchmark dataset and their performances are critically compared and analyzed based on the performance evaluation parameters specific to streaming data mining.
How to automatically perform clustering over data stream
This paper is a review of the different methods of data stream clustering. A quick definition of what is data mining and data stream clustering is made as an introduction. Followed by the review which tries to list a few relevant and popular methods and links them together. The methods are basically listed historically regarding their use in data stream clustering history or their originality.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.