Efficient Parallel Data Processing in the Cloud

International Journal of Grid and Distributed Computing, 2016

Cloud computing is a technology in which the Cloud Service Providers (CSP) provide many virtual servers to the users to store their information in the cloud. The faults occurring on the assignment and dismission of the virtual machines, the processing cost in the allocation of resources must also be considered. The parallel processing of the information on the virtual machines must be done effectively and in an efficient manner. A variety of systems were developed to facilitate Many Task Computing (MTC). These systems aim to hide the issues of parallelism and fault tolerant and they are used in many applications. In this paper, we introduced Nephele, a data processing framework to exploit dynamic resource provisioning offered by IaaS clouds. The performance evaluation of the virtual machines has been evaluated and the allocation and deallocation of job tasks to the specific virtual machines has also been considered. A performance comparison with the well known data processing framework hadoop has been done. Thus this paper tells about the effective and efficient manner of processing the data by parallel processing and allocating the correct resources for the desired task. It also helps to reduce the cost of resource utilization by exploiting the dynamic resource utilization.

Journal of Physics: Conference Series, 2014

In recent years, large scale computer systems have emerged to meet the demands of high storage, supercomputing, and applications using very large data sets. The emergence of Cloud Computing oers the potentiel for analysis and processing of large data sets. Mapreduce is the most popular programming model which is used to support the development of such applications. It was initially designed by Google for building large datacenters on a large scale, to provide Web search services with rapid response and high availability. In this paper we will test the clustering algorithm K-means Clustering in a Cloud Computing. This algorithm is implemented on MapReduce. It has been chosen for its characteristics that are representative of many iterative data analysis algorithms. Then, we modify the framework CloudSim to simulate the MapReduce execution of K-means Clustering on dierent Cloud Computing, depending on their size and characteristics of target platforms. The experiment show that the implementation of K-means Clustering gives good results especially for large data set and the Cloud infrastructure has an inuence on these results.

Because of rapid development of computer and network technologies, development and usage of parallel, distributed, and cloud computing have been widely used for the problem solution which requires the high performance computing power. Developments on parallel computing and their applications are summarized. Parallel computing and their applications are briefly explained. Research studies on distributed computing have been increased because of developments on network technologies and increased data transfer rate per unit time (network bandwidth). Research studies have been focused on cloud computing by using many computers recent years. In this study, parallel, distributed, and cloud computing are examined for various parameters.

Vol. 18 No. 11 NOVEMBER 2020 International Journal of Computer Science and Information Security (IJCSIS), 2020

There is an increasing demand for processing tremendous volumes of data, which promotes the research on systems and techniques to optimize Big Data processing time. Traditional Systems do not offer the required flexibility, scalability and they are often inefficient in processing complex analytical tasks in finite and allowable time and scalable storage to accommodate the growing data. In this paper, we propose a system that improves Big Data processing by implementing a MapReduce program, two Cloud Services and several optimization techniques. The system is implemented in Cloud Computing using Cloud Big Data Platform and compares several frameworks such as Hadoop, Apache Spark, HBase etc. Cloud Computing offers a complete and efficient infrastructure, powerful, scalable, and especially infinite resources during high seasonal demands. The evaluations turn out that the proposed system has more accurate results and the processing time is improved up to 85 times faster than the Traditional System.

—With the rapid growth of emerging applications like social network analysis, semantic Web analysis and bioin-formatics network analysis, a variety of data to be processed continues to witness a quick increase. Effective management and analysis of large-scale data poses an interesting but critical challenge. Recently, big data has attracted a lot of attention from academia, industry as well as government. This paper introduces several big data processing technics from system and application aspects. First, from the view of cloud data management and big data processing mechanisms, we present the key issues of big data processing, including cloud computing platform, cloud architecture, cloud database and data storage scheme. Following the MapReduce parallel processing framework , we then introduce MapReduce optimization strategies and applications reported in the literature. Finally, we discuss the open issues and challenges, and deeply explore the research directions in the future on big data processing in cloud computing environments.

Today, we " re surrounded by data like oxygen. The exponential growth of data first presented challenges to cutting-edge businesses such as Google, Yahoo, Amazon, Microsoft, Facebook, Twitter etc. Data volumes to be processed by cloud applications are growing much faster than computing power. This growth demands new strategies for processing and analyzing information. Hadoop-MapReduce has become a powerful Computation Model addresses to these problems. Hadoop HDFS became more popular amongst all the Big Data tools as it is open source with flexible scalability, less total cost of ownership & allows data stores of any form without the need to have data types or schemas defined. Hadoop MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes. In this paper I have provided an overview, architecture and components of Hadoop, HCFS (Hadoop Cluster File System) and MapReduce programming model, its various applications and implementations in Cloud Environments.

Concurrency and Computation: Practice and Experience, 2015

Efficiently scheduling MapReduce tasks is considered as one of the major challenges that face MapReduce frameworks. Many algorithms were introduced to tackle this issue. Most of these algorithms are focusing on the data locality property for tasks scheduling. The data locality may cause less physical resources utilization in non-virtualized clusters and more power consumption. Virtualized clusters provide a viable solution to support both data locality and better cluster resources utilization. In this paper, we evaluate the major MapReduce scheduling algorithms such as FIFO, Matchmaking, Delay, and multithreading locality (MTL) on virtualized infrastructure. Two major factors are used to test the evaluated algorithms: the simulation time and the energy consumption. The evaluated schedulers are compared, and the results show the superiority and the preference of the MTL scheduler over the other existing schedulers. Also, we present a comparison study between virtualized and non-virtualized clusters for MapReduce tasks scheduling. Q. ALTHEBYAN ET AL.

In recent years ad hoc parallel data processing has emerged to be one of the killer applications for Infrastructure-as-a-Service (IaaS) clouds. Major Cloud computing companies have started to integrate frameworks for parallel data processing in their product portfolio, making it easy for customers to access these services and to deploy their programs. However, the processing frameworks which are currently used have been designed for static, homogeneous cluster setups and disregard the particular nature of a cloud. Consequently, the allocated compute resources may be inadequate for big parts of the submitted job and unnecessarily increase processing time and cost. In this paper, we discuss the opportunities and challenges for efficient parallel data processing in clouds and present our research project Nephele. Nephele is the first data processing framework to explicitly exploit the dynamic resource allocation offered by today's IaaS clouds for both, task scheduling and execution. Particular tasks of a processing job can be assigned to different types of virtual machines which are automatically instantiated and terminated during the job execution. Based on this new framework, we perform extended evaluations of Map Reduce-inspired processing jobs on an IaaS cloud system and compare the results to the popular data processing framework Hadoop.

Infrastructure-as-a-Service (Iaas) is one of the emerging services provided by the Cloud Companies. Today there are so many frameworks for parallel data processing to provide Infrastructureas-a-Service. Some of these framwork use static, homogeneous resource allocation technique, which can't satisfy dynamic nature of cloud computing. Some of the new frameworks like Nephele overcome these drawbacks by using Dynamic resource allocation but still contain some setbacks like Resource under utilization. Nephele doesn't de allocate the Virtual Machine-(VM) which contains some intermediate results even it completed its execution to provide those results to other Virtual machines. Our new framework introduces Alternative Virtual Machine (AVM) to store intermediate results so wastage of resources can be reduced.