Hadoop Technology

This paper is an effort to present the basic understanding of BIG DATA and HADOOP and its usefulness to an organization from the performance perspective. Along-with the introduction of BIG DATA, the important parameters and attributes that make this emerging concept attractive to organizations has been highlighted. The paper also evaluates the difference in the challenges faced by a small organization as compared to a medium or large scale operation and therefore the differences in their approach and treatment of BIG DATA. As Hadoop is a Substantial scale, open source programming system committed to adaptable, disseminated, information concentrated processing. A number of application examples of implementation of BIG DATA across industries varying in strategy, product and processes have been presented. This paper also deals with the technology aspects of BIG DATA for its implementation in organizations. Since HADOOP has emerged as a popular tool for BIG DATA implementation. Map reduce is a programming structure for effectively composing requisitions which prepare boundless measures of information (multi-terabyte information sets) in- parallel on extensive bunches of merchandise fittings in a dependable, shortcoming tolerant way. A Map reduce skeleton comprises of two parts. They are “mapper" and "reducer" which have been examined in this paper. The paper deals with the overall architecture of HADOOP along with the details of its various components in Big Data.

Big data plays a major role in all aspects of business and IT infrastructure. Today many organizations, Social Media Networking Sites, E-commerce, Educational institution, satellite communication, Aircrafts and others generate huge volume of data on a daily basis. This data is in the form of structured, semi-structured and unstructured. So this huge voluminous amount of data is coined as big data. These big data should be stored and processed in the effective manner. But, in the traditional distributed system this data cannot be effectively handled because of lack of resources. So the term Hadoop comes in to the picture. Hadoop stores and process the huge voluminous amount of data with their strong Hadoop ecosystem. It contains many modules for processing the data, storing the data, allocating the resources, Configuration Management, retrieving the data and for providing highly fault tolerance mechanism. In this paper it focuses on big data concepts, characteristics, real time examples of big data, Hadoop Modules and their pros and cons.

Technology is evolutionary field, what seems new today became old by time. Now a day's concept of big data is in news, on the page of news paper, and it is a topic of research and enthusiasm in world of data. The term Big Data is new but technologies incorporate into it are old like high-speed networks, high-performance computing, task management, thread management, and data mining. People always have attraction and enthusiasm whenever new technologies come in market. If today's organizations do not adopt new technologies then they will be left far behind in their market position. But it would not be wise if we are blindly adopting new technologies without knowing its concept and values. The term Big Data is introduced in data world to process, mange and support massive amount of data. Many organizations are using big data to handle their large amount of data chunks and to gain some meaningful result set from it.Big Data is not just about lots of data, it is actually a concept providing an opportunity to find new insight into your existing data as well guidelines to capture and analysis your future data. It makes any business more agile and robust so it can adapt and overcome business challenges. Hadoop is the core platform for structuring Big Data, and solves the problem of formatting it for subsequent analytics purposes. Hadoop uses a distributed computing architecture consisting of multiple servers using commodity hardware, making it relatively inexpensive to scale and support extremely large data stores.

Today’s era is the era of big data. This paper documents an attempt that gives a consolidated description of big data while indulging its other unique and defining characteristics by considering definitions from practitioners and academics. In this paper, brief introduction of big data and an overview of Hadoop, which is the core platform of big data and used for processing the data, which uses a map reduce paradigm to process the data, is given. Big data is a set of techniques and technologies that require new forms of integration to uncover large hidden values from large datasets that are diverse, complex, and of a massive scale. Big data environment is used to acquire, organize and analyze the various types of data. There is an observation about Map Reduce framework that framework generates large amount of intermediate data. Therefore, as well as the tasks finishes there is need of throwing that abundant data, because MapReduce is unable to utilize them.

Big data is a combination of big and complex data sets that have the vast volume of data, social media analytics, data management efficiency, real-time data. Big data analytics is the procedure of study huge amounts of data. Big Data is represent by the dimensions volume, variety, velocity and veracity .For big data processing there is a method which is called Hadoop which uses the map reduce paradigm to process the data.

Big data is a data or data sets so large or complex that traditional data processing applications are inadequate and distributed databases are needed. Firms like Google, eBay, LinkedIn, and Face book were built around big data from the beginning. It is a collection of massive and complex data sets that include the huge quantities of data, social media analytics, data management capabilities, real time data etc. Challenges include sensor design, capture, data curation, sharing, storage, analysis, visualization, information privacy etc. Big data refers to datasets high in variety and velocity, so that very difficult to handle using traditional tools and techniques. The process of research into massive data to reveal secret correlations named as big data analytics. Big Data is a data whose complexity requires new techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it. We need a different platform named Hadoop as the core platform for structuring Big Data, and solve the problem of making it useful for analytics purposes.

2014 IEEE International Advance Computing Conference (IACC), 2014

Hadoop is an open source cloud computing platform of the Apache Foundation that provides a software programming framework called MapReduce and distributed file system, HDFS. It is a Linux based set of tools that uses commodity hardware, which are relatively inexpensive, to handle, analyze and transform large quantity of data. Hadoop Distributed File System, HDFS, stores huge data set reliably and streams it to user application at high bandwidth and MapReduce is a framework that is used for processing massive data sets in a distributed fashion over a several machines. This paper gives a brief overview of Big Data, Hadoop MapReduce and Hadoop Distributed File System along with its architecture.

Now days, The 21st century is emphasized by a rapid and enormous change in the field of information technology. It is a non-separable part of our daily life and of multiple other industries like education, genetics, entertainment, science & technology, business etc. In this information age, a vast amount of data generation takes place. This vast amount of data is referred as Big Data. There is a number of challenges present in the Big Data such as capturing data, data analysis, searching of data, sharing of data, filtering of data etc. Today Big Data is applied in various fields like shopping websites such as Amazon, Flipkart, Social networking sites such as Twitter, Facebook, and so on. It is reviewed from some literature that, the Big data tends to use different analysis methods, like predictive analysis, user analysis etc. This paper represents the fact that, Big Data required an open source technology for operating and storing huge amount of data. This paper greatly emphasizes on Apache Hadoop, which has become dominant due to its applicability for processing of big data.Hadoop supports thousands of terabytes of data. Hadoop framework facilitates the analysis of big data and its processing methodologies as well as the structure of an ecosystem.

Data are now woven into every sector and function in the global economy, and, like other essential factors of production such as hard assets and human capital, much of modern economic activity simply could not take place without them. The use of Big Data — large pools of data that can be brought together and analyzed to discern patterns and make better decisions — will become the basis of competition and growth for individual firms, enhancing productivity and creating significant value for the world economy by reducing waste and increasing the quality of products and services. Big Data demand cost-effective, fault tolerant, scalable and flexible and innovative forms of information processing for decision making. This paper emphasis on the features, architectures, and functionalities of Big data, Hadoop,Map Reduce, HDFS.

This paper is an effort to present the basic understanding of BIG DATA is and it's usefulness to an organization from the performance perspective. Along-with the introduction of BIG DATA, the important parameters and attributes that make this emerging concept attractive to organizations has been highlighted. The paper also evaluates the difference in the challenges faced by a small organization as compared to a medium or large scale operation and therefore the differences in their approach and treatment of BIG DATA. A number of application examples of implementation of BIG DATA across industries varying in strategy, product and processes have been presented. The second part of the paper deals with the technology aspects of BIG DATA for it's implementation in organizations. Since HADOOP has emerged as a popular tool for BIG DATA implementation, the paper deals with the overall architecture of HADOOP along with the details of it's various components. Further each of the components of the architecture has been taken up and described in detail.

Big Data Analytics (BDA) is the matter of great concern in the research and analysis field of data science today. Each and every day, data in petabytes are processed and analyzed from different sources on the Internet. Big organizations and Internet giants leave no stone to use the hidden information of Big Data. Hence, they make use of one algorithm or another algorithm of data science to do the analytics of Big Data. The most successful algorithm that has been developed so far to do effective Big Data Analytics is MapReduce – an algorithm developed and successfully implemented by Google. The data scientists of data science field have been trying hard to build such an architecture that can implement the philosophy of MapReduce to achieve efficient, effective, and the most economical way of doing BDA. In this effort, Apache Hadoop appears to be the most promising technology build so far. This article is an effort to bring the underlying details of Big Data Analytics done through Hadoop.

2006

With the growing developments and advancements in the fields of computing, it is necessary for institutions and organizations to handle large masses of data at faster speeds. Not only are the sizes of data increasing, so are the varied file types. Due to the inadequacy of traditional file management systems to handle this kind of large data, a need for a more appropriate system arose. This need led to the introduction and development of Big Data Technology. Big Data Technology includes different modules capable of moving beyond exabytes of data. In this paper, we provide a comparison between relational and non-relational database systems, their uses, implementations, advantages and disadvantages. Apart from this, we also provide an in-depth overview of the modules related to Hadoop, a Big Data management framework.

2016

Every eighteen months the volume of existing data in the world doubles in size, making it increasingly becomes difficult to store and query the information that is derived from multiple data sources. This paper introduces Apache Hadoop as a solution to problems involving Big Data. BIG Data is the term used to refer to the great mass of data, from different sources of information, stored in various locations, updated all the time and these are the three V's (Volume, Speed, Variety). Big Data is not a technology, it is a concept where voluminous and complex databases, and can be structured, semi-structured and unstructured communicate, but not always perform operations on the waiting time, making some impossible tasks using traditional storage technologies . The objective of this paper is to present a solution for Big Data storage, distribution and mining data from different sources, with a large volume of information and agile way. For the development of the research tool Apache ...

International Journal of Advance Engineering and Research Development, 2017

These last years, the new technology are producing a large quantities of data i.e Big data. Companies are faced with certain problems of collecting, storing, analyzing and exploiting these large volumes of data in order to create the added value. The complete issue, for organizations and administrations, is not to pass by valuable information drowned in the mass. It is here where the technology named as the "Big Data" intervenes. This technology is based on an analysis of very fine large number of data. It is interesting to note that there are several Organizations who offer distributions ready to use for managing a system Big Data namely Hortonworks , Cloudera, MapR, etc. The different distributions have an approach and a different positioning in relation to the vision of a platform Hadoop. These solutions are the termed as Apache Projects and therefore available in nowadays. Yet, the interest of a complete package lies in the compatibility between these components, the simplicity of installation as well as support. In this article, we shall focus the era of big data by defining these characteristics and its architecture. Then we shall talk about Cloudera Distribution for Hadoop Platform, and finally, we shall conclude by a study on the tools of Hadoop distributions of Big Data provided by Cloudera.