Big DATA and Hadoop

International Journal of Advance Research and Innovative Ideas in Education, 2020

This paper aims to give complete information about Big Data in easy to comprehensible language. It begins with giving the brief introduction about what Big Data is and why is it needed in today’s era of technological advancement. Popular tools that are commonly used to work in the world of big data are also explained briefly. Main focus remains on the major players in the market now a days- Apache Hadoop and Apache Spark. Detailed information is given so that reader gets a clear picture about the case where which tool can be used on the basis of reliability as well as performance in real time scenario.

Journal of Energy and Power Engineering, 2016

In recent years, due to the widespread use of electronic services and the use of social network as well, large volumes of information are being made that this information contains various types of things such as videos, photos, texts etc. besides large volume. Due to the high volume and the lack of specificity of this information, covering them through traditional and relational databases is not possible and modern solutions should be used for processing them, so that processing speed is also covered. Data storage for processing and the way of accessing to them in memory, network communication, covering required features for distributed system in solutions that are in use for storing big data, are the items that should be covered. In this paper, a collection of advantages and challenges of big data, special features and characteristics of them has been provided and with the introduction of technologies in use, storage methods are studied and research opportunities to continue the way will be introduced.

Big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate. Challenges include analysis, capture, data curation, search, sharing, storage, transfer, visualization, and information privacy. The term often refers simply to the use of predictive analytics or other certain advanced methods to extract value from data, and seldom to a particular size of data set. Accuracy in big data may lead to more confident decision making. And better decisions can mean greater operational efficiency, cost reductions and reduced risk. Analysis of data sets can find new correlations, to "spot business trends, prevent diseases, combat crime and so on." Scientists, practitioners of media and advertising and governments alike regularly meet difficulties with large data sets in areas including Internet search, finance and business informatics. Scientists encounter limitations in e-Science work, including meteorology, genomics, connectomics, complex physics simulations, and biological and environmental research. Data sets grow in size in part because they are increasingly being gathered by cheap and numerous information sensing mobile devices, aerial (remote sensing), software logs, cameras, microphones, radio-frequency identification (RFID) readers, and wireless sensor networks. The world's technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s; as of 2012, every day 2.5 Exabyte's (2.5×1018) of data were created; The challenge for large enterprises is determining who should own big data initiatives that straddle the entire organization. Work with big data is necessarily uncommon; most analysis is of "PC size" data, on a desktop PC or notebook that can handle the available data set. Relational database management systems and desktop statistics and visualization packages often have difficulty handling big data. The work instead requires "massively parallel software running on tens, hundreds, or even thousands of servers". What is considered "big data" varies depending on the capabilities of the users and their tools, and expanding capabilities make Big Data a moving target. Thus, what is considered to be "Big" in one year will become ordinary in later years. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."

2019

The term Big Data (or BigData) is widely used in scientific, educational, and business literature; however, there does not exist a single definition that can be unreservedly called "canonical". A careless use of Big Data term to promote commercial software further emphasizes the importance of this issue. In this paper, we have performed a review of definitions of Big Data and highlighted the principal features that are attributed to Big Data. We compared all these principal features with features of databases compiled using Edgar F. Codd's publications, and showed that they are not unique and can also be attributed to the databases. Having studied C. Lynch original work, we proposed the definition of Big Data based on the so-called conservation institution. The key point of this definition is a shift from purely technical attitude towards public institutions. Since the current use of the Big Data term may lead to a loss of meaning, there is a need not only to spread out best practices but also to eliminate or minimize the use of dubious or misleading ones.

The term 'Big Data' describes innovative techniques and technologies to capture, store, distribute, manage and analyse petabyte-or larger-sized datasets with high-velocity and different structures. Big data can be structured, unstructured or semi-structured, resulting in incapability of conventional data management methods. Data is generated from various different sources and can arrive in the system at various rates. In order to process these large amounts of data in an inexpensive and efficient way, parallelism is used. Big Data is a data whose scale, diversity, and complexity require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it. Hadoop is the core platform for structuring Big Data, and solves the problem of making it useful for analytics purposes. Hadoop is an open source software project that enables the distributed processing of large data sets across clusters of commodity servers. It is designed to scale up from a single server to thousands of machines, with a very high degree of fault tolerance.

Technology is evolutionary field, what seems new today became old by time. Now a day's concept of big data is in news, on the page of news paper, and it is a topic of research and enthusiasm in world of data. The term Big Data is new but technologies incorporate into it are old like high-speed networks, high-performance computing, task management, thread management, and data mining. People always have attraction and enthusiasm whenever new technologies come in market. If today's organizations do not adopt new technologies then they will be left far behind in their market position. But it would not be wise if we are blindly adopting new technologies without knowing its concept and values. The term Big Data is introduced in data world to process, mange and support massive amount of data. Many organizations are using big data to handle their large amount of data chunks and to gain some meaningful result set from it.Big Data is not just about lots of data, it is actually a concept providing an opportunity to find new insight into your existing data as well guidelines to capture and analysis your future data. It makes any business more agile and robust so it can adapt and overcome business challenges. Hadoop is the core platform for structuring Big Data, and solves the problem of formatting it for subsequent analytics purposes. Hadoop uses a distributed computing architecture consisting of multiple servers using commodity hardware, making it relatively inexpensive to scale and support extremely large data stores.

International Journal of Advance Research and Innovative Ideas in Education, 2019

Big data is a very large volume of structured and unstructured data. An example of big data is petabytes i.e.1,024 terabytes or Exabyte. 1,024 petabytes of data consist of billions to trillions of data of millions of people. Its a process of collecting and processing the huge amount of data. Big companies are using mostly big data for specific surveys. Hadoop is a platform provided that is used for big data. Hadoop stores massive amount of data that have massive power and can process multiple things at a single time. Hadoop Files are called as HDFS. Big data is data that contains large amount in increasing volumes and higher velocity. Hadoop is used to store large amounts of data. Big data consist of very larger, and more complex data sets. These data sets are so large so difficult manage but it is used to solve business problems

This paper is an effort to present the basic importance of Big Data and also its importance in an organization from its performance point of view. The term Big data, refers the data sets, whose volume, complexity and also rate of growth make them more difficult to capture, manage, process and also analyzed. For such type of data –intensive applications, the Apache Hadoop Framework has newly concerned a lot of attention. Hadoop is the core platform for structuring Big data, and solves the problem of making it helpful for analytics idea. Hadoop is an open source software project that enables the distributed processing of enormous data and framework for the analysis and transformation of very large data sets using the MapReduce paradigm. This paper deals with the architecture of Hadoop with its various components.

Big data is the term for any collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications. The challenges include analysis, capture, curation, search, sharing, storage, transfer, visualization, and privacy violations. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, prevent diseases, combat crime and so on." Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers". Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time. Big data "size" is a constantly moving target, as of its ranging from a few dozen terabytes to many petabytes of data.

Research Advances in the Integration of Big Data and Smart Computing

Today is the Computer Era, where the data is increasing exponentially. Managing such a huge data is a challenging job. Under the explosive increase of global data, the term of big data is mainly used to describe enormous datasets. The state-of-the-art of big data is discussed here. The discussions aim to provide a comprehensive overview and big-picture to readers of this existing research area. This chapter discusses the different models and technologies for Big Data; It also introduces Big data Storage. Big data has been a potential topic in various research fields and areas like healthcare, public sector, retail, manufacturing personal data, etc.