Chapter Two

Overview for Data Science

Learning outcomes
After completing this lesson you should be able to

Differentiate the difference between data and information

Explain data types and their representation

Explain the data value change

Explain basic concepts of big data

An Overview of Data Science
Data science is a multidisciplinary field that uses

scientific methods, processes and algorithm systems to extract

knowledge, Insights from structured, semi-structured and
unstructured data

Data science continues to evolve as one of the most promising and in-demand
career paths for skilled professionals
An Overview of Data Science
It is multidisciplinary
What is data?
A representation of facts, concepts, or instructions in a formalized manner,
which should be suitable for communication, interpretation, or
processing by human or electronic machine

Data can be described as unprocessed facts and figures

It can also defined as groups of non-random symbols in the form of text,

images, and voice representing quantities, action and objects
What is Information?
Organized or classified data, which has some meaningful values for the
receiver

Processed data on which decisions and actions are based. Plain collected
data as raw facts cannot help much in decision-making

Interpreted data created from organized, structured, and processed data in

a particular context
What is Information?
For the decision to be meaningful, the processed data must qualify for the
following characteristics

Timely − Information should be available when required

Accuracy − Information should be accurate

Completeness − Information should be complete

Data Processing Cycle
Data processing is the re-structuring or re-ordering of data by people or
machine to increase their usefulness and add values for a particular
purpose

Data processing consists the following steps

Input

Processing

Output
Data Processing Cycle
Input

The input data is prepared in some convenient form for processing

The form will depend on the processing machine

For example, when electronic computers are used, the input data can be
recorded on any one of the several types of input medium, such as flash
disks, hard disk, and so on
Data Processing Cycle
Processing

In this step, the input data is changed to produce data in a more useful
form

For example, a summary of sales for a month can be calculated from the
sales orders data
Data Processing Cycle
Output

At this stage, the result of the proceeding processing step is collected

The particular form of the output data depends on the use of the data

For example, output data can be total sale in a month

Data types and its representation
In computer science and computer programming, a data type or simply type
is an attribute of data which tells the compiler or interpreter how the
programmer intends to use the data

Common data types include

Integers, Boolean, Characters, Floating-Point Numbers,

Alphanumeric Strings

This data type defines the operations that can be done on the data, the
meaning of the data, and the way values of that type can be stored
Classifications of Data
Data can be classified into the following categories

Structured

Unstructured

Semi-structured

Meta-data
Structured Data
Data that adheres to a predefined data model and is therefore
straightforward to analyze

Conforms to a tabular format with relationship between different rows and

columns

Common examples

Excel files or SQL databases

Structured Data
Structured data depends on the existence of a data model – a model of how
data can be stored, processed and accessed

Because of a data model, each field is discrete and can be accessed

separately or jointly along with data from other fields

it is possible to quickly aggregate data from various locations in the database

Unstructured Data
Data that either does not have a predefined data model or is not
organized in a predefined manner

It is typically text-heavy, but may contain data such as dates, numbers, and
facts as well

Common examples

audio, video files, NoSQL, pictures, pdfs ...

Unstructured Data
The ability to store and process unstructured data has greatly grown in
recent years, with many new technologies and tools coming to the market that
are able to store specialized types of unstructured data
Unstructured Data
The ability to analyze unstructured data is especially relevant in the context
of Big Data, since a large part of data in organizations is unstructured

Think about pictures, videos or PDF documents

The ability to extract value from unstructured data is one of main drivers
behind the quick growth of Big Data
Semi-structured Data
A form of structured data that does not conform with the formal structure
of data models associated with relational databases or other forms of data
tables

But contain tags or other markers to separate semantic elements and

enforce hierarchies of records and fields within the data

Therefore, it is also known as self-describing structure

Semi-structured Data
Examples of semi-structured data

JSON and XML

Semi-structured Data
It is considerably easier to analyze than unstructured data

Many Big Data solutions and tools have the ability to ‘read’ and process either
JSON or XML
Metadata – Data about Data
It provides additional information about a specific set of data

For example

Metadata of a photo could describe when and where the photos were
taken

The metadata then provides fields for dates and locations which, by
themselves, can be considered structured data

Because of this reason, metadata is frequently used by Big Data solutions for
initial analysis.
What Is Big Data?
Large datasets together with the category of computing strategies and
technologies that are used to handle them
Data Value Chain
Describe the information flow within a big data system as a series of steps
needed to generate value and useful insights from data

The Big Data Value Chain identifies the following key high-level activities

Data Acquisition, Data Analysis, Data Curation, Data Storage, Data

Usage,
Data Acquisition
It is the process of gathering, filtering, and cleaning data before it is put in a
data warehouse or any other storage solution on which data analysis can be
carried out

Data acquisition is one of the major big data challenges in terms of

infrastructure requirements
Data Acquisition
The infrastructure required for data acquisition must

deliver low, predictable latency in both capturing data and in executing

queries

be able to handle very high transaction volumes, often in a distributed

environment

support flexible and dynamic data structures

Data Analysis
Involves exploring, transforming, and modelling data with the goal of
highlighting relevant data, synthesising and extracting useful hidden
information with high potential from a business point of view

Related areas include data mining, business intelligence, and machine

learning
Data Curation
Active management of data over its life cycle to ensure it meets the
necessary data quality requirements for its effective usage

Data curation processes can be categorized into different activities

content creation, selection, classification, transformation, validation,

and preservation
Data Curation
Data curators (also known as scientific curators, or data annotators) hold the
responsibility of ensuring that data are trustworthy, discoverable,
accessible, reusable, and fit their purpose

A key trend for the curation of big data utilizes community and crowd
sourcing approaches
Data Storage
It is the persistence and management of data in a scalable way that satisfies
the needs of applications that require fast access to the data

Relational Database Management Systems (RDBMS) have been the main, and
almost unique, solution to the storage paradigm for nearly 40 years
Data Storage
The ACID (Atomicity, Consistency, Isolation, and Durability) properties of the
relational database that guarantee database transactions, lack flexibility
with regard to schema changes and the performance and fault tolerance
when data volumes and complexity grow, making them unsuitable for big
data scenarios

NoSQL technologies have been designed with the scalability goal in mind
and present a wide range of solutions based on alternative data models
Data Usage
Covers the data-driven business activities that need access to data, its
analysis, and the tools needed to integrate the data analysis within the
business activity

In business decision-making , it can enhance competitiveness through

reduction of costs, increased added value, or any other parameter that can
be measured against existing performance criteria
Why Are Big Data Systems Different?
The goal of most big data systems is to surface insights and connections
from large volumes of heterogeneous data that would not be possible using
conventional methods

The “three Vs of big data” characterizes big data

Volume, Velocity, Variety

Volume
Large datasets can be orders of magnitude larger than traditional
datasets, which demands more thought at each stage of the processing and
storage life cycle

Often, because the work requirements exceed the capabilities of a single

computer, this becomes a challenge of pooling, allocating, and coordinating
resources from groups of computers

Cluster management and algorithms capable of breaking tasks into smaller

pieces become increasingly important
Velocity
The speed that information moves through the system

Data is frequently flowing into the system from multiple sources and is
often expected to be processed in real time to gain insights and update the
current understanding of the system

This focus on near instant feedback has driven many big data practitioners
away from a batch-oriented approach and closer to a real-time streaming
system
Velocity
Data is constantly being added, massaged, processed, and analyzed in
order to keep up with the influx of new information and to surface valuable
information early when it is most relevant

Requires robust systems with highly available components to guard against

failures along the data pipeline
Variety
Wide range of data in terms of both the sources being processed and their
relative quality

Data can be ingested from internal systems like application and server
logs, from social media feeds and other external APIs, from physical
device sensors, and from other providers

Big data seeks to handle potentially useful data regardless of where it’s
coming from by consolidating all information into a single system
Variety
The formats and types of media can vary significantly as well

Rich media like images, video files, and audio recordings are ingested
alongside text files, structured logs, etc
Veracity
"Veracity" in the context of data refers to the accuracy, reliability, and
trustworthiness of the data.

The variety of sources and the complexity of the processing can lead to
challenges in evaluating the quality of the data ( biases, noise and
abnormality in data)
Variability
Variability in data refers to the extent to which data points in a dataset deviate
or differ from each other.

Variation in the data leads to wide variation in quality

Additional resources may be needed to identify, process, or filter low quality

data to make it more useful
Value
"Value in data" refers to the usefulness, relevance, and actionable insights that
data can provide when properly analyzed and interpreted.

The ultimate challenge of big data is delivering value

Sometimes, the systems and processes in place are complex enough that
using the data and extracting actual value can become difficult
Clustered Computing
Because of the qualities of big data, individual computers are often
inadequate for handling the data at most stages

To better address the high storage and computational needs of big data,
computer clusters are a better fit

Big data clustering software combines the resources of many smaller

machines, seeking to provide a number of benefits

Resource Pooling, High Availability, Easy Scalability

Clustered Computing
Resource Pooling

Combining the available Storage Space, CPU and Memory

Processing large datasets requires large amounts of all three of these

resources
Clustered Computing
High Availability

Clusters can provide varying levels of fault tolerance and availability

guarantees to prevent hardware or software failures from affecting
access to data and processing

This becomes increasingly important as we continue to emphasize the

importance of real-time analytics
Clustered Computing
Easy Scalability

Clusters make it easy to scale horizontally by adding additional

machines to the group

This means the system can react to changes in resource requirements

without expanding the physical resources on a machine
Clustered Computing
Using clusters requires a solution for managing cluster membership,
coordinating resource sharing, and scheduling actual work on individual
nodes

Cluster membership and resource allocation can be handled by software like

Hadoop’s YARN (which stands for Yet Another Resource Negotiator) or
Apache Mesos
Clustered Computing

https://www.geeksforgeeks.
org/hadoop-ecosystem/
Life Cycle Big Data
Ingesting data into the system

Persisting the data in storage

Computing and Analyzing data

Visualizing the results

Ingesting Data into the System
The process of taking raw data and adding it to the system

The complexity of this operation depends heavily on the format and quality of
the data sources and how far the data is from the desired state prior to
processing

During the ingestion process, some level of analysis, sorting, and labelling
usually takes place

This process is sometimes called ETL, which stands for extract, transform,
and load
Persisting the Data in Storage
The ingestion processes typically hand the data off to the components that
manage storage, so that it can be reliably persisted to disk

The volume of incoming data, the requirements for availability, and the
distributed computing layer make more complex storage systems necessary

Usually a distributed file system for raw data storage is used

Persisting the Data in Storage
Solutions such as Apache Hadoop’s HDFS filesystem, Ceph, and GlusterFS
allow large quantities of data to be written across multiple nodes in the cluster

Distributed databases, especially NoSQL databases, are well-suited for this

role because they are often designed with the same fault tolerant
considerations and can handle heterogeneous data
Computing and Analyzing Data
Once the data is available, the system can begin processing the data to
surface actual information

Data is often processed repeatedly, either iteratively by a single tool or by

using a number of tools to surface different types of insights
Computing and Analyzing Data
There are two ways of processing big data

Batch Processing

Real-time (Stream) Processing

Batch processing
This process involves breaking work up into smaller pieces, scheduling
each piece on an individual machine, reshuffling the data based on the
intermediate results, and then calculating and assembling the final result

These steps are often referred to individually as splitting, mapping,

shuffling, reducing, and assembling, or collectively as a distributed map
reduce algorithm

Batch processing is most useful when dealing with very large datasets that
require quite a bit of computation
Batch processing
This is the strategy used by Apache Hadoop’s MapReduce
Real-time processing
Demands that data be processed and made ready immediately and
requires the system to react as new data becomes available

One way of achieving this is stream processing, which operates on a

continuous stream of data composed of individual items

Another common characteristic of real-time processors is in-memory

computing, which works with representations of the data in the cluster’s
memory to avoid having to write back to disk

Apache Storm, Apache Flink, and Apache Spark provide different ways of
achieving real-time or near real-time processing
Visualizing the Results
Visualizing data is one of the most useful ways to spot trends and make
sense of a large number of data points

Real-time processing is frequently used to visualize application and server

metrics

Technologies for visualization

Prometheus, Elastic Stack, Jupyter Notebook, and Apache Zeppelin