Chapter 2
Data Manipulation and
Analysis
1
Data Manipulation and Analysis
• Data collection
• Data Preprocessing
Why Data Preprocessing?
Preprocessing
Reading data
Selecting filtering data
Data cleaning
Filtering missing values
Dropping/replacing missing values
Data integration
Data transformation
Manipulating data
Data reduction
Data Discretization and concept hierarchy generation
• Exploratory Data Analysis (EDA)
• Introduction to Pandas and Numpy
2 2
Data Manipulation and Analysis
Data collection:
is a systematic process of gathering observations
or measurements.
Whether you are conducting research for
business, governmental, or academic purposes,
data collection allows you to gain first-hand
knowledge and original insights into your research
problem.
3
Data Manipulation and Analysis
Steps in Data collection:
1. Define the Aim of Your Research
2. Choose Your Data Collection Method
3. Plan Your Data Collection Procedures
4. Collect the Data
4
Data Manipulation and Analysis
Steps in Data collection:
1. Define the Aim of Your Research
Clarify your research objectives.
Write problem statement and Formulate
research questions.
Decide data type: Quantitative (Numeric) or
qualitative (expressed in words) or mixed
approach.
5
Data Manipulation and Analysis
Steps in Data Collection
2. Choose Your Data Collection Method:
o Based on the data you want to collect, decide on the most appropriate method:
Surveys and Questionnaires: Gather information through structured questions.
Observations: Observe and record behaviors, events, or phenomena.
Interviews: Conduct one-on-one or group interviews to gather in-depth insights.
Existing Data: Use data that already exists (e.g., historical records, databases).
Experiments: Manipulate variables to observe their effects.
Case Studies: Investigate a specific individual, group, or situation.
Sampling: Collect data from a subset of the population.
Sensor Data: Use sensors or devices to collect real-time data.
Social Media Data: Analyze content from social platforms.
Field Notes: Record observations during fieldwork.
Diaries or Journals: Collect self-reported data over time.
6
Data Manipulation and Analysis
Steps in Data collection:
3. Plan Your Data Collection Procedures:
Develop a detailed plan for data collection:
Sampling Strategy: Decide how to select participants or
cases.
Data Collection Tools: Prepare surveys, interview guides, or
observation protocols.
Data Recording: Specify how you’ll record data (e.g., paper
forms, digital tools).
Ethical Considerations: Ensure informed consent and protect
participants’ privacy.
Pilot Testing: Test your data collection procedures before full
implementation.
7
Data Manipulation and Analysis
Steps in Data collection:
4. Collect the Data:
Execute your plan, following the established
procedures.
Be consistent, accurate, and thorough in recording
observations or measurements.
Address any unexpected challenges during data
collection.
8
Data Manipulation and Analysis
Why Data Preprocessing?
Quality decisions must be based on quality data
Data in the real world is full of dirty
incomplete:
lacking attribute values that is vital for decision making so they have
to be added,
lacking certain attributes of interest in certain dimension and should
be again added with the required value,
containing only aggregate data so that the primary source of the
aggregation should be included
noisy: containing errors or outliers that deviate from the expected
inconsistent: containing discrepancies in codes or names of the
organization or domain
etc
9
Data Manipulation and Analysis
Why Data Preprocessing?
Incomplete, noisy and inconsistent data are commonplace
properties of large real world databases and data sources
Data cleaning routine work to clean such problems so that
results can be accepted
Before starting data preprocessing, it will be advisable to have
overall picture of the data we have so that it tell us high level
summary such as
General property of the data
Which data values should be considered as noise or outliers
This can be done with the help of exploratory data analysis
10
Data Manipulation and Analysis
Exploratory Data Analysis (EDA) or Descriptive
Data Summarization
Descriptive summary about data can be
generated with the help of
measure of central tendency of the data,
measure of dispersion of the data and
their graphic display
11
Data Manipulation and Analysis
Exploratory Data Analysis (EDA) or Descriptive
Data Summarization
Descriptive summary about data can be
generated with the help of
measure of central tendency of the data,
measure of dispersion of the data and
their graphic display
Measure of central tendency includes
Mean, Median, and Mode
12
Data Manipulation and Analysis
Exploratory Data Analysis (EDA) or Descriptive
Data Summarization
Measure of dispersion includes
Range, Quartiles, Interquartile and range (IQR)
The five number summary (based on Quartiles)
minimum, Q1, median (Q2), Q3, IQR, and maximum
Variance and Standard deviation
13
Data Manipulation and Analysis
Exploratory Data Analysis (EDA) or Descriptive Data
Summarization
Graphical Methods
The BoxPlots
Can be plotted based on the five number summary
It is useful tool for identifying outliers
It is also one of the popular way of visualizing a distribution
14
Data Manipulation and Analysis
Exploratory Data Analysis (EDA) or Descriptive Data
Summarization
The BoxPlots
The end of the box are the quartiles Q1 and Q3 so that the length of
the box is the IQR
The median is marked by a line within the box
Two lines (called whiskers) outside the box extends to the smallest
(Minimum) and largest (Maximum) observation
The whiskers should extended to the extreme low and high value
only if these values are less than 1.5IQR beyond the quartiles.
Otherwise the whiskers terminates at the most extreme observation
occurring within 1.5IQR of the quartiles
The remaining observations are plotted individually to show outliers
15
Data Manipulation and Analysis
Exploratory Data Analysis (EDA) or Descriptive Data
Summarization
The BoxPlots
Boxplot for the unit price data for items sold at four branches
16
Data Manipulation and Analysis
Exploratory Data Analysis (EDA) or Descriptive Data
Summarization
Other graphical Methods
Pie charts
Bar charts
Histograms
Quantile plots
q-q plots
Scatter plots
etc.
17
Data Manipulation and Analysis
Major Tasks in Data Preprocessing
Data pre-processing in data analytics activity refers to the
processing of the various data elements to prepare for the
analytics operation.
Any activity performed prior to mining the data to get
knowledge out of it is called data pre-processing
This involves:
Data cleaning
Data integration
Data transformation
Data reduction
Data Discretization and concept hierarchy generation
18
Data Manipulation and Analysis
Major Tasks in Data Preprocessing
Data Cleaning: Refers to the process of
filling in missing values,
smooth noisy data,
identify or remove outliers, and resolve inconsistencies
19
Data Manipulation and Analysis
Major Tasks in Data Preprocessing
Data integration:
Combines data from multiple sources (databases, data
cubes, or files) into a coherent store
There are a number of issues to consider during data
integration. Some of these are:
Schema integration issue
Entity identification issue
Data value conflict issue
Avoiding redundancy issue
20
Data Manipulation and Analysis
Major Tasks in Data Preprocessing
Data Transformation
• Data transformation is the process of transforming or
consolidating data into a form appropriate for mining which is
more appropriate for measurement of similarity and distance
This involves:
Smoothing
Aggregation
Generalization
Normalization
Attribute/feature construction
21
Data Manipulation and Analysis
Major Tasks in Data Preprocessing
Data reduction
Data sources may store terabytes of data
Complex data analysis/mining may take a very long time to
run on the complete dataset
Data reduction tries to obtain a reduced representation of the
data set that is much smaller in volume but yet produces the
same (or almost the same or better) analytical results
22
Data Manipulation and Analysis
Major Tasks in Data Preprocessing
Data Discretization and concept hierarchy generation
Data discritization refers to transforming the data set which
is usually continous into discrete interval values
Concept hierarchy refers to generating the concept levels so
that data mining function can be applied at specific concept
level
Can be used to reduce the number of values for a given
continuous attribute by dividing the range of attribute into
intervals
Interval labels can be used to replace actual data values
This leads to concise, easy to use, knowledge level
representation of mining result
23
Introduction to Numpy and Pandas
NumPy
Is a fundamental package for scientific computing with Python.
It provides support for large, multi-dimensional arrays and
matrices, along with a collection of high-level mathematical
functions to operate on these arrays.
Some key features of NumPy include:
Multi-dimensional array objects (ndarray)
Mathematical functions for fast operations on arrays
Tools for reading and writing array data to disk
Linear algebra and random number generation capabilities
24
Introduction to Numpy and Pandas
Pandas
Is a powerful and easy-to-use open-source data analysis and
manipulation tool built on top of the Python programming
language.
It offers data structures and data analysis tools that are ideal for
working with structured data.
Key features of pandas include:
DataFrame object for data manipulation with integrated indexing
Tools for reading and writing data between in-memory data
structures and different file formats
Data alignment and handling of missing data
Reshaping and pivoting of data sets
Time series functionality
25
