Data Mining
Why Data
Preprocessing:
Introduction
Data Preprocessing - Introduction
What is Data
Pre-processing
Process raw data to
prepare it for another
processing procedure
Transforming raw
data
into an
understandable
format.
Data Preprocessing - Introduction
Why DP
• No quality = No DM
• Decisions = QD
• Data is dirty in real
world
• Noisy
• Incomplete
• Inconsistent
Data Preprocessing - Introduction
Noisy & Inconsistent
Data
Noisy data
Random variance
and/or error in
measurement
Containing errors or
outliers
Data Preprocessing - Introduction
Incomplete data
Lacking attribute
values
Lacking certain
attributes of interest
Containing only
aggregate data
Data Preprocessing - Introduction
Inconsistent data
Containing
discrepancies in codes
or names
Age=“42”
birthday=“03/07/1997”
Rating “1,2,3”,
“A, B, C”
Data Mining
Why data
Preprocessing:
Why is data dirty
Why is data dirty
Reasons
• Noise
• Incompleteness
• Inaccuracy
• Inconsistency
• Timeliness
Why is data dirty
Reason of Noise
• Faulty data
collection
instruments
• Human or computer
error at data entry
• Errors in data
transmission
Why is data dirty
Incompleteness
“Not applicable” data
value when collected
Data collection &
analysis time difference
Human/HW/SW
problems
Why is data dirty
Reasons of Inaccuracy
• Data
transmission
• Inconsistent
naming
conventions,
• Duplicate tuples
• Inaccurate data
collection
Why is data dirty
Inconsistency &
Timeliness
Different data
sources
Functional
dependency violation
Data collection not
on required
frequency
Data Mining
Why data
Preprocessing:
Multi-Dimensional
Measure of Data
Quality
Measuring Data Quality
Measure of Data Quality
• Accuracy
• Completeness
• Consistency
• Timeliness
• Believability
• Value added
• Interpretability
• Accessibility
Measuring Data Quality
Accuracy &
Completeness
Data stored is correct
or not.
Unambiguous.
Assures that all data
for required
information is
available or not.
Measuring Data Quality
Consistency &
Timeliness
Data is in same format
at all time and from
different sources.
Availability of data in
required time.
Measuring Data Quality
Believability & Value
added
How much data can
be trusted that it is
true
What impact new
data has on existing
Measuring Data Quality
Interpretability &
Accessibility
How easily data can
be understood.
How and how easily
data can be
accessed
Data Mining
Data Cleaning
Introduction
Data Cleaning
Introduction
fill in missing values
smooth out noise
identifying outliers
correct
inconsistencies
Data Cleaning
Advantage
False, inaccurate or
misdirecting
conclusions
Make data more
reliable and
accurate
Data Cleaning
Need
Transmission error
Faulty equipment
Error due to different
conventions or scales
Availability of data
Data Mining
Data Cleaning
Missing Data
Missing Data
Missing data
Missing data is
unavailability of
essential data
which is required to
draw a conclusion
or information.
Missing Data
Reasons for Missing
Equipment
malfunction
Inconsistent with
recorded data/deletion
Data not entered
Not register history or
changes of the data
Missing Data
Handling missing values
Ignore the tuple
Fill in the missing
value manually
Fill in automatically
a global constant
Attribute mean
Most probable value
Data Mining
Data Cleaning
Noisy Data
Introduction
Noisy Data Intro
Missing data
Random error or
variance in a
measured variable.
Noisy data can be
expressed as
meaningless or
corrupt data that
cant be understood
by machine.
Noisy Data Intro
Reasons for Missing data
faulty instruments
data entry problems
transmission problems
technology limitation
Inconsistency in
naming convention
Noisy Data Intro
Handling Techniques
Binning
Regression analysis
Outlier analysis in
clustering
Combined computer
and human
inspection
Data Mining
Data Cleaning
Binning
Binning
Binning
Smooth sorted
data by
neighborhood
The sorted values
are distributed
into a number of
buckets or bins.
Binning
Binning Methods
Bin Medians, Bin Boundaries
Data Mining
Data Cleaning
Models
Data Cleaning - Models
Models
Linear Regression
Clustering
Data Cleaning - Models
Linear Regression
Line to fit two attributes
One att to predict other
Fit the data into fns.
Approx fn to capture
imp patterns/values
FN to find data set
values
Data Cleaning - Models
Clustering
Similar values into
groups or clusters
Detect and remove
outliers.
Procedure
