CLASSIFICATION: Business Analytics
DECISION TREES Lecture 7/8
LEARNING OBJECTIVES
• Explain what is classification
• Define decision tree
• Compare the advantages and disadvantages of decision tree
• Building decision a tree
• Evaluating a decision tree
A. Explain What is Classification
WHAT IS CLASSIFICATION
• Classification is a data mining function that assigns items into
categories or classes.
• The goal of classification is to accurately predict the target class for
each case in the data.
A. Explain What is Classification
EXAMPLES OF
CLASSIFICATION TASKS
• Identifying loan applicants as low, medium, or high credit risks.
• Predicting tumour cells as benign or malignant
• Classifying credit card transactions as legitimate or fraudulent
A. Explain What is Classification
CLASSIFICATION RULES
• Classification rules help assign new objects to classes.
E.g., given a new automobile insurance applicant, should he or she be
classified as low risk, medium risk or high risk?
• Classification rules for the above example could use a variety of data, such
as educational level, salary, age, etc.
Person P, P.degree = master and P.income > 75,000 ⇒ P.credit = excellent
Person P, P.degree = bachelorsand (P.income > 25,000 and P.income< 75,000) ⇒ P.credit = good
• Rules are not necessarily exact - there may be some misclassifications
• Classification rules can be represented by a decision tree.
B. Define Decision Tree
WHAT IS A DECISION TREE?
• A decision tree is a hierarchical collection of rules
that describes how to divide a collection of records
into successively smaller groups of records.
• The aim of the division is to have resulting segments
become more and more similar (pure) to one another
with respect to the target.
• It is a predictive model based on a branching series
of tests
• Can be used for binary or multiple outcomes
• Allows us to understand which variables are
important
• Spot unexpected patterns
B. Define Decision Tree
STRUCTURE OF A DECISION TREE
• Consist of root, nodes, leaves, and splits
• At each node, a decision is made on which
variable to split
• These variables are the most important
• All records landing at the same leaf get the
same prediction
C. Advantages and Disadvantages of Decision Tree
PROS AND CONS
OF DECISION TREES
Pros Cons
+ Reasonable training time - Cannot handle complicated
relationship between attributes
+ Fast application
- Problems are created with lots of
+ Easy to interpret
missing data
+ Easy to implement
+ Can handle large number of
attributes
D. Building a Decision Tree
PURPOSE OF A DECISION TREE
• Given a collection of records (training set )
‒ Each record contains a set of attributes.
• One of the attributes is the class.
• The aim is to find a model for the class attribute as a function of the values
of other attributes.
• Goal: previously unseen records should be assigned a class as accurately as
possible.
‒ A test set is used to determine the accuracy of the model.
‒ Usually, the given data set is divided into training and test sets, with
training set used to build the model and test set used to validate it.
D. Building a Decision Tree
DECISION TREE
CLASSIFICATION TASK
D. Building a Decision Tree
DECISION TREE USING HUNT’S
ALGORITHM
Hunt's algorithm
grows a decision tree
in a recursive fashion
by partitioning the training records
into successively purer subsets.
D. Building a Decision Tree
BUILDING THE DECISION TREE
• We start at the root node with all records in the training set
• Drawn from left to right
• Consider every split on every variable
• Choose the split that maximizes a measure of purity
• For each child of the root node, we again search for the best split
• Eventually, the process stops when no good split is available or leaves are
pure
D. Building a Decision Tree
BUILDING THE DECISION TREE
Step 1
Don’t Cheat (4 )
Cheat (2)
D. Building a Decision Tree
BUILDING THE DECISION TREE
Step 2
Don’t Cheat (1 )
Cheat (3)
D. Building a Decision Tree
BUILDING THE DECISION TREE
Step 3
D. Building a Decision Tree
BUILDING THE DECISION TREE
Step 1
Step 3
Step 2
E. Evaluating a Decision Tree
BUILDING THE DECISION TREE
E. Evaluating a Decision Tree
APPLY MODEL TO TEST DATA
E. Evaluating a Decision Tree
APPLY MODEL TO TEST DATA
E. Evaluating a Decision Tree
APPLY MODEL TO TEST DATA
E. Evaluating a Decision Tree
HOW TO SPLIT DATA
FOR TEST CONDITION
• Depends on attribute types
‒ Nominal (Categorical)
‒ Ordinal (Categorical but ordered for example education level)
‒ Continuous (any value can be represented)
• There are two types of splits
‒ Multi-way split
‒ 2-way split (binary)
E. Evaluating a Decision Tree
SPLITTING BASED ON
NOMINAL ATTRIBUTES
E. Evaluating a Decision Tree
SPLITTING BASED ON
NOMINAL ATTRIBUTES
E. Evaluating a Decision Tree
SPLITTING BASED ON
CONTINUOUS ATTRIBUTES
Different ways of handling
• Change value to form an ordinal categorical attribute
• Binary Decision: (A < v) or (A >=v)
‒ Consider all possible splits and finds the best cut
E. Evaluating a Decision Tree
FINDING A GOOD SPLIT
AT A DECISION TREE NODE
• There are many ways to find a good split
• But, they have two things in common - Splits are preferred where
‒ The children are similar in size
‒ Each child is as pure as possible
• Most algorithms seek to maximize the purity of each of the children
E. Evaluating a Decision Tree
HOW TO DETERMINE
THE BEST SPLIT
• Nodes with homogeneous class distribution are preferred
E. Evaluating a Decision Tree
HOW TO DETERMINE
THE BEST SPLIT
• Before Splitting: 10 records of class 0, 10 records of class 1
E. Evaluating a Decision Tree
PERFORMANCE MEASURES FOR
DECISION TREES
• After a decision tree is constructed, each leaf node has a score
• A leaf score is the likelihood that the more common class arises
• A decision tree also has an accuracy score which is calculated as
follows:
Accuracy = # Correctly classified/ Total #
E. Evaluating a Decision Tree
CALCULATING THE ACCURACY OF
A DECISION TREE
E. Evaluating a Decision Tree
EVALUATION OF
CLASSIFICATION MODELS
• Counts of test records that are correctly (or incorrectly) predicted by
the classification model
• Confusion matrix
E. Evaluating a Decision Tree
EXERCISE FOR DECISION TREE -
SHOULD WE GO SAILING?
yes (2)
no (3)
E. Evaluating a Decision Tree
EXERCISE FOR DECISION TREE -
SHOULD WE GO SAILING?
yes (1)
no (1)
E. Evaluating a Decision Tree
EXERCISE FOR DECISION TREE -
SHOULD WE GO SAILING?
E. Evaluating a Decision Tree
EXERCISE FOR DECISION TREE -
SHOULD WE GO SAILING?
QUESTIONS?
