Embed Size (px)
Citation preview
CpSc 810: Machine Learning
Decision Tree Learning
2
Copy Right Notice
Most slides in this presentation are adopted from slides of text book and various sources. The Copyright belong to the original authors. Thanks!
3
Review
Concept learning as search through H
S and G boundaries characterize learner’s uncertainty
A learner that make no a priori assumption regarding the identity of the target concept has no rational basis for classifying any unseen instance.
Inductive leaps possible only if learner is biased
4
Decision Tree Representation for PlayTennis
Outlook
Humidity
SunnyOvercast
Rain
High Normal Strong Weak
A Decision Tree for the concept PlayTennis
Wind
5
Decision Trees
Decision tree representationEach internal node tests an attributeEach branch corresponds to attribute valueEach leaf node assign a classification
Decision tree represent a disjunction of conjunctions of constraints on the attribute values of instances.
6
Appropriate Problems for Decision Tree Learning
Instances are represented by discrete attribute-value pairs (though the basic algorithm was extended to real-valued attributes as well)
The target function has discrete output values (can have more than two possible output values --> classes)
Disjunctive hypothesis descriptions may be required
The training data may contain errors
The training data may contain missing attribute values
7
ID3: The Basic Decision Tree Learning Algorithm
Top-down induction of decision tree
8
Which attribute is best?
9
Which attribute is best?
Choose the attribute that minimize the Disorder in the subtree rooted at a given node.
Disorder and Information are related as follows: the more disorderly a set, the more information is required to correctly guess an element of that set.
10
Information theory
Information: What is the best strategy for guessing a number from a finite set of possible numbers? i.e., how many questions do you need to ask in order to know the answer (we are looking for the minimal number of questions). Answer Log2(S), where S is the set of numbers and |S|, its cardinality.
Q1: is it smaller than 5?Q2: is it smaller than 2?
E.g.: 0 1 2 3 4 5 6 7 8 9 10
Q1Q2
11
Information theory
Information theory: optimal length code assigns -Log2p bits to message having probability p. So expected number of bits to encode + or – of random member of S is
)log()log( 22 pppp
12
Entropy
S is a set of training examples
p+ is the proportion of positive examples in S
p- is the proportion of negative examples in S
Entropy measures the impurity of S
Entropy(S) = expected number of bits needed to encode class (+ or -) of randomly drawn member of S (under the optimal, shortest-length code
ppppsEntropy 22 loglog)(
13
Information Gain
Gain(S, A) = expected reduction in entropy due to sorting on A
)()(),()(
vAValuesv
v SEntropyS
SSEntropyASGain
14
Training Examples
Entropy?
15
Which attribute is the best?
S: [9+. 5-]E=0.94
Humidity
[3+, 4-] [6+, 1-]
High Normal
E=0.985 E=0.592
Gain(S, Humidity) = 0.94 – (7/14)*0.985-(7/14)*0.592 = 0.151
16
Which attribute is the best?
S: [9+. 5-]E=0.94
Wind
[6+, 2-] [3+, 3-]
Weak Strong
E=0.811 E=1.0
Gain(S, Wind) = 0.94 – (8/14)*0.811-(6/14)*1.0 = 0.048
17
Which attribute is the best?
S: [9+. 5-]E=0.94
Temperature
[2+, 2-] [3+, 1-]
Hot Cool
E=1.0 E=0.811
Gain(S, Temperature) = 0.94 – (4/14)*1.0-(6/14)*0.918-(4/14)*0.811 = 0.029
[4+, 2-]
E=0.918
Mild
18
Which attribute is the best?
S: [9+. 5-]E=0.94
Outlook
[2+, 3-] [3+, 2-]
Sunny Rain
E=0.970 E=0.970
Gain(S, Outlook) = 0.94 – (5/14)*0.970-(4/14)*0.0-(5/14)*0.970 = 0.246
[4+, 0-]
E=0
Overcast
19
Which attribute is the best?
Gain(S, Humidity) = 0.151
Gain(S, Wind) = 0.048
Gain(S, Temperature) = 0.029
Gain(S, Outlook) = 0.246
Outlook
20
Selecting the Next attribute
S: [9+. 5-]E=0.94
Outlook
[2+, 3-] [3+, 2-]
Sunny Rain
E=0.970 E=0.970
[4+, 0-]
E=0
Overcast
? ?
Which attribute should be tested here?
21
Selecting the Next attribute
S: [9+. 5-]E=0.94
Outlook
[2+, 3-] [3+, 2-]
Sunny Rain
E=0.970 E=0.970
[4+, 0-]
E=0
Overcast
? ?
Gain(Ssunny, Humidity) = 0.97-(3/5)*0.0-(2/5)*0.0=0.97
Gain(Ssunny, Temperature) = 0.97-(3/5)*0.0-(2/5)*1.0-(1/5)0.0=0.57
Gain(Ssunny, Wind) = 0.97-(2/5)*1.0-(3/5)*0.918=0.019
22
Hypothesis Space Search in ID3
Hypothesis Space: Set of possible decision trees.
Hypotheses space is complete space of finite discrete-valued functions.The target function surely in there.
Search Method: Simple-to-Complex Hill-Climbing Search
only output a single hypothesis ( from candidate-elimination method).Unable to explicitly represent all consistent hypotheses.No Backtracking!!! – Local minima.
23
Hypothesis Space Search in ID3
Evaluation Function: Information Gain Measure
Batch Learning: ID3 uses all training examples at each step to make statistically-based decisions ( from candidate-elimination method which makes decisions incrementally).
the search is less sensitive to errors in individual training examples.
24
Inductive Bias in ID3
Note H is the power set of instance X -> unbiased?
Not ReallyPreference for short treesPreference for trees with high information gain attributes near the root.
Restriction Bias vs. Preference BiasID3 search a complete hypothesis space, but it searches incompletely through this space. (preference or search bias) Candidate-Elimination searches an incomplete hypothesis space. However, it search this space completely. (restriction or language bias)
25
Why Prefer Short Hypotheses?
Occam’s razor: Prefer the simplest hypothesis that fits the data [William Occam (Philosopher), circa 1320]
Scientists seem to do that: E.g., Physicist seem to prefer simple explanations for the motion of planets, over more complex ones
Argument: Since there are fewer short hypotheses than long ones, it is less likely that one will find a short hypothesis that coincidentally fits the training data.
Problem with this argument: it can be made about many other constraints. Why is the “short description” constraint more relevant than others?
Two learner use different representation will result different hypotheses.
Nevertheless: Occam’s razor was shown experimentally to be a successful strategy.
26
Overfitting
Consider error of hypotheses h over
Training data: errortrain(h)
Entire distribution D of data: errorD(h)
Hypothesis h H overfits training data if there is an alternative hypothesis h’ H such that
and
)'()( herrorherror traintrain
)'()( herrorherror DD
27
Overfitting in Decision Trees
Consider adding a noisy training example to the PlayTennis
<Outlook=Sunny, Temperature=Hot, Humidity=Normal, Wind=Strong, PlayTennis = No>
What effect the earlier tree?
28
Overfitting in Decision Trees
S: [9+. 5-]E=0.94
Outlook
[2+, 3-] [3+, 2-]
Sunny Rain
E=0.970 E=0.970
[4+, 0-]
E=0
Overcast
Humidity Wind
High Normal
{D1, D2, D8} {D9, D11}[3-] [2+]
New Data add a - here
{D9, D11, D15}[2+, 1-]
29
Overfitting in Decision Trees
30
Avoiding Overfitting
How can we avoid overfitting?Stop growing the tree when data split not statistically significantGrow full tree, then post-prune it.
How to select “best” tree:Measure performance over training data
Apply a statistical test to estimate whether pruning is likely to improve performance.
Measure performance over separate validation data setMinimum Description Length (MDL):
Minimize size(tree)+size(misclassifications(tree))
31
Reduced-Error Pruning
Split data into training and validation set
Do until further pruning is harmful:1. Evaluate impact on validation set of pruning each possible node (plus those below it).2. Greedily remove the on that most improves validation set accuracy
Produces smallest version of most accurate tree
Drawback: what if data is limited?
32
Effect of Reduced-Error Pruning
33
Rule Post-Pruning
Most frequently used method
34
Converting a Tree to Rules
IF (Outlook = Sunny) ∩ (Humidity = High)Then PlayTennis = No
IF (Outlook = Sunny) ∩ (Humidity = Normal)Then PlayTennis = Yes
35
Why converting to rule
Converting to rules allows distinguishing among the different contexts in which a decision node is used
Converting to rules removes the distinction between attribute tests that occur near the root of the tree and those occur near the leaves.
Converting to rules improves readability.
36
Incorporating Continuous-Valued Attributes
For a continuous valued attribute A, create a new boolean attribute Ac that is true if A < c and false otherwise.
Select the threshold c that produces the greatest information gain.Example:
Two possible threshold c1=(48+60)/2=54; c2=(80+90)/2=85
Which one is better? (homework)
37
Alternative Measures for Selecting Attributes: GainRatio
Problem: information gain measure favors attributes with many values.
Example: add a Date attribute in our example
One solution: GainRatioSplit Information
Where Si is subset of S for which A have value vi
Gain Ratio
S
S
S
SASmationSplitInfor ic
i
i2
1log),(
),(
),(),(
ASmationSplitInfor
ASGainASGainRatio
38
Attributes with Differing Costs
In some learning tasks the instance attributes may have associated cost. How to learn a consistent tree with low expected cost?
Approaches:Tan and Schlimmer (1990)
Nunez(1988)
Where determines importance of cost
)(
),(2
ACost
ASGain
w
ASGain
ACost )1)((
12 ),(
]1,0[w
39
Training Examples with Missing Attribute Values
If some examples missing values of attribute A, use training examples anyway.
Assign most common value of A among other examples with same target values
Assign probability pi to each possible vi value of A. Then, assign fraction pi of example to each descendant in tree.
Classify new examples in same fashion.
40
Summary Points
Decision tree learning provides a practical method for concept learning and for learning other discrete-valued functions
ID3 searches a complete hypothesis space.Target function always present in the hypothesis space.
The inductive bias implicit in ID3 includes a preference for smaller trees.
Overfitting the training data is an important issue in decision tree learning.
A large variety of extensions to the basic ID3 algorithm has been developed.
