Induction and Decision Trees

Artificial Intelligence

The design and development of computer systems that exhibit intelligent behavior.

What is intelligence?

Turing test: Developed in 1950 by Alan Turing (pioneer in computer

science) Computer and human in one room Human “interrogator” in another room Interrogator asks questions...human OR computer answers If interrogator cannot tell whether the human or the computer

is answering, then the computer is “intelligent”

Classification of AI Systems

Knowledge Representation Systems Capture existing expert knowledge and use it to consult end-users

and provide decision support Main types: Rule-based expert systems, Case-base reasoning systems, Frame-based

knowledge systems, Semantic networks

Machine Learning Algorithms that use mathematical or logical techniques for finding

patterns in data and discovering or creating new knowledge Main types: Artificial neural networks, genetic algorithms, inductive decision

trees, Naïve Bayesian algorithms, Clustering and pattern-recognition algorithms

Data mining involves primarily a “machine learning” form of AI

Data Mining

Textbook definition: Knowledge discovery in databasesUsing statistical, mathematical, AI, and

machine learning techniques to extract useful information and subsequent knowledge from large databases

Key point: identifying patterns in large data sets

5

Microsoft SQL Server Data Mining Algorithms

Decision TreesNaïve BayesianClusteringSequence ClusteringAssociation RulesNeural NetworkTime Series

Decision Trees for Machine Learning

Based on Inductive Logic

Three types of logical structures commonly used in AI systems:DeductionAbduction Induction

Deduction

Premise (rule): if p then q

Fact (axiom, observation): p

Conclude: q

This is classical logic (Modus Ponens). If the rule is correct, and the fact is correct, then you know that the conclusion will be correct.

We are given the rule

Abduction

Premise (rule): if p then q

Fact (axiom, observation): q

Conclude: p

This form of reasoning is a logical fallacy called “affirming the consequent” (Post hoc ergo propter hoc). The conclusion may be wrong, but it is a plausible explanation of the fact, given the rule. Useful for diagnostic tasks.

We are given the rule

Induction

1. Observe p and q together

2. .

3. .

4. .

n. Observe p and q together

Conclude: if p then q

This is stereotypical thinking…highly error prone.

We create the rule

Example – Baby in the kitchen

ID3 Decision Tree Algorithm

“Iterative Dichotomizer”

Developed by Ross Quinlan (1979)

This is the basis for many commercial induction products

The goal of this algorithm is to find rules resulting in YES or NO values. (Therefore, the output of generated rules have 2 possible outcomes)

ID3 generates a tree, where each path of the tree represents a rule. The leaf node is the THEN part of the rule, and the nodes leading to it are the ANDS of attribute-value combinations in the IF part of the rule.

ID3 Algorithm

Starting Point: an empty tree (this tree will eventually represent the final

rules created by ID3) a recordset of data elements (e.g. records from a database) a set attributes (fields), each with some finite number of

possible values NOTE: one of the attributes is the “decision” field, with a

YES or NO value (or some other 2-valued option...GOOD/BAD, HIGH/LOW, WIN/LOSE, etc.)

Output: a tree, where each path of the tree represents a rule

ID3 algorithmIf all records in your recordset are positive (i.e. have YES values for their decision attribute), create a YES node and stop (end recursion)

If all records in your recordset are negative, create a NO node and stop (end recursion)

Select the attribute that best discriminates among the records (using an entropy function)

Create a tree-node representing that attribute, with n branches, where n is the number of values for the selected attribute

Divide the records of the recordset into subsets subrecordset 1, subrecordset 2, ..., subrecordset n corresponding with each value of the selected attribute

Recursively apply the algorithm to each subrecordset i, with reduced attribute set (don’t include already used attributes further down the path)

Calculating Entropy

Entropy = mixture, chaos

We want to pick the attribute with the lowest entropy: ideally, a particular value for the input attribute

leads to ALL yes or ALL no in the outcome attribute…or come as close to this as possible

An attribute’s entropy =

Where n is the total number of possible values for the attributeand xi is the ith value

Baby’s RecordSet of Oven-Touching Experiences

ID3 Applied to Baby-in-the-Kitchen

Which attribute to start with? Based on Entropy measure (assuming log base 2),Touch stove entropy = 0.918Mom in kitchen entropy = 1.0To see this, note that:

Probability of touching stove leading to ouch is .67, and not leading to ouch is .33

.67 * .33 = .22

Probability of mom being in kitchen leading to ouch is .5 and mom being in kitchen not leading to ouch is also .5

.5 * .5 = .25

Applying the Touch Stove Attribute

Recurse …apply the Mom in Kitchen attribute where needed

Resulting decision rules

If Touch_Oven = No then BOO_BOO = No

If Touch_Oven = Yes and Mom_In_Kitchen = Yes then BOO_BOO = Yes

If Touch_Oven = Yes and Mom_In_Kitchen = No then BOO_BOO = No

Now we’ll do this with Microsoft SQL Server