Empirical Explorations with Empirical Explorations with The Logical Theory Machine:The Logical Theory Machine:

A Case Study in HeuristicsA Case Study in Heuristics

by Allen Newell, J. C. Shaw, & H. A. Simonby Allen Newell, J. C. Shaw, & H. A. Simon

The Logic Theory Machine (LT)The Logic Theory Machine (LT)

: a system for finding proofs of theorems in : a system for finding proofs of theorems in elementary symbolic logicelementary symbolic logic

: devised to understand the complex problem-: devised to understand the complex problem-solving processsolving process

: the first heuristic program (1956): the first heuristic program (1956)

The Logic Theory Machine in OperationThe Logic Theory Machine in Operation

- LT’s Task is to prove theorems in the- LT’s Task is to prove theorems in the

sentential calculus ( in sentential calculus ( in Principia Mathematica Principia Mathematica ))

- The proof is the sequence of expressions that leads fro- The proof is the sequence of expressions that leads from the axioms and known theorems to the desired exprm the axioms and known theorems to the desired expression e.g.) ession e.g.)

- Why are such problems difficult ? - Why are such problems difficult ?

- What features of LT account for its successes - What features of LT account for its successes

and failures ? and failures ?

Problems, Algorithms, and HeuristicsProblems, Algorithms, and Heuristics

problemproblem

: if he is given : if he is given some process for generatingsome process for generating the the elements of a set of possible solutions in some order, elements of a set of possible solutions in some order, and and a test for verifyinga test for verifying whether a given element of whether a given element of this set is in fact a solution to his problem. this set is in fact a solution to his problem.

problem solvingproblem solving : if the costs are not too large in relation to the time : if the costs are not too large in relation to the time

and computing power available for solution and computing power available for solution

Problems, Algorithms, and HeuristicsProblems, Algorithms, and Heuristics

algorithmalgorithm : the process that guarantee the solution of a given p: the process that guarantee the solution of a given p

roblem. ex) opening a combination saferoblem. ex) opening a combination safe

heuristicheuristic : the process that may solve a given problem but off: the process that may solve a given problem but off

ers no guarantees of doing so. ers no guarantees of doing so.

ex) playing chess ex) playing chess

The Problem of Proving Theorems iThe Problem of Proving Theorems in Logicn Logic

- the set of all possible sequences of logic expression - the set of all possible sequences of logic expression

: : EE ( given implicitly by rules for generating) ( given implicitly by rules for generating)

- the set of sequences that are proofs : - the set of sequences that are proofs : PP

- the expression to be proved : - the expression to be proved : XX

- the set of sequences that have - the set of sequences that have XX as their final expression : as their final expression : TTxx

PP :proof :proof

sequencesequence

TTxx : sequences : sequences

ending in ending in XX

EE : all sequences of : all sequences of

logic expressionslogic expressions

Proofs of Proofs of XX

The difficulty of proving theorems depends onThe difficulty of proving theorems depends on

1. The scarcity of elements in the intersection of 1. The scarcity of elements in the intersection of PP and and TTxx , relative to the number of elements in , relative to the number of elements in EE

the cost and speed of the available generators,the cost and speed of the available generators,

the cost and speed of making tests (whether the cost and speed of making tests (whether PP or or TTxx)

2. Whether generators can be found that guarantee that that any element they produce automatically satisfies somany element they produce automatically satisfies some of the conditions.e of the conditions.

3. What heuristics can be found to guide the selection.3. What heuristics can be found to guide the selection.

The British Museum AlgorithmThe British Museum Algorithm : reveal the basic nature of theorem proving: reveal the basic nature of theorem proving

the algorithm constructs all possible proofs in the algorithm constructs all possible proofs in systematic manner, checking each time (1) to systematic manner, checking each time (1) to eliminate duplicates, and (2) to see if the final eliminate duplicates, and (2) to see if the final theorem in the proof coincides with the expression theorem in the proof coincides with the expression to be provedto be proved

the set of n-step proofs : obtained from the set of the set of n-step proofs : obtained from the set of (n-1)-step proofs by making all the permissible (n-1)-step proofs by making all the permissible substitutions and replacements, detachments.substitutions and replacements, detachments.

Figure 2 . Number of proofs generated by first few steps Figure 2 . Number of proofs generated by first few steps

of British Museum algorithmof British Museum algorithm

: only use the simple substitution : only use the simple substitution

and the replacementsand the replacements

: If specializations are included: If specializations are included

11-steps, 1000 theorems11-steps, 1000 theorems

: among the first 246 theorems,: among the first 246 theorems,

only 5 theorem ( in chap. 2 of only 5 theorem ( in chap. 2 of

principia), 1000 theorems, one principia), 1000 theorems, one

moremore how many theorem to prove all theorems (60-odd) how many theorem to prove all theorems (60-odd)

: a hundred million: a hundred million how many times : hundreds of thousands of years how many times : hundreds of thousands of years

0

100

200

0 1 2 3 4 5 6 7 8

Proof steps

Num

ber of

pro

ofs

The Logic Theory MachineThe Logic Theory Machine

Methods Methods : the major type of heuristic that LT uses: the major type of heuristic that LT uses

The Substitution MethodThe Substitution Method

: by finding an axiom or proved theorem that can be transfor: by finding an axiom or proved theorem that can be transformed, by a series of substitutions for variables and replacemmed, by a series of substitutions for variables and replacements of connectivesents of connectives

The Detachment MethodThe Detachment Method

: if problem : if problem B , B , search for an axiom or theorem “ search for an axiom or theorem “ AA implies implies BB ” , ” , AA is set up as a new subproblem is set up as a new subproblem

Methods Methods (continued)(continued)

The Chaining MethodThe Chaining Method

: use the tansitivity of the relation of implication to create a: use the tansitivity of the relation of implication to create a

new subproblemnew subproblem

- forward chaining method- forward chaining method

: “: “ a a implies implies c c ”, search an axiom or theorem “ ”, search an axiom or theorem “ aa implies implies bb ”, ”, set up “set up “ b b implies implies c c ” as new subproblem” as new subproblem

- backward chaining method- backward chaining method

: search “ : search “ bb implies implies cc ”, set up “ ”, set up “ a a implies implies bb ” as subproblem ” as subproblem

Methods Methods (continued)(continued)

no guaranteeno guarantee

: a target axiom or a theorem can be found: a target axiom or a theorem can be found

: the generated subproblem is part of the desired pro: the generated subproblem is part of the desired proof sequence, or part of any proof sequenceof sequence, or part of any proof sequence

: the beginning of the sequence can be completed wi: the beginning of the sequence can be completed with axioms or proved theoremsth axioms or proved theorems

: the combination of four methods comprise a suffici: the combination of four methods comprise a sufficient set of methods to prove all theoremsent set of methods to prove all theorems

ex) “ ex) “ pp or not-not-not- or not-not-not-pp ” (2.13) ” (2.13)

Methods Methods (continued)(continued)

Why do the methods transform LT into an effectivWhy do the methods transform LT into an effective problem-solver ?e problem-solver ?

1. unitizing effect 1. unitizing effect

: the methods organize the sequences of individual processi: the methods organize the sequences of individual processing steps into larger units ng steps into larger units

2. backward approach2. backward approach

: only one theorem to be proved, a number of known true t: only one theorem to be proved, a number of known true theorems heorems

The Executive RoutineThe Executive Routine

1. For a theorem list, the substitution 1. For a theorem list, the substitution

method is triedmethod is tried

2. The detachment method is tried,2. The detachment method is tried,

following the substitution, if fail,following the substitution, if fail,

the subproblem is added to the the subproblem is added to the

subproblem listsubproblem list

3. Forward chaining, backward3. Forward chaining, backward

chaining.chaining.

4. For the subproblem list, repeat. 4. For the subproblem list, repeat.

The Executive Routine (continued)The Executive Routine (continued)

This process continues until (1) a proof is found, This process continues until (1) a proof is found,

(2) the time allotted for finding a proof is used up, (2) the time allotted for finding a proof is used up,

(3) there is no more available memory space, (3) there is no more available memory space,

(4) no untried problems remain on the subproblem(4) no untried problems remain on the subproblem

listlist

Examples Examples

The Executive Routine (continued)The Executive Routine (continued)

not (not (pp or or qq) implies not-) implies not-pp ((A A impliesimplies B B) implies (not) implies (not-B -B

implies notimplies not-A-A) (theorem 2.1) (theorem 2.1

6)6)

subprblems subprblems [ [ p p impliesimplies (p (p oror q q)] )]

by substitution methodby substitution method

A implies (A or B)A implies (A or B)

The Matching ProcessThe Matching Process

What allows the efficient search ?What allows the efficient search ? the required times : the required times : BM algorithm vs LT ( 5~100:1) BM algorithm vs LT ( 5~100:1)

ex) “ ex) “ pp implies ( implies (qq implies implies p p) ” (158 sec : 10 sec)) ” (158 sec : 10 sec)

pp implies ( implies (q q or or pp) (axiom 1.3)) (axiom 1.3)

pp implies (not- implies (not-qq or or pp))

pp imlies ( imlies (qq implies implies pp) )

11

99

55

1212

881010

4466

22

1111

77

33

The Matching Process (continued)The Matching Process (continued)

Matching in the Substitution MethodsMatching in the Substitution Methods

: componentwise matching: componentwise matching - a feedback of the resul- a feedback of the results of a substitution or replacementts of a substitution or replacement

: Of the 52 theorems of : Of the 52 theorems of PrincipiaPrincipia, 38 proofs , 38 proofs

substitution alone - 17 proof substitution alone - 17 proof Matching in Detachment and ChainingMatching in Detachment and Chaining : matching the subexpressions of the problem or th: matching the subexpressions of the problem or th

eorems. eorems.

Similarity Tests and DescriptionsSimilarity Tests and Descriptions

Screening Process : reject any theorem for Screening Process : reject any theorem for matching that has low likelihood of successmatching that has low likelihood of success

The Similarity Test : 1) the maximum numbers of The Similarity Test : 1) the maximum numbers of levels , 2) the number of distinct variables, 3) the levels , 2) the number of distinct variables, 3) the number of variable placesnumber of variable places

table 1. table 1. Percent Percent similar of matched Theorems Theorems Theorems theorems of theoremsMethod considered similar matched considered similar

Substitution 11,298 993 37 8.8 3.7Detachment 1,591 406 210 25.5 51.7Chain. forward 869 200 120 23.0 60.0Chain. backward 673 146 63 21.7 43.2

Similarity Tests and Descriptions (cont.)Similarity Tests and Descriptions (cont.)

similarity testsimilarity test 의 문제점의 문제점 : type II error ex) : type II error ex) pp implies ( implies (pp or or pp) ) pp implies ( implies (qq or or pp) )

: type I error: type I error

: the cost of similarity test: the cost of similarity test

Effort in LTEffort in LT measuring efforts

: the total number of primitives executed

computing effort and performance : substitution vs detach. vs chain. ( 1: 2 : 3)

Similarity Tests and Descriptions (cont.)Similarity Tests and Descriptions (cont.)

- precomputed, recomputed description- precomputed, recomputed description

- the effort required is proportional to the number of theorems co- the effort required is proportional to the number of theorems considered.nsidered.

- the number of theorems considered is an effort measure for eval- the number of theorems considered is an effort measure for evaluating a heuristicuating a heuristic

Evaluation of the Similarity TestEvaluation of the Similarity Test : the full similarity vs the modified similarity vs no similarity: the full similarity vs the modified similarity vs no similarity

: when precomputed, full : no (10 to 60 perc. 3534/5206): when precomputed, full : no (10 to 60 perc. 3534/5206)

when recomputed, (26,739/22,914)when recomputed, (26,739/22,914)

: modified similarity test (18,035/22,914): modified similarity test (18,035/22,914)

SubproblemsSubproblems

figure 6. Distribution of LT’s proofs by effortfigure 6. Distribution of LT’s proofs by effort

the limitation of LTthe limitation of LT

: a “plateau” require quite different heuristics : a “plateau” require quite different heuristics

The Subproblem TreeThe Subproblem Tree

The Subproblem TreeThe Subproblem Tree

theorem 2.17 theorem 2.17 : the proof that cost LT 89,000 primitives. : the proof that cost LT 89,000 primitives.

(not-q implies not-p) implies (p implies q) (theorem 2.17)(not-q implies not-p) implies (p implies q) (theorem 2.17)

1. 1. AA implies not-not- implies not-not-AA

2. 2. pp implies not-not- implies not-not-pp

3. (3. (AA implies implies BB) implies [() implies [(BB implies implies CC) implies () implies (AA implies implies CC)] (theorem )] (theorem

2.06)2.06)

4. (4. (pp implies not-not- implies not-not-pp) implies [(not-not-) implies [(not-not-pp implies implies qq) implies () implies (pp implies implies qq))

5. (not-not-5. (not-not-pp implies implies qq) implies () implies (pp implies implies qq) (det. 4 from 3)) (det. 4 from 3)

6. (not-6. (not-AA implies implies BB) implies (not-) implies (not-BB implies implies AA))

7. (not-7. (not-qq implies not- implies not-pp) implies (not-not-) implies (not-not-pp implies implies qq))

8. (not-8. (not-qq implies not- implies not-pp) implies () implies (pp implies implies qq) (chain 7 and 5) ) (chain 7 and 5)

The Subproblem TreeThe Subproblem Tree step 3 step 4step 3 step 4

: the difference in total effort can be attributed largely to the d: the difference in total effort can be attributed largely to the difference in number of subproblems generatedifference in number of subproblems generated

: an algorithmic procedure to govern its generation of subprob: an algorithmic procedure to govern its generation of subproblem, an algorithm to determine the order to trylem, an algorithm to determine the order to try

Modification of the Logic Theory MachineModification of the Logic Theory Machine

1. The unit cost of processing subproblems can be reduced.1. The unit cost of processing subproblems can be reduced.

2. LT can be modified to screen subproblems before they are put 2. LT can be modified to screen subproblems before they are put on the subproblem list.on the subproblem list.

3. Another way is to reduce selectively the number of subproblem3. Another way is to reduce selectively the number of subproblems generated, by limiting the lists for theorems available for gens generated, by limiting the lists for theorems available for generating subproblems.erating subproblems.

Cf) Fig. 7. A list fo 20 theorems, fig. 8. 5 theoremsCf) Fig. 7. A list fo 20 theorems, fig. 8. 5 theorems

ex) theorem (2.48) ex) theorem (2.48)

two condition: all theorems vs axiom + one theoremtwo condition: all theorems vs axiom + one theorem

the longer list : in two steps, 51,450 primitives of effortthe longer list : in two steps, 51,450 primitives of effort

the shorter list : in tree step, 18,558 primitivesthe shorter list : in tree step, 18,558 primitives

ConclusionConclusion

- Heuristics give the program power to solve - Heuristics give the program power to solve problems in reasonable computing timeproblems in reasonable computing time

- The limitations of the present program of LT- The limitations of the present program of LT

- Some of directions that improvement would have - Some of directions that improvement would have

to take to extend it powers to problems at new to take to extend it powers to problems at new

levels of difficultylevels of difficulty

The Definitions of the Sentential Calculus The Definitions of the Sentential Calculus

- - variablesvariables p, q, r,p, q, r, … A, B, C … … A, B, C …

- - connectivesconnectives not, or, implies not, or, implies

- - expressionexpression “not- “not-pp”, “”, “pp or or qq”, “”, “pp implies implies qq””

- - axiomaxiom the universally true expressionsthe universally true expressions

- - theoremtheorem

the true expressions that are derived from axioms the true expressions that are derived from axioms by means of various rules of inference by means of various rules of inference

- - rules of inferencerules of inference formalization of logical operati formalization of logical operation ( e.g. deduction)on ( e.g. deduction)

the system of axiomsthe system of axioms

( ( pp or or pp ) implies ) implies p p (1.2) (1.2)

pp implies ( implies ( qq or or pp ) (1.3) ) (1.3)

( ( pp or or q q ) implies ( ) implies ( qq or or pp ) (1.4) ) (1.4)

[ [ pp or ( or ( qq or or r r ) ] implies [ ) ] implies [ q q or ( or ( pp or or rr ) ] (1.5) ) ] (1.5)

( ( pp implies implies qq ) implies [ ( ) implies [ ( r r or or pp ) implies ( ) implies ( r r or or qq ) ] (1.6) ) ] (1.6)

three rules of inferencethree rules of inference 1. Substitution : “ A” for “ 1. Substitution : “ A” for “ p p ” , “ ” , “ pp or or qq ” for “ ” for “ pp ” ”

2. Replacement : “ 2. Replacement : “ pp implies implies q q ” with “ not-” with “ not-pp or or qq ” ”

3. Detachment : if “ A ” and “ A implies B ”, then “ B”3. Detachment : if “ A ” and “ A implies B ”, then “ B”

( ( pp implies not- implies not-pp) implies not-) implies not-pp (theorem 2.0(theorem 2.01)1)

1. ( 1. ( AA or or A A ) implies) implies A A (axiom 1.2)(axiom 1.2)

2. ( not-2. ( not-AA or not- or not-A A ) implies not-) implies not-AA

3. ( 3. ( AA implies not- implies not-A A ) implies not-) implies not-A A

4. ( 4. ( pp implies not implies not-p -p ) implies not-) implies not-pp

: LT works for about 10 seconds : LT works for about 10 seconds

not ( not ( pp or or q q ) implies not- ) implies not-p p (theorem 2.45) (theorem 2.45)

1. 1. AA implies ( implies (AA or or BB ) (theorem 2.2) ) (theorem 2.2)

2. 2. pp implies ( implies ( p p or or qq ) )

3. (3. (AA implies implies BB ) implies (not- ) implies (not-BB implies not- implies not-AA) (theorem 2.16)) (theorem 2.16)

4. [4. [ p p implies ( implies ( pp or or q q )] implies [ not ()] implies [ not (pp or or q q) implies not-) implies not-pp))

5. not ( 5. not ( pp or or q q ) implies not-) implies not-pp

works for about 12 minutes and successworks for about 12 minutes and success

[ [ pp or ( or ( qq or or rr ) ] implies [ ( ) ] implies [ ( pp or or qq ) or ) or rr ] ] (theorem 2.31)(theorem 2.31)

works for about 23 minutes and fails to prove works for about 23 minutes and fails to prove