1 Cognitive Tutor Development Kit (TDK) Tutorial Cognitive Modeling & Intelligent Tutoring Systems Ken Koedinger Vincent Aleven

1

Cognitive Tutor Development Kit (TDK) Tutorial

Cognitive Modeling & Intelligent Tutoring Systems

Ken KoedingerVincent Aleven

2

Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model

tracing• Control flow and production chains• Writing hints and bug productions• Creating a curriculum

3

Production Rules in the Tutor Development Kit (TDK)

• How do you go about writing a production rule?

• The TDK Production System– Working Memory:

• Make up of “Working Memory Elements” (WMEs)

– Production Rules– Interpreter (Match, Conflict Resolution, Fire)

4

Implementing a Production Rule Model in TDK

• Simple example: a model for single-column addition without carrying!

• How would you define:– Working Memory representation for the

problem states– Production rules that transform working

memory

4+ 3 7

4+ 3

4+ 3 7 Done

5

A Notation for Working Memory Elements (“WMEs”) in TDK

PROBLEM4+3> ISA SINGLE-COLUMN-ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil

4+ 3 WME Name

Slot Values

Slots

WME TypeSeparator

6

Working Memory Transitions

PROBLEM4+3> ISA SINGLE-COLUMN-

ADDITION-PROBLEM

FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil

4+ 3 7

4+ 3

4+ 3 7 Done


ADDITION-PROBLEM

FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE nil


ADDITION-PROBLEM

FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE T

Production: ADD Production: DONE

7

TDK Production Rule Notation(defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2

==>

!eval! =sum (+ =num1 =num2)

=problem> result =sum)

ADD (English version)IFThe goal is to do =problem, a

single-column addition problemAnd no result has been found yetAnd the first addend is =num1And the second added is =num2THENSet =sum to the sum of

=num1 and =num2

Write =sum as the result

8

TDK Production Rule Notation(defproduction add single-column-addition (=problem)

=problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2

==>



If-part

Then-part

9



==>



If-part

Then-part

WME pattern

WME pattern

Computation in Lisp

10



==>



WME pattern

Variables(“bound” to values through matching) Separator

11

Matching a Production Rule to Working Memory—Find Values for Each Variable

Working Memoryproblem4+3> isa single-column- addition-

problem first-addend 4 second-addend 3 result NIL done NIL

Find value for each variable

Variable=problem=num1=num2=sum

ValueProblem4+3437

Match!

Modify WME

problem4+3> isa single-column-

addition-problem first-addend 4 second-addend 3 result 7 done NIL

Production Rule

add =problem> isa single-column-

addition-problem result NIL first-addend =num1 second-addend =num2==>!eval! =sum (+ =num1 =num2)=problem> result =sum)

This WME pattern in the THEN-part means change slot value of existing WME

What changes are made to Working Memory?

12

Summary—Components of a TDK Production Rule Model

• Working Memory is a collection of WMEswme-name>

isa wme-typeslot1 value1slot2 value2

• Production rules specify working memory transitions– WME patterns in the IF-part are matched against

Working Memory=wme-variable>

isa wme-typeslot1 value-pattern1slot2 value-pattern2

– WME patterns in the THEN-part specify changes to Working Memory

13



14

WME Representation

26+ 51

26+ 51 7

26+ 51 77

26+ 51 77 Done

Value of a slot in a WME can be a list.

Value of a slot in a WME can be a list.

Initial StatePROBLEM26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN2 COLUMN1) DONE nilCOLUMN1> ISA COLUMN FIRST-ADDEND 6 SECOND-ADDEND 1 RESULT nilCOLUMN2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT nil

Final StatePROBLEM26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN2 COLUMN1) DONE TCOLUMN1> ISA COLUMN FIRST-ADDEND 6 SECOND-ADDEND 1 RESULT 7COLUMN2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT 7

15

Production Rules Set New Goals & Perform Actions

FOCUS-ON-FIRST-COLUMN,FOCUS-ON-NEXT-COLUMN

ADD

26+ 51 77

Goal: Solve the addition problem

Goal: Process column C

Action: Write the sum

Action: Mark as done

DONE

16

Goals in Production Rules

FOCUS-ON-FIRST-COLUMNIF The goal is to do a multi-column

addition problem And C is the rightmost column of

the problem And there are numbers to add in C

And there is no result yet in CTHEN Set a subgoal to process column C

FOCUS-ON-NEXT-COLUMNIF The goal is to do a multi-column

addition problem And C is the rightmost column with

numbers to add and no resultTHEN Set a subgoal to process column C

WRITE-RESULTIF There is a goal to process column CTHEN Set Sum to the sum of the

addends in column C Write Sum as the result in column C

DONEIF The goal is to do a multi-column

addition problem And there is no column left with

numbers to add and no result

THEN Mark the problem as done

17

Figuring Out Which ColumnFOCUS-ON-FIRST-COLUMNIF The goal is to do a multi-column

addition problem And C is the rightmost column of

the problem And there are numbers to add in C

And there is no result yet in CTHEN Set a subgoal to process column C

FOCUS-ON-NEXT-COLUMNIF The goal is to do a multi-column

addition problem And C is the rightmost column with

numbers to add and no resultTHEN Set a subgoal to process column C

WRITE-RESULTIF There is a goal to process column CTHEN Set Sum to the sum of the

addends in column C Write Sum as the result in column C

DONEIF The goal is to do a multi-column

addition problem And there is no column left with

numbers to add and no result

THEN Mark the problem as done

Example Production Rule(defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition

(=process-column-goal))

FOCUS-ON-NEXT-COLUMNIFThe goal is to do =problem, a multi-column

addition problemAnd =next-column and =previous-column are

columns next to each otherAnd there is a result in =previous-columnAnd there is no result in =next-column And =num1 is the 1st addend in =next-column And =num2 is the 2nd addend in =next-column And there is no result yet in =next-column And =num1 or =num2 is not blankTHENSet a subgoal to process =next-column

TDK Production Rule Notation(defproduction focus-on-next-column multi-column-addition

(=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal))

If-part

Then-partWME pattern

WME pattern

WME pattern

WME pattern



Variablesmultiple occurrences of same variable in IF-part must match the same value



Variables within a list whose name starts with $ match any number of items (0 or more).

22

Matching a Production Rule to Working Memory—Find Values for Each Variable

(1)Production Rulefocus-on-next-column =problem> isa multi-column-

addition-problem columns

($ =next-col =prev-col $) =prev-col> isa column - result NIL =next-col> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-col

Working Memoryproblem26+51> isa multi-column-

addition-problem columns (column2 column1) done nilcolumn1> isa column first-addend 6 second-addend 1 result 7column2> isa column first-addend 2 second-addend 5 result nil

Find value for each variable

Variable=problem1st $=next-col=prev-col2nd $=num1=num2

Valueproblem26+51()column2column1()61

Match!

Match!

Create new WME<process-column-goal 0> isa process-column-goal column column2

What changes are made to Working Memory?

When a WME pattern with “isa” is in the THEN-part of a rule, it means create a new WME rather than modify an existing one.

23

Value to be matched: (COLUMN4 COLUMN3 COLUMN2 COLUMN1) Pattern: ($ =next-column =previous-column $)

Match 1Left $ ()=next-column COLUMN4=previous-column COLUMN3Right $ (COLUMN2 COLUMN1)

Match 2Left $ (COLUMN4)=next-column COLUMN3=previous-column COLUMN2Right $ (COLUMN1)

Match 3Left $ (COLUMN4 COLUMN3)=next-column COLUMN2=previous-column COLUMN1Right $ ()

Pattern Matching with Lists—Patterns with$ variable(s) often create multiple matches

Common pattern to extract the next element of a list.

24

TDK Production Rule Notation

• “==>” separates if-part & then-part• If- & then-parts are made up of WME patterns of form:

=wme-variable>isa wme-typeslot1 value-pattern1slot2 value-pattern2

• An atom starting with either “=” or “$” is a variable– Ex: “=num” “$columns”– “=” vars match atoms– “$” variables match 0 or more elements of a list– “=” or “$” by itself is just an unnamed (dummy)

variable

25

The TDK Production System Interpreter

• Match phase:– Attempt to match each WME pattern with 1 or more WMEs in

working memory– Variables are “bound” to a value. Subsequent references to those

variables must have the same value.– A set of “bindings” for all variables in the if-part is called a “instance”

of a production.– If a consistent set of bindings is found, the production “matches”,

otherwise it “fails”.– A production can match working memory in more than one way, that

is, it can have multiple instances (or rule instantiations).

• Conflict resolution phase: – In stand-alone mode: Determined by hand or arbitrarily– In tutor mode: Production priorities & model tracing heuristics

• Act phase (fire a production):– Use the variable bindings to modify or create the WME indicated in

the then-part.– If “isa” slot is present, create a WME, otherwise modify.

26



27

Modeling Alternative Strategies

• Consider multiple solution paths– Reordering of steps

• Write sum first or write carry in next column first

– Alternative strategies• Left to right strategy• Use calculator and write result strategy

– Shortcuts• Implicit carrying

• Which of these will the model cover?• Motivate choices pedagogically!

28

Execution Space Diagram Illustrating Flexibility

1 264 + 716 80

1 264 + 716 0

264 + 716

1 264 + 716 980

264 + 716 0

1 264 + 716

29



ADD-CARRY

264 + 716FOCUS-ON-FIRST-COLUMN,

FOCUS-ON-NEXT-COLUMN


Goal: Write carry in next column

MUST-CARRY

WRITE-SUM


Action: Write the carry

WRITE-CARRY

Goal: Write sum in column C

ADD-ADDENDS

30

Model Tracing 1 — Using a Cognitive Model to Evaluate Students’ Solution Steps

• Cognitive model is an expert system that– is psychologically plausible– accounts for all reasonable solutions– accounts for commonly occurring errors (“bugs”)

• To verify solution step by student– Use model to generate correct and buggy next steps– If student step matches

• Correct step: Accept– update Working Memory to reflect new problem state

• Buggy step: Give error-specific feedback• No step: Reject without feedback

31



32

Communicating About Student Solution Steps between Interface and Tutor

• Represent each observable action in the user interface as a selection-action-input triple.

Selection = (column2)Action = write-carryInput = <whatever student typed>

Selection = (column1)Action = write-sumInput = <whatever student typed>

33

Execution Space Diagram with Selection-Action-Input Triples

1 264+ 716 80

1 264+ 716 0

264+ 716

1 264+ 716 980

264+ 716 0

1 264+ 716

Selection: Column1Action: Write-ResultInput: 0



Selection: Column2Action: Write-CarryInput: 1


Selection: Column2Action: Write-CarryInput: 1

1 264+ 716 980 Done.

Selection: -Action: DoneInput: -

34

Interface Constraints—Matching Student Steps Against Model Steps

• Production rule writer can attach “model-tracing” or “interface constraints” to rules.– Specify tests on the selection-action-input

triples.

• During model-tracing TDK will not consider productions whose constraints are violated.

35

CONSTRAINTS—consider rule only if:Selection is:Action is:Input is:

Example of Interface Constraints in English

WRITE-SUMIF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been addedTHEN Write Sum as the result in column C And remove the goal

Same as value of variable CWrite-ResultEqual to the value of variable Sum

Q: Do we really need a constraint on the Selection AND a constraint on the Action AND a constraint on the Input? What if we left out one?

A: Yes, each is necessary. Without a constraint on the Selection, the rule will accept the student step if the number typed is correct, regardless of the column in which it was typed. Without a constraint on the Action, the the rule will accept the step if the right number was typed in the right column, regardless of whether it was in the carry spot or at the bottom of the column. Etc.

36

Example of Interface Constraints in English

WRITE-SUMIF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been addedTHEN Write Sum as the result in column C And remove the goal

CONSTRAINTS—student step matches only if:Selection is:Action is:Input is:

Same as value of variable CWrite-ResultEqual to the value of variable Sum

37

More Detail on Interface Constraints

• In Tutor Mode– When evaluating student input

• TDK will not consider productions whose constraints are violated.

– When responding to a help request• Tries to find the production path with the least

severe constraint violation.– from least to most severe: input, action, selection.

• In Stand-Alone Mode, interface constraints are ignored

• Convenient for development—can run model without having to supply input.

TDK Notation for Interface Constraints(defproduction write-sum addition (=problem) =problem>

isa addition-problem . . . =subgoal> isa write-sum-goal . . .==>

=column>

result =sum =problem> subgoals ($sg1 $sg2) ; the remaining subgoals

:nth-selection 0 =column :action 'write-result :input =sum #'equal-value-p)

See Example 4 from the Examples menu.

The first item of Selections must be equal to the value of =column.

The Action must be write-result.

The value of Input must be equal to the value of =sum. Use function equal-value-p to compare.

39

Syntax of TDK Notation for Interface Constraints

:nth-selection n value [test-fn]:action value [test-fn]:input value [test-fn]

40


tracing• Control flow and production

chains• Writing hints and bug productions• Creating a curriculum

41

Creating a Control Flow Diagram

• Control Flow Diagram elements:– Boxes are goals or actions– Arrows are production rules– A “path” or “chain” through the diagram starts at a

goal and ends with an action

• Crucial idea:– Each chain should produce exactly one action– When you “cycle” the TDK production system a “chain”

of productions fire

42



ADD-CARRY





MUST-CARRY

WRITE-SUM



WRITE-CARRY


ADD-ADDENDS

43

(defproduction focus-on-next-column multi-column-addition


Chaining of Production Rules—Multiple Productions per Step

(defproduction write-result multi-column-addition (=subgoal)

=subgoal> isa process-column-goal column =column =column> isa column first-addend =num1 second-addend =num2 ==> !eval! =sum (+ (or =num1 0) (or =num2 0)) =column> result =sum)

Parameters

!chain! specifies the production set and parameters and initiates another attempt to match productions to this parameter & WM.

Name of the production rule set

No !chain! so this rule ends the chain.

44

Important Design Principle

• Design your production rule model so that there is exactly one cycle per observable (student) action.

45



46

Example Hint Sequences for Addition Tutor

Start with the column on the right. This is the "ones" column.You need to add the two digits in this column. Adding 4 and 6 gives 10.The sum that you have, 10, is greater than 9. You need to 10 of the 10 into the

second column. And you need to write the rest of the 10 at the bottom of the first column.

Write 0 at the bottom of the rightmost column.

You need to complete the work on the first column.Write the carry from the first to the next column.Write 1 at the top of the second column from the right.

Now move on to the second column from the right. This is the "tens" column.You need to add the two digits in this column. Adding 6 and 1 gives 7.There is a carry into this column, so you need to add the value carried in. This gives

7 + 1 equals 8.Write 8 at the bottom of the second column from the right

264+ 716

264+ 716 0

1 264+ 716

47

Design of Hint Messages

– What makes a good hint sequence?• Short• Clear (minimum of jargon, short sentences)• Tells student what to do• But also why/when that action is appropriate• Goes from the general to the specific

– Writing hints can help in designing the production rule model.

– Hint sequences often combine hints from multiple productions in a chain.

48

(defproduction focus-on-next-column addition (=problem) =problem> isa addition-problem . . . =previous-column> isa column . . . name =name position =pos !eval! (or =num1 =num2 =carry) ==> . . . !chain! addition (=problem)

:messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.)))

Example Hint Template

Attach list of hint templates to the rule.

49

• Example hints1. Now move on the the second column from the right.

This is the “tenths” column.

2. Next big event: the second column from the right!!!

• Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.)

`(Next big event #\: the #\space ,=pos column from the right #\! #\! #\! )

)

2nd Example Hint Template

50






)


51






)


52

• Example hints from this template1. Now move on the the second column from the right.





)


For punctuation and spaces, use #\ notation.

To insert value of variable: comma followed by variable name.

~n produces new line.

Backquote.

When inserting a string, put a space in front of it.

53

Sequences Combining Hints from Multiple Productions

Start with the column on the right. This is the "ones" column.You need to add the two digits in this column. Adding 4 and 6

gives 10.The sum that you have, 10, is greater than 9. You need to

move 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column.

Write 0 at the bottom of the rightmost column.

You need to complete the work on the first column.Write the carry from the first to the next column.Write 1 at the top of the second column from the right.

Now move on to the second column from the right. This is the "tens" column.

You need to add the two digits in this column. Adding 6 and 1 gives 7.

There is a carry into this column, so you need to add the value carried in. This gives 7 + 1 equals 8.

Write 8 at the bottom of the second column from the right

264+ 716

264+ 716 0

1 264+ 716

FOCUS-ON-FIRST-COLUMN

ADD-ADDENDS

MUST-CARRY

WRITE-SUM

WRITE-CARRY


ADD-ADDENDS

ADD-CARRY

WRITE-SUM

54

How could you model students who don’t carry?

• Instead of doing the addition correctly:

• Can you model a student who writes:

• Can you write a “bug rule” that models this error?

1 264+ 716 980

264+ 716 970

55



ADD-CARRY





MUST-CARRY

WRITE-SUM



WRITE-CARRY


ADD-ADDENDS

56

How to come up with a bug rule?

• Intuition: Need to model the “forgetting” of the subgoal to carry into the next column.

• Normal processing:

• If you then

• You will pretty soon be in situation:

264+ 716

Subgoal 1: write-sum-goal write 0 as result in the first column.Subgoal 2: write-carry-goal write 1 as carry in second column

Forget Subgoal 2, and continue as if nothing has happened

264+ 716 70

Solution 1: Explicit Forgetting Rule;; BUG-FORGOT-WRITE-CARRY-GOAL;; IF The goal is to do an addition problem;; And there is a pending subgoal to write a carry;; THEN Delete that pending subgoal (w/o carrying it out).;; (And carry on with the addition problem in the usual manner.)

(defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) ; Remove the write-carry-goal. !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) ))

Use defproduction-bug instead of defproduction.

Use :messages (bug … ) instead of :messages (help … )

;; BUG-FORGOT-WRITE-CARRY-GOAL;; IF The goal is to do an addition problem;; And there is a pending subgoal to write a carry;; THEN Delete that pending subgoal (w/o carrying it out).;; (And carry on with the addition problem in the;; usual manner.)

(defproduction-bug BUG-FORGOT-WRITE-CARRY-GOALaddition (=problem)

=problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) ))

Solution 1: Explicit Forgetting RuleSome questions & details•Why does the rule chain?•Why does the rule fire only when the student forgets to add a carry, and not for any wrong answer?•Does this work when action “done” follows after 97+42=39?•Does this rule fire under different circumstances than expected?•YES - after MUST-CARRY in the same cycle. Not intended but not a problem.

Some questions & details•Why does the rule chain?•Why does the rule fire only when the student forgets to add a carry, and not for any wrong answer?•Does this work when action “done” follows after 97+42=39?•Does this rule fire under different circumstances than expected?•YES - after MUST-CARRY in the same cycle. Not intended but not a problem.

59



60

Setting Up a Curriculum

• Organize curriculum early on during the development of your tutoring system.

• Organize the tutor curriculum– Sequencing of skills– Break down curriculum into lessons and

sections– Each section involves a handful of “skills”– Assign problem sets to each section

61

Setting Up Knowledge Tracing and Mastery Learning• Indicate how to group the crucial

productions into traced skills.– Each traced skill corresponds to one or more

productions.

• For each skill, provide values for the four knowledge tracing parameters.

• For each problem, indicate which productions are needed to solve it.– Enables tutor to select problems that

involve un-mastered skills.

Example: Addition Skills• Start — Start in the rightmost column

FOCUS-ON-FIRST-COLUMN

• Add — Add the addends in a columnADD-ADDENDS

• Write — Write the sum in a columnWRITE-SUM ADDITION

• Done — Determine that all columns are doneDONE ADDITION

• Next — Focus on the next columnFOCUS-ON-NEXT-COLUMN

• Regroup — Realize that 10 goes to next columnMUST-CARRY

• Write Carry — Write the carry

WRITE-CARRY • Add Carry — Add the carry to the sum

ADD-CARRY ADDITION

63

Example from Addition Tutor• Structure curriculum, based on skills

Section1 — one column, no carrynew skills: Start, Add, Write, Done

Section2 — two columns, no carrynew skills: Next

Section3 — two columns, possible carrynew skills: Regroup, Mark Carry, Add Carry

Section4 — three columns, possible carrynew skills: -

64

Excerpt from Lesson File

(Make-Section :Required-Problems '(Problem11 Problem12 Problem15) :Remedial-Problems '(Problem13 Problem14 Problem16 Problem17 Problem18 Problem19) :Skills '(((Focus-On-First-Column Addition) :Description "Start") ((Done Addition) :Description "Done") ((Add-Addends Addition) :Description "Add") ((Write-Sum Addition ) :Description "Write") ))

65

Set Knowledge-Tracing Parameters

Parameter(setf *p-initial* 0.33)

(setf *p-learn* 0.20)

(setf *p-guess* 0.20)

(setf *p-slip* 0.10)

(setf *p-known-p* 0.95)

Represents probability that

the student has learned the skill prior to the instruction

the student learns the skill when successfully applying it

the student gets the answer right even though the skill is not mastered

the student gets the answer wrong even though the skill is mastered

Represents mastery threshold

66

Provide Production Counts

For problem

(or any other 2-column problem without carrying)

(set-problem-production-counts problem26+51 '(((FOCUS-ON-FIRST-COLUMN ADDITION) . 1) ((FOCUS-ON-NEXT-COLUMN ADDITION) . 1) ((ADD-ADDENDS ADDITION) . 2) ((WRITE-SUM ADDITION) . 2) ((DONE ADDITION) . 1) ))))

26+ 51 77

67

Collecting Logs ofStudent-Tutor Interactions in

TDK• Create “protocol collectors” when

starting tutor (make-instance 'global protocol-collector :broadcast-station broadcast-station) (make-instance 'protocol-collector :broadcast-station broadcast-station) ; add to function ; make-addition-tutor

• Set directory parameters(setf *global-protocol-directory* "Home:Globals;")(setf *protocol-directory* "Home:Protocols;")

• Flip the switch(setf *collect-protocols* t)

Example Protocol from Addition Tutor

*000:00:01 START STUDENT; "Student3", USER-DATA; NIL, PROBLEM; PROBLEM15, NPRESENTED; (0 0), START-TIME; "07/18/01 21:57", FILE-NAME; "Macintosh HD:Desktop

Folder:Vincent:CIRCLE Summer School:TDK:Protocols:LESSON1:Student3:1%PROBLEM15%0.p-0",

LESSON; LESSON1, LESSON-PRINT-NAME; "", SECTION; SECTION155, SECTION-PRINT-NAME; "",.*000:00:24 INITIAL-PROBABILITIES p/p-1; (0.6762857412475228 (ADD-ADDENDS

ADDITION)), p/p-2; (0.9451767563766541 (DONE

ADDITION)), p/p-3; (0.6762857412475228 (FOCUS-ON-

FIRST-COLUMN ADDITION)), p/p-4; (0.6762857412475228 (WRITE-SUM

ADDITION)),.*000:00:40 FOCUS SELECTED-WMES; (COLUMN1),.*000:14:926 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; (COLUMN1), ACTION; WRITE-RESULT, INPUT; "5",.

*000:14:939 UPDATE-P-KNOW META; NIL, PRODUCTION; (FOCUS-ON-FIRST-COLUMN

ADDITION), SUCCESS?; NIL, P-KNOW; 0.3656548145323001,.*000:14:945 UPDATE-P-KNOW META; NIL, PRODUCTION; (ADD-ADDENDS ADDITION), SUCCESS?; NIL, P-KNOW; 0.3656548145323001,.*000:14:952 UPDATE-P-KNOW META; NIL, PRODUCTION; (WRITE-SUM ADDITION), SUCCESS?; NIL, P-KNOW; 0.3656548145323001,.*000:14:958 NO-CONFLICT-SET .*000:14:962 READY .*000:17:41 SUGGESTED-SELECTION ITEMS; (WRITE-RESULT COLUMN1),.*000:17:46 HELP SELECTION; (COLUMN1), NHINT; 0,.#000:17:50 HELP-TEXT You need to add the two digits in this column.

Adding 3 and 3 gives 6..

*000:17:53 HELP-PATH P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),.*000:17:68 READY .*000:19:246 SUGGESTED-SELECTION ITEMS; (WRITE-RESULT COLUMN1),.*000:19:252 HELP SELECTION; (COLUMN1), NHINT; 1,.#000:19:256 HELP-TEXT Write 6 at the bottom of the rightmost column..*000:19:259 HELP-PATH P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),.*000:19:270 READY .*000:21:94 WINDOW-SELECT WINDOW-TITLE; ADDITION-WINDOW,.*000:23:765 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; (COLUMN1), ACTION; WRITE-RESULT, INPUT; "6",.

*000:23:774 GOOD-PATH .*000:23:778 HISTORY P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),.*000:23:782 READY .*000:25:84 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; NIL, ACTION; DONE, INPUT; NIL,.*000:25:94 UPDATE-P-KNOW META; NIL, PRODUCTION; (DONE ADDITION), SUCCESS?; T, P-KNOW; 0.9898195480465753,.*000:25:99 GOOD-PATH .*000:25:102 HISTORY P-1; (DONE ADDITION),.*000:25:106 READY .*000:25:109 PROBLEM-COMPLETED .*000:25:113 FINAL-PROBABILITIES p/p-1; (0.3656548145323001 (ADD-ADDENDS

ADDITION)), p/p-2; (0.9898195480465753 (DONE ADDITION)), p/p-3; (0.3656548145323001 (FOCUS-ON-FIRST-COLUMN

ADDITION)), p/p-4; (0.3656548145323001 (WRITE-SUM ADDITION)),.

69

Example Uses of Log Data• To improve model and tutor

– Is skill decomposition accurate?• If “learning curves” for skills do not follow power law,

crucial difficulty factors may have been overlooked.

– Tune knowledge tracing parameters• Fit the parameters to the log data, use the new

parameter values next time around.

– What kind of errors do students make?• Any use for new bug messages?

– Do we have enough different problems?

• For research– Relation between help use and learning outcomes.– Compare learning rates in comparative studies.

70

END

Documents

1 Cognitive Tutor Development Kit (TDK) Tutorial Cognitive Modeling & Intelligent Tutoring Systems Ken Koedinger Vincent Aleven