Intelligent Tutoring Systems

Jim WarrenProfessor of Health Informatics

Outline

• Dimensions of intelligent tutoring systems (ITS)

• Examples– AutoTutor– LISP tutor

• Implications for future of learning and MOOCs

Some basics from Brusilovskiy

• Student model types– What does it hold?

• Scalar estimate of expertise (‘B+ for COMPSCI 101’)• Overlay of the domain topics

– 0/1 (or more granular) for each domain concept (‘loop structure, tick; recursion, cross’)

• Error (or ‘buggy’) model– Include common ways that student might misconceive (‘always

borrowing from leftmost digit in 3 or 4 digit subtraction’)

– Is it executable?• Can I ‘run’ the model to estimate what a student at a certain

level might say/do?

How to acquire/update thestudent model?

• Implicit– Watch how they act and infer their knowledge (‘ah, he didn’t initialise

the iterator variable; he’s not very familiar with writing loops’)• Explicit

– Ask them a question that tests a concept and rate the correctness of the answer(note that’s not the same kind of explicit approach as asking the student whether they think they know something)

• Inferred– From domain structure: ‘he doesn’t know about variables so it’s safe

to say he won’t know about arrays’– From background: ‘well, she passed COMPSCI 101, so she must know

about variables’ or ‘he’s solving these problems quickly, I’ll skip ahead’

AutoTutor

• Design with a theory of how you will influence the user– “AutoTutor adopts the educational philosophy that

students learn by actively constructing explanations and elaborations of the material”

• The dialog tactics are aimed at implementing the theory– E.g. pumps (“What else?”)– Hints, prompts, assertions– Backchannel feedback (nodding at important nouns)

AutoTutor

• https://www.youtube.com/watch?v=aPcoZPjL2G8 and Sidney D'mello and Art Graesser. 2013. AutoTutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans. Interact. Intell. Syst. 2, 4, Article 23 (January 2013), 39 pages.

Solving further problems

• Latent Semantic Analysis (LSA) allows assessment of student response– Good/bad and which of a set of expected concepts

are involved• Other refinements are addressing limitations of

the established approach– ATLAS: knowledge construction dialogs to elicit

fundamental principles– WHY2: engaging in qualitative reasoning on physics

concepts

LISP tutor• Developed way back in the early 1980s at Carnegie-

Mellon– A tutor for the (then at least) popular AI programming

language, LISP– Had an underlying rule-based system for the rules of

programming as well as how to tutor

• Interface had 3 windows– User code– Tutor feedback– Goal hierarchy (as reminder of task)

LISP tutor

Example tutor feedback

Example coding rule

Thoughts on the LISP tutor:First, the system itself needs

to be an expert

• Productions for use of LISP language– Simple rules for how to use particular functions– Higher level rules about how to tackle programming

tasks• Unclear– to what extent it could author its own solutions– how the problem is expressed

• In practice, for the tutorials the general structure of the solution appears to be pretty well spelled out for the system

Second, know how students go wrong

• LISP Tutor is a great example of a system with a strong ‘buggy rule’ component– A lot of the system development effort in buggy

rules: “325 production rules about planning and writing LISP programs and 475 buggy versions of those rules”

Third, have a model for learning (which drives the model for interaction)

• Hard to find a deep educational theory here, but dedicated to rapid feedback

• So the system is a ‘critic’– Offers support only when it detects that it’s needed

(or when asked)• This requires synchronization to be closely following the

student’s intentions• Aided by the template of the intelligent editor

– Could call it ‘mixed-initiative’• Student can request explanation• Also, system is constantly prompting student to select

options (generally only one of which is ‘correct’)

Fourth, have a lesson plan

• A curriculum of 8 progressively more challenging topics

Fifth, evaluate

• Have a meaningful control– Human tutor and ‘usual’ (on your own, or in lecture)

• Dependent measures– E.g. time to learn

• They found– 40 hours for lecture (according to poll of students)– 26.5 hours (extrapolating drop-outs) for learning on-your-own– 15 hours with LISP tutor– 11.4 hours with experience human tutors– Performance after completing recursion model: about equal

Lastly, utterly fail to realiseyour potential

• The last sentence leave us to expect results that are ‘nothing short of revolutionary’ once access to computers with 1MB RAM is commonplace

• Where are the artificially intelligent tutors?

MOOCs (Massive Open Online Courses):A good thing

• Not quite sure why it didn’t all happen immediately after LISP tutor, but… here we are

• Reasons that this might be agood level of global freeeducation service– E.g. it scales so well that it

can be a ‘public good’function of a coalition ofuniversities

An ITS research agenda for MOOCs:1. Agents in MOOC design

• “Intelligent” or otherwise– Could offer ‘critic’ functionality on design• Like Fischer’s critics, have rules for good educational

material design and point out violations• Could apply to individual exercises or to larger schema

(e.g. too much focus on one type of presentation or favouring one learning style)

2. Agents on MOOC delivery/analytics

• Lots of users provides strong indication of usage patterns– Potential to communicate patterns to the course

manager, tutor or design (for next offering) about• Little-used segments• Areas with poor performance (e.g. requiring many

attempts and being a point of drop-out)

– Maybe be able to offer diagnostic critique on the likely problem

3. Agents for delivery:modelling the user

• Learn who the users (i.e. student users) are– Background knowledge, learning style, aims

• Individual starting point for attaining specific competencies

• Individualise presentation style• Don’t necessarily need to provide the same syllabus to

everyone if aims differ– Depending on what it means in the end to have ‘completed’

the MOOC• All the usual techniques apply– Ask, directly assess, infer

4. MOOCs for assessment

• As per user model of aims– Maybe not the same test for everyone

• And can we help with the ‘cheating’ problem?– Easier if we use speech, and maybe video• E.g. detect individual prosody of speech

– Mismatch of user-modelled performance and actual performance could be at least a cue for human attention

5. Oh, and, actual intelligent tutors

• Y’know, Lisp Tutor etc.• Maybe it could be better when adding the

social MOOC aspects to the individual tutoring

Conclusion

• ITS model the user– Long-term: to guide instruction across a

curriculum– Medium-term: to assess learning achievement and

interests– Short-term: to structure dialog and provide

feedback (including back-channels)• They really should be able to revolutionize

learning with enough focused effort