Upload
louisa-preston
View
239
Download
2
Embed Size (px)
Citation preview
Introduction to the Cognitive Tutor Authoring Tools (CTAT) and Example-Tracing Tutors
Bruce McLarenSystems Scientist, Co-Manager of the CTAT Project Team
Human-Computer Interaction InstituteCarnegie Mellon University
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Kinds of Computer Tutors
Intelligent tutoring systems
e.g., Sherlock Model-tracing tutors
e.g., Andes Cognitive tutors
e.g., Algebra
Tutoring systems
CAI e.g., Microsoft’s PersonalTutor
Constraint-based tutors
e.g., SQL Tutor
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Cognitive Tutors: Real-world success of Intelligent Tutoring Systems technology
• Cognitive Tutor Courses– Computer-based one-to-one tutoring
• Based on computational models of student thinking• Based on the concept of Model Tracing
• Used by many students in many schools– Algebra Cognitive Tutor: In over 2,000 schools in the USA,
300,000 students per year– Geometry Cognitive Tutor: In 350 schools– Most widely used intelligent tutoring system
• University created a company, Carnegie Learning, to disseminate
Algebra Cognitive Tutor Sample
Use graphs, graphics calculator
Analyze real world problem scenarios
Use table, spreadsheet
Use equations, symbolic calculator
Tutor learns about each student
Tutor follows along, provides context-sensitive Instruction
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
ACT-R: A Cognitive Theory of Learning and Performance
• Big theory … key tenets:– Learning by doing, not by listening or watching
– Production rules represent performance knowledge:
These units are: Instruction implications:• modular • context specific
isolate skills, concepts, strategiesaddress "when" as well as "how"
Anderson, J.R., & Lebiere, C. (1998). The Atomic Components of Thought. Erlbaum.
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
• Cognitive Model: A system that can solve problems in the various ways students can
Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d
Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a
Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d
Cognitive Tutor Technology:Use ACT-R theory to individualize instruction
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology:Use ACT-R theory to individualize instruction• Cognitive Model: A system that can solve problems in
the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d
If goal is solve a(bx+c) = dThen rewrite as abx + c = d
If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology:Use ACT-R theory to individualize instruction• Cognitive Model: A system that can solve problems in
the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d
If goal is solve a(bx+c) = dThen rewrite as abx + c = d
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
Hint message: “Distribute a across the parentheses.”
Bug message: “You need tomultiply c by a also.”
• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing
Known? = 85% chance Known? = 45%
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
CTAT motivation: Make tutor development easier and faster!• Development costs of instructional technology are, in
general, quite high– E.g., ~300 dev hours per hour of instruction for Computer Aided
Instruction (Murray, 1999)
• Cognitive Tutors: – Large student learning gains as a result of detailed cognitive
modeling– ~200 dev hours per hour of instruction (Koedinger et al, 1997)– Requires PhD level cog scientists and AI programmers
• Solution: Easy to use Cognitive Tutor Authoring Tools (CTAT)
Murray, T. (1999). Authoring Intelligent Tutoring Systems: An Analysis of the state of the art. The International Journal of Artificial Intelligence in Education, 10, 98-129.
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big
city. The International Journal of Artificial Intelligence in Education, 8, 30-43.
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
How to reduce the Authoring Cost?
• Less programming, more automation– Drag & drop interface construction– Demonstration-based programming
• Human-Computer Interaction methods– User studies, summer schools, informal & formal
comparison studies
• Exploit tools already in use– Component-based architecture & standard inter-
process communication protocols
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Authoring with CTAT• Cognitive Tutors
– Difficult to build; for programmers– General for a class of problems
• Example-Tracing Tutors– Much easier to build; for non-programmers– Limited to a single problem instance
• New: “Mass Production” feature greatly facilitates authoring of multiple problem instances (in Excel)
• Typical Approach– Build Example-Tracing Tutor first
• Cognitive Task Analysis• Rapid prototypes - perhaps sufficient for the problem
– Develop Cognitive Tutor from Example-Tracing Tutors• Examples guide planning• Serve as semi-automated test cases
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Where does CTAT fit into the Realm of Authoring Tools?
Murray survey (1999) discusses 7 categories of authoring tools for intelligent tutoring systems (ITS), including:
• Device Simulation - RIDES (Munro et al 1997)• Tutoring Strategies - REDEEM (Ainsworth et al 2003;
Major et al 1997)• Domain Expert System - Demonstr8 (Blessing 1997)
Cognitive Tutor Authoring Tools:• “Domain Expert System” category of authoring systems• No commercially successful tools for authoring model-
tracing, Cognitive Tutors• First to provide functionality to simplify & speed cog.
tutor development
Behavior Recorder
Data Shop
CTAT’s Modular Architecture
Student Interface(or external problem-solving environment)
Cool Modes
CyclePad
Java Swing
Flash
Learner Management System
LMS
Tutor Shop
Cognitive Model Development Tools
TDK (Lisp-based)
Jess tools (Java-based)
Editor
Eclipse
Tutor Engine
Tertle (Lisp-based) + Model Tracer
Jess (Java-based) + Model Tracer
Example Tracer (Java-based)
Example Tracer (Flash-based)
GUI Builder
IntelliJ
Dreamweaver
Code Warrior
Netbeans
Flash MX 2004
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
CTAT’s Track Record• CTAT-based tutors have been used in
experiments in Geometry, Chemistry, Chinese, and French
• Over 200 users
• Three releases in past year (1.4, 1.5, 1.6)
• 13 papers published during 2005-2006
• CTAT Web site– 36,000 unique visitors in 2005– 8,800 so far in 2006
• CTAT Downloads– 840 in 2005 (27% CMU & Pitt)– 952 so far in 2006 (4.6% CMU & Pitt)
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
http://ctat.pact.cs.cmu.edu
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
PSLC Example-Tracing Tutors built with CTAT
Chinese - tone study
Chinese - listen and readFrench culture
FrenchChemistry (stoichiometry)
Geometry (self-assessment) Geometry (on-line testing)
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
PSLC Case study - Chemistry (Stoichiometry)
• Goals:
o Test hypotheses about personalization and worked examples
o Exercise and test the Cognitive Tutor Authoring Tools (CTAT) in a live, web-based experiment
• Results:
o Students learned: Significant difference between pre and post
o No sig differences due to personalization or worked examples
McLaren, B. M. et al. (2006). Studying the Effects of Personalized Language and Worked Examples in the Context of a Web-Based Intelligent Tutor. In the Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, June 26-30, 2006.
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
PSLC Case study - Chemistry (Stoichiometry)
Behavior Recorder
Flash authoringenvironment with
tutor interface shown
Web browser withExample-Tracing tutor
embedded in HTML
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
CTAT Features used in Case Study • Web delivery
– Use of Macromedia Flash MX 2004 to build Student Interface
– Tutor Shop handles problem sequencing on the web
• Template-based tutor development– Use of Excel lowers skill threshold even more– More efficient authoring– Easier maintenance
• Support for experiments– Pre/post tests adminstered and graded automatically– CTAT-built tutors log all student-tutor interactions– PSLC Data Shop provides storage and data analysis
facilities
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Development time comparison• Past estimates of ITS development
200:1 for robust beta versions used in real classrooms
• Example-Tracing Tutor Development25:1 for initial alpha versions: (Koedinger et al, 2004)
# Of PseudoTutors
DesignTime
Dev.Time
InstructionalTime
Design/Devto Instr.
Economics 11 3600 2190 180 32.2
Math Assistments 20 810 1170 98 20.2
LSAT 3 240 3000 180 18.0
Language Learning 8 210 575 50 15.7
Totals 4860 6935 508 23.2
• If creating a robust beta doubles dev time, we would have ~50:1 ratio -- a reduction of 4 times!
Koedinger, K., Aleven, V., Heffernan, N., McLaren, B. M., and Hockenberry, M (2004). Opening the Door to Non-Programmers: Authoring Intelligent Tutor Behavior by Demonstration;. In the Proceedings of the Seventh International Conference on Intelligent Tutoring Systems (ITS-2004).
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
CTAT Example-Tracing Tutor Web-Delivery Options
• Java– WebStart– Utility may depend on the situation– We are still gaining experience with this; can be a bit
tricky
• Flash (from Macromedia)– With plug-in, compatible with virtually all browsers– Easy to deploy
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Flash or Java for Authoring and Delivery?
Cost Web Delivery
Look-and-Feel of Student Interface
Maturity Availability ofDevelopers
Flash - License must be purchased from Macromedia(Academic: $100, $150, for authoring only)
+ More compatible across platforms & browsers
+ Emphasis on presentation; Easy to include movies, graphics, other visuals (Currently 6 widgets)
-First released as part of CTAT in Feb 2005; Much newer than Java
- Less established programming environment, so less programmers available
Java + NetBeans and Java are free downloads
- Less compatible; Use of apps like WebStart for web delivery
- Visual elements not as crisp; more difficult to include other media (Currently 15+ widgets)
+ Available for over 2 1/2 years
+ More established programming language, so more programmers available
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Movie Showing How an Example-Tracing Tutor is built
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
Hands-On Assignment: Build your own fraction addition Example-Tracing Tutor• To get started:
– Start Flash MX2004 (From Start menu: All Programs:Programming Tools:MX2004)
– Start the Behavior Recorder (Double click the “Behavior Recorder for Flash” icon on the desktop)
– In Flash open the partially built fraction-addition GUI (W:/Cognitive Tutor Authoring Tools/Projects/Examples/Exercise1-ExampleTracingTutor/fractionAddition.fla)
• To finish the GUI: Follow the hand-out “Building a student interface for fraction addition” (Because you have a partially built user interface, you can just review the first three pages; begin work in the middle of page 4)
• To build the Example-Tracing Tutor: Follow the hand-out: “Building a student interface for fraction addition”
PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006
THE END (Of “Introduction to CTAT and Example-Tracing Tutors”)