Upload
clifton-long
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Educational data mining overview & Introduction to Exploratory Data Analysis with DataShop
Ken Koedinger CMU Director of PSLC
Professor of Human-Computer Interaction & Psychology
Carnegie Mellon University
Overview
DataShop Overview Logging model DataShop Features
Quantitative models of learning curves Power law, logistic regression Contrasting KC models
Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing
Logging & Storage Models
Education technologies are “instrumented” to produce log data
We encourage a standard log format XML format generalized from Ritter & Koedinger
(1995) Also convert log data from other formats
Example activity generating “click stream” data
Geometry Cognitive Tutor: “Making Cans” problem Find the area of scrap metal left over after removing a circular area (the end of a can) from a metal
square. Student enters values in worksheet
Tutor provides feedback & instruction Records student’s actions & tutor responses
Logs stored in files on school server or database at Carnegie Learning Later imported into DataShop
DataShop logging model
Main constructs: Context message: the student, problem, and
session with the tutor Tool message: represents an action in the tool
performed by a student or tutor Tutor message: represents a tutor’s response to a
student action
DataShop XML format: Context message<context_message context_message_id="C2badca9c5c:-7fe5" name="START_PROBLEM"> <dataset> <name>Geometry Hampton 2005-2006</name> <level type="Lesson"> <name>PACT-AREA</name> <level type="Section"> <name>PACT-AREA-6</name> <problem> <name>MAKING-CANS</name> </problem> </level> </level> </dataset></context_message>
Dataset name
Course unit
Course section
Problem
DataShop XML format: Tool & Tutor Messages
<tool_message context_message_id="C2badca9c5c:-7fe5"> <semantic_event transaction_id="T2a9c5c:-7fe7" name="ATTEMPT" /> <event_descriptor> <selection>(POG-AREA QUESTION2)</selection> <action>INPUT-CELL-VALUE</action> <input>200.96</input> </event_descriptor></tool_message><tutor_message context_message_id="C2badca9c5c:-7fe5"> <semantic_event transaction_id="T2a9c5c:-7fe7" name="RESULT" /> <event_descriptor> … [as above] … </event_descriptor> <action_evaluation>CORRECT</action_evaluation></tutor_message>
Example Stored Transactions Student interactions (or transactions) are stored in a relational
database, can be exported as table Example: Student S01 on Making-Cans problem
Transactions
Info for each transaction student(s), session, time, problem, problem step,
attempt number, student action tutor response, number of hints, knowledge
component code Logging of on-line tools (e.g., a virtual lab)
does not include tutor response
Step & Transaction Definitions
A problem-solving activity typically involves many tool & tutor messages.
“Steps” represent completion of possible subgoals or pieces of a problem solution
“Transactions” are attempts at a step or requests for instructional help
Overview
DataShop Overview Logging model DataShop Features
Quantitative models of learning curves Power law, logistic regression Contrasting KC models
Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing
DataShop Analysis Tools
Dataset Info Performance Profiler Learning Curve Error Report Export Sample Selector
• Meta data for given dataset
• PI’s get ‘edit’ privileges, others must request it
• Meta data for given dataset
• PI’s get ‘edit’ privileges, others must request it
15
Papers and Files storage
Papers and Files storage
Dataset MetricsDataset Metrics
Problem Breakdown table Problem Breakdown table
Dataset Info
Performance Profiler
Aggregate by• Step• Problem• KC• Dataset Level
Aggregate by• Step• Problem• KC• Dataset Level
View measures of• Error Rate• Assistance Score• Avg # Hints• Avg # Incorrect• Residual Error Rate
View measures of• Error Rate• Assistance Score• Avg # Hints• Avg # Incorrect• Residual Error Rate
Multipurpose tool to help identify areas that are too hard or easy
Multipurpose tool to help identify areas that are too hard or easy
View by KC or Student, Assistance Score or Error Rate
View by KC or Student, Assistance Score or Error Rate
Time is represented on the x-axis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC
Time is represented on the x-axis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC
Visualizes changes in student performance over time
Visualizes changes in student performance over time
Learning Curve
• Provides a breakdown of problem information (by step) for fine-grained analysis of problem-solving behavior
• Attempts are categorized by student
• Provides a breakdown of problem information (by step) for fine-grained analysis of problem-solving behavior
• Attempts are categorized by student
View by Problem or KCView by Problem or KC
Error Report
Sample Selector
Filter by • Condition• Dataset Level• Problem• School• Student• Tutor Transaction
Filter by • Condition• Dataset Level• Problem• School• Student• Tutor Transaction
Easily create a sample/filter to view a smaller subset of data
Easily create a sample/filter to view a smaller subset of data
Shared (only owner can edit) and private samples
Shared (only owner can edit) and private samples
Export• Two types of export available
• By Transaction• By Step
• Anonymous, tab-delimited file• Easy to import into Excel!
You can also export the Problem Breakdown table and LFA values!
You can also export the Problem Breakdown table and LFA values!
Help/Documentation
• Extensive documentation with examples• Contextual by tool/report• http://learnlab.web.cmu.edu/datashop/help
• Extensive documentation with examples• Contextual by tool/report• http://learnlab.web.cmu.edu/datashop/help
Glossary of common terms, tied in with PSLC Theory wiki
Glossary of common terms, tied in with PSLC Theory wiki
New Features
Manage Knowledge Component models Create, Modify & Delete KC models within
DataShop Addition of Latency Curves to Learning Curve
Reporting Time to Correct Assistance Time
Problem Rollup & Export Enhanced Contextual Help
Overview
DataShop Overview Logging model DataShop Features
Quantitative models of learning curves Power law, logistic regression Contrasting KC models
Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Recall learning curve story
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Without decomposition, using just a single “Geometry” KC,
no smooth learning curve.
But with decomposition, 12 KCs for area concepts,
a smooth learning curve.
Upshot: A decomposed KC model fits learning & transfer data better than a “faculty theory” of mind
Learning curve analysis
The Power Law of Learning (Newell & Rosenbloom, 1993) Y = a Xb
Y – error rateX – opportunities to
practice a skilla – error rate on 1st opportunity b – learning rateAfter the log transformation“a” is the “intercept” or starting point of the learning curve“b” is the “slope” or steepness of the learning curve
More sophisticated learning curve model Generalized Power Law to fit learning curves
Logistic regression (Draney, Wilson, Pirolli, 1995)
Assumptions Different students may initially know more or less
=> use an intercept parameter for each student Students learn at the same rate
=> no slope parameters for each student Some productions may be more known than others
=> use an intercept parameter for each production Some productions are easier to learn than others
=> use a slope parameter for each production
These assumptions are reflected in detailed math model …
More sophisticated learning curve model
Probability of getting a step correct (p) is proportional to:- if student i performed this step = Xi,
add overall “smarts” of that student = i
- if skill j is needed for this step = Yj, add easiness of that skill = j
add product of number of opportunities to learn = Tj & amount gained for each opportunity = j
( ) jjjjjiipp TYYX ∑ ∑∑ ++=− γβα1ln p
Use logistic regression because response is discrete (correct or not) Probability (p) is transformed by “log odds” “stretched out” with “s curve” to not bump up against 0 or 1
(Related to “Item Response Theory”, behind standardized tests …)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Different representation, same model Predicts whether student is correct depending on knowledge & practice Additive Factor Model (Draney, et al. 1995, Cen, Koedinger, Junker, 2006)
The Q MatrixThe Q Matrix
How to represent relationship between knowledge components and student tasks?
Tasks also called items, questions, problems, or steps (in problems) Q-Matrix (Tatsuoka. 1983)
2* 8 is a single-KC item 2*8 – 3 is a conjunctive-KC item, involves two KCs
29
Item | KC Add Sub Mul Div
2*8 0 0 1 0
2*8 - 3 0 1 1 0
30
Model Evaluation
• How to compare cognitive models?• A good model minimizes prediction risk by balancing fit
with data & complexity (Wasserman 2005)• Compare BIC for the cognitive models
• BIC is “Bayesian Information Criteria”• BIC = -2*log-likelihood + numPar * log(numOb)• Better (lower) BIC == better predict data that haven’t seen
• Mimics cross validation, but is faster to compute
31
Model Title LL BIC numPar
G-2,175 4,566 26
Original -1,911 4,271 54
Item -1,720 5,554 254
• Data: the Geometry Area Unit• 24 students, 230 items, 15 KCs
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Not a smooth learning curve -> this knowledge component model is wrong. Does not capture genuine student difficulties.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
More detailed cognitive model yields smoother learning curve. Better tracks nature of student difficulties & transfer
(Few observations after 10 opportunities yields noisy data)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Best BIC (parsimonious fit) for Default (original) KC model
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Better than simpler Single-KC model
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
And better than more complex Unique-step (IRT) model
Overview
DataShop Overview Logging model DataShop Features
Quantitative models of learning curves Power law, logistic regression Contrasting KC models
Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing
Exploratory Data Analysis Exercise Goals:
1) Get familiar with data 2) Learn/practice Excel skills
Tasks: 1) create a “step table” 2) graph learning curves
Exported File Loaded into Excel
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Overview
DataShop Overview Logging model DataShop Features
Quantitative models of learning curves Power law, logistic regression Contrasting KC models
Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing