A Framework for Automatic Empirically-Based Metadata Generation Intelligent Learning Object Guide...
Click here to load reader
prev
next
of 32
A Framework for Automatic Empirically-Based Metadata Generation Intelligent Learning Object Guide (iLOG) S.A. Riley a, L.D. Miller a, L.-K. Soh a, A. Samal
A Framework for Automatic Empirically-Based Metadata Generation
Intelligent Learning Object Guide (iLOG) S.A. Riley a, L.D. Miller
a, L.-K. Soh a, A. Samal a, and G. Nugent b a University of
NebraskaLincoln: Department of Computer Science and Engineering b
University of NebraskaLincoln: Center for Research on Children,
Youth, Families and Schools
Slide 2
Overview Intelligent Learning Object Guide 2 Introduction: What
is a Learning Object (LO)? Why do we need LO metadata? Metadata
problems and iLOG solution iLOG Framework LO Wrapper MetaGen
(metadata generator) Data Logging Data Extraction Data Analysis
(feature selection, rule mining, statistics) Conclusions and Future
Work
Slide 3
Introduction: What is a learning object? Intelligent Learning
Object Guide 3 Self-contained learning content Ideally, each covers
a single topic Serve as building blocks for lessons, modules, or
courses Can be reused in multiple instructional contexts Learning
Object LO Metadata iLOG Learning Object structure: Content:
tutorial, exercises, assessment Metadata
Slide 4
Introduction: What is a learning object? Intelligent Learning
Object Guide 4 Tutorial Assessment Exercises The iLOG LOs contain a
tutorial, exercises, and assessment Each covers a bite-sized
introductory computer science topic
Slide 5
Intelligent Learning Object Guide 5 Repositories for LOs are
being constructed However, there are barriers to effective
utilization of these repositories: Learning Context: not all LOs,
even on the same topic, are suitable for use in a given learning
context Uncertainty: we cannot be certain what will happen with
real-world usage Search and Retrieval: current metadata is not
machine-readable, and thus is not adequate to automate the search
for LOs Introduction: Why do we need LO metadata? Learning Object
LO Metadata LO Repository Learning Object LO Metadata Learning
Object LO Metadata Learning Object LO Metadata Learning Object LO
Metadata
Slide 6
Introduction: Why do we need LO metadata? Intelligent Learning
Object Guide 6 Learning Context: Students are highly varied:
Pre-existing knowledge, cultural background, motivation,
self-efficacy, etc. Uncertainty: Cannot be certain what will happen
when actual students use an actual LO: Good for students with low
self-efficacy Inherent gender bias Bad for students without
Calculus experience Search and Retrieval: Metadata is fundamental
to an instructors ability to use LOs: Guide in the LO selection
process Help prevent the feeling that e-learning is too
complicated
Slide 7
Intelligent Learning Object Guide 7 So how do we enable
instructors to locate appropriate LOs for their students???
Slide 8
Introduction: Metadata problems and iLOG solution Current
MetadataIdeal Metadata Intelligent Learning Object Guide 8 Manual
generation by course designer Based only on designer intuition
Metadata format inconsistent / incomplete Human- but not machine-
readable Automated generation Based on empirical usage Consistent
metadata suitable for guiding LO selection Both human and machine-
readable Current metadata standards are insufficient (Freisen,
2008) There are ample opportunities for making e-learning more
intelligent (Brooks et al., 2006)
Slide 9
Intelligent Learning Object Guide 9 The iLOG solution is:
General: iLOG is based on established learning standards We use the
SCORM learning object standard, the IEEE LOM metadata standard, and
the Blackboard LMS Furthermore, it is compatible with existing LOs
and does not require modification to the LOs (noninvasive) The iLOG
framework can also be applied to other standards Automatic: iLOG
metadata is automatically generated and updated Interpretable: iLOG
metadata is both human and machine readable Introduction: Metadata
problems and iLOG solution
Slide 10
Intelligent Learning Object Guide 10 LO Wrapper: logs student
behaviors when using LO MetaGen : generates empirical usage
metadata using data mining techniques Works noninvasively with
pre-existing LOs using standard learning management systems (LMSs)
Introduction: Metadata problems and iLOG solution LO Wrapper
Learning Object LO Metadata LO Wrapper Learning Object LO Metadata
Learning Management System (LMS) LO Wrapper Learning Object LO
Metadata
Slide 11
Related Work Intelligent Learning Object Guide 11 Automatic
metadata generation Primarily focuses on content taxonomies (Roy et
al., 2008; Jovanovic et al., 2006) Mining student behavior log
files Mining has been shown to have a positive impact on
instruction and learning (Kobsa et al., 2007) Standardization of
educational log file data Significant progress has been made with
tutor-message format standard (PSLC DataShop)
Slide 12
Overview Intelligent Learning Object Guide 12 Introduction:
What is a Learning Object (LO)? Why do we need LO metadata?
Metadata problems and iLOG solution iLOG Framework LO Wrapper
MetaGen (metadata generator) Data Logging Data Extraction Data
Analysis (feature selection, rule mining, statistics) Conclusions
and Future Work
Slide 13
iLOG Framework Intelligent Learning Object Guide 13 iLOG
dataset Log Files and Existing Metadata Data- base Rule Mining
Feature Subset Feature Selection Statistics Generation MetaGen
Rules and Statistics LO Wrapper LO Metadata Learning Object Two
components: LO Wrapper and MetaGen Data Analysis Data Extraction
Data Logging
Slide 14
iLOG Framework: LO wrapper Intelligent Learning Object Guide 14
LO Wrapper: Wraps around an existing LO Intercepts student
interactions and logs them to a database Does not require changing
the LO LO Wrapper Learning Object LO Metadata
Slide 15
iLOG Framework: MetaGen Intelligent Learning Object Guide 15
iLOG dataset Data- base Rule Mining Feature Subset Feature
Selection Statistics Generation MetaGen Rules and Statistics
MetaGen modules: Data Logging, Data Extraction, Data Analysis Data
Analysis Data Extraction Data Logging
Slide 16
iLOG Framework: MetaGenLogging Intelligent Learning Object
Guide 16 Potential data sources: Interactions: clicks, time spent,
etc. Surveys: demographic, motivation, self-efficacy, evaluation
Assessment scores Log Files Data- base Data Logging MetaGen LO
Wrapper LO Metadata Learning Object
Slide 17
iLOG Framework: MetaGenLogging Intelligent Learning Object
Guide 17 Static Learner Data Static LO DataInteraction Data
Baseline motivation Baseline self- efficacy Gender Major GPA
SAT/ACT score Topic Length Degree of difficulty Level of feedback.
Blooms level for assessment questions Total time on tutorial Total
time on exercises Total time on assessment Min time spent on a
tutorial page Max time spent on a tutorial page Avg. time per
assessment question Data sources used in our iLOG deployment:
Slide 18
iLOG Framework: MetaGenExtraction Intelligent Learning Object
Guide 18 Data Extraction: Uses Java application to query the
relational database and extract a flat dataset suitable for data
mining: Student Behaviors: Average time per tutorial page, Total
time on assessment, etc. Student Characteristics: Total motivation
self-rating, GPA, Gender, etc. iLOG dataset Log Files and Existing
Metadata Data- base Data Extraction Data Logging MetaGen LO Wrapper
LO Metadata Learning Object
Slide 19
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 19 Data Analysis (feature selection): Uses ensemble of
feature selection algorithms Seeks to identify student behaviors
and characteristics that are relevant to learning outcomes iLOG
dataset Log Files and Existing Metadata Data- base Feature Subset
Feature Selection MetaGen LO Wrapper LO Metadata Learning Object
Data Analysis Data Extraction Data Logging
Slide 20
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 20 Feature selection (FS) is used to find a subset of
variables (features) that is sufficient to describe a dataset
(Guyon et al., 2003) Different techniques may generate different
results Instead, our goal was to find ALL features relevant to
learning outcomes Thus, the feature selection ensemble members vote
on which features they identify as most relevant All features FS#1
FS#2 FS#3
Slide 21
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 21 Notable Results: Relevant features varied widely across
LOs Discovered unexpected patterns: Possible gender bias, Calculus
bias, etc. Logic 2 Searching Attribute Number of Times Selected
AttributeNumber of Times Selected highestMath gender takenCalculus
assessStdDevSecAboveAvg? wasAnyPartConfusing? 16 13 GPA
assessMinSecPageBelowAvg? assessmentMinScondsOnAPage
believeLODifficultToUnderstand courseLevel 14 11 10 9
Slide 22
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 22 Rule Mining: Uses Tertius algorithm for predictive rule
mining Generates rules from selected features (along with rule
strength) iLOG dataset Log Files and Existing Metadata Data- base
Rule Mining Feature Subset Feature Selection MetaGen LO Wrapper LO
Metadata Learning Object takenCalculus? = no fail (.52)
currentTotalMotivationAboveAvg? = no fail (.52) gender = female
fail (.36) Data Analysis Data Extraction Data Logging
Slide 23
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 23 Statistics Generation: Empirical data: time to complete,
pass/fail rates, and student ratings of LO iLOG dataset Log Files
and Existing Metadata Data- base Rule Mining Feature Subset Feature
Selection Statistics Generation MetaGen LO Wrapper LO Metadata
Learning Object successRate = 51% averageTime = 433 seconds
averageStudentRating = 4.3/5.0 Data Analysis Data Extraction Data
Logging
Slide 24
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 24 Logic 2Intro CS for non-majors
assessmentStdDevSecondsAboveAvg? = yes fail (.35)
assessmentMaxSecondsOnAQuestion = high fail (.33) highestMath =
precalculus fail (.28) gender = female fail (.24) successRate = 51%
averageTime = 433 seconds averageStudentRating = 4.3/5.0 Logic
2--Intro CS for majors baselineStdDevMotivation = low fail (.72)
takenCalculus? = no fail (.52) currentTotalMotivationAboveAvg? = no
fail (.52) successRate = 38% averageTime = 688 seconds
averageStudentRating = 4.16/5.0 Logic 2Honors Intro CS for majors
OpinionOfLOUsability = negative fail (.59)
BelieveLOAnAidToUnderstanding = yes pass (.49)
BelieveLONeedsMoreDetail = yes fail (.43) gender = female fail
(.36) successRate = 55% averageTime = 799 seconds
averageStudentRating = 3.43/5.0 Appear to be different predictors
of success for different learning contexts: Honors: student
impression of LO, gender Majors: motivation, math experience
Non-majors: long time spent on assessment, math experience,
gender
Slide 25
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 25 Logic 2Intro CS for non-majors
assessmentStdDevSecondsAboveAvg? = yes fail (.35)
assessmentMaxSecondsOnAQuestion = high fail (.33) highestMath =
precalculus fail (.28) gender = female fail (.24) successRate = 51%
averageTime = 433 seconds averageStudentRating = 4.3/5.0 Logic
2--Intro CS for majors baselineStdDevMotivation = low fail (.72)
takenCalculus? = no fail (.52) currentTotalMotivationAboveAvg? = no
fail (.52) successRate = 38% averageTime = 688 seconds
averageStudentRating = 4.16/5.0 Logic 2Honors Intro CS for majors
OpinionOfLOUsability = negative fail (.59)
BelieveLOAnAidToUnderstanding = yes pass (.49)
BelieveLONeedsMoreDetail = yes fail (.43) gender = female fail
(.36) successRate = 55% averageTime = 799 seconds
averageStudentRating = 3.43/5.0 Inverse relationship: time spent on
LO and student ratings: Advanced students may have higher
expectations (lower ratings) Advanced students may care more about
the material (time spent)
Slide 26
iLOG Framework: MetaGenAnalysis Intelligent Learning Object
Guide 26 Rules and Statistics: Usage statistics and rules are
combined to form empirical usage metadata iLOG dataset Log Files
and Existing Metadata Data- base Rule Mining Feature Subset Feature
Selection Statistics Generation MetaGen LO Wrapper LO Metadata
Learning Object Rules and Statistics Data Analysis Data Extraction
Data Logging
Slide 27
iLOG Framework: Our Implementation Intelligent Learning Object
Guide 27 LO wrapper: HTML document that uses Java-script to record
and timestamp student interactions with the LO (e.g., page
navigation, clicks on a page, etc.). Uses a modification of the
Easy SCO Adapter 1 to interface with the SCORM API and retrieve
student assessment results from the LMS. Uses JavaScript to
transmit interaction data to MetaGen MetaGen: Data logging: uses
PHP to store student interaction data into a MySQL database. Data
extraction: uses Java to query the database and process the data
into the iLOG dataset. Data analysis: uses the Weka (Witten, 2005)
implementations of several feature selection algorithms to generate
the iLOG data-subset and the (Flach, 2001) predictive rule mining
algorithm to generate empirical usage metadata rules. 1
[http://www.ostyn.com/standards/demos/SCORM/wraps/easyscoadapterdoc.htm#license]http://www.ostyn.com/standards/demos/SCORM/wraps/easyscoadapterdoc.htm#license
Slide 28
Overview Intelligent Learning Object Guide 28 Introduction:
What is a Learning Object (LO)? Why do we need LO metadata?
Metadata problems and iLOG solution iLOG Framework LO Wrapper
MetaGen (metadata generator) Data Logging Data Extraction Data
Analysis (feature selection, rule mining, statistics) Conclusions
and Future Work
Slide 29
Conclusions Intelligent Learning Object Guide 29 iLOG: a
framework for automatic, empirical metadata generation: LO Wrapper
component: Wraps noninvasively around pre-existing learning objects
(LOs) Automatically collects and logs student interaction data
Resulting LOs can be played on a standard LMS, such as Blackboard
MetaGen component (metadata generator): Uses data mining to create
empirical usage metadata: Feature selection: provides insights on
which student characteristics and behaviors may contribute to
success in different learning contexts. Rule mining: uses salient
features to generate rules predicting success Usage statistics:
empirical evidence of time to complete, scores, etc. iLOGs
empirical use metadata should enable instructors to locate LOs that
are appropriate to their students learning context
Slide 30
Future Work: Closing the Loop Intelligent Learning Object Guide
30 iLOG dataset Log Files and Existing Metadata Data- base Rule
Mining Feature Subset Feature Selection Statistics Generation
MetaGen Rules and Statistics LO Wrapper LO Metadata Learning Object
Method to automatically write empirical usage metadata to the LO
metadata file Method to integrate new metadata with existing
metadata Data Analysis Data Extraction Data Logging
Slide 31
References Intelligent Learning Object Guide 31 IEEE
1484.12.1-2002 Standard for Learning Object Metadata (LOM).
Retrieved January 7, 2009, from
http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf N.
Friesen, The International Learning Object Metadata Survey.
Retrieved August 7, 2008, from
http://www.irrodl.org/index.php/irrodl/article/view/195/277/ C.
Brooks, J. Greer, E. Melis, C. Ullrich, Combining ITS and eLearning
Technologies: Opportunities and Challenges, Proc. 8 th Int. Conf.
on Intelligent Tutoring Systems (2006), 278-287. D. Roy, S Sarkar,
S. Ghose, Automatic Extraction of Pedagogic Metadata from Learning
Content, Int. J. of Artificial Intelligence in Education 18 (2008),
287-314. J. Jovanovic, D. Gasevic, V. Devedzic, Ontology-Based
Automatic Annotation of Learning Content, Int. J. on Semantic Web
and Information Systems, 2(2) (2006), 91-119. B. Jong, T. Chan, Y.
Wu, Learning Log Explorer in E-Learning Diagnosis, IEEE
Transactions on Education 50(3) (2007), 216- 228. E. Garcia, C.
Romero, S. Ventura, C. Castro, An architecture for making
recommendations to courseware authors using association rule mining
and collaborative filtering, User Modeling and User-Adaptive
Interaction (to appear). E. Kobsa, V. Dimitrova, R. Boyle, Adaptive
Feedback Generation to support teachers in web-based distance
education, User Modeling and User-Adapted Interaction 17 (2007),
379-413. I. Guyon, A. Elisseeff, An Introduction to Variable and
Feature Selection, Journal of Machine Learning Research 3 (2003),
1157-1182. P.A. Flach, N. Lachiche, Confirmation-Guided Discovery
of First-Order Rules with Tertius, Machine Learning 42 (2001),
61-95. Ian H. Witten and Eibe Frank "Data Mining: Practical machine
learning tools and techniques",2nd Edition, Morgan Kaufmann, San
Francisco, 2005.
Slide 32
Contact and Acknowledgement Intelligent Learning Object Guide
32 iLOG project website: http://cse.unl.edu/agents/ilog
http://cse.unl.edu/agents/ilog Authors: S.A. Riley a, L.D. Miller
a, L.-K. Soh a, A. Samal a, and G. Nugent b Email:
[email protected], [email protected], [email protected],
[email protected], [email protected] This material is based upon
work supported by the National Science Foundation under Grant No.
0632642 and an NSF GAANN fellowship.