Embed Size (px)
Citation preview
P r e s e n t e d b y
Members of the Apereo LAI Community
March 18, 2015Sandeep Jayaprakash, Marist
Gary Gilbert, Unicon
OPEN-SOURCE ACADEMIC EARLY ALERT &
RISK ASSESSMENT API
Presenters
Sandeep JayaprakashLearning Analytics Specialist, Marist College
Gary GilbertSoftware Architect, UniconIntegrations & Analytics
Agenda
Marist early Alert framework
Open Learning Analytics vision
Learning Analytics Processor
Demo
Discussion
OAAI: Overview and Impact
EDUCAUSE Next
Generation Learning
Challenges (NGLC)
Funded by Bill and
Melinda Gates Foundations
$250,000 over a 15 month period
Goal: Leverage Big Data concepts to create an
open-source academic early alert system and
research “scaling factors”
OAAI: Overview and Impact
Build learning analytics-based early alert system
Sakai Collaboration and Learning Environment
Secure data capture process for extracting LMS data
Pentaho Business Intelligence Suite
Open-source data mining, integration, analysis & reporting
OAAI Predictive Model released under open license
Predictive Modeling Markup Language
Researching learning analytics scaling factors
How “portable” are predictive models?
What intervention strategies are most effective?
Student Aptitude Data
(SATs, current GPA, etc.)
Student Demographic
Data (Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive
Model
Scoring
Identifies students
“at risk” to not
complete course
SIS
Dat
aLM
S D
ata
OAAI Early Alert System Overview
Intervention Deployed
“Awareness” or Online
Academic Support
Environment (OASE)
“Creating an Open Academic Early Alert System”
Model DevelopedUsing Historical Data
Step #1: Developed
model using historical
data
Academic Alert
Report (AAR)
Predictors of
Student Risk
Some predictors
were discarded if
not enough data
was available.
LMS predictors were
measured relative
to course averages.
OAAI Predictive Process
Research Design
Deployed OAAI system to 2200 students across four
institutions
Two Community Colleges
Two Historically Black Colleges and Universities
Design > One instructor teaching 3 sections
One section was control, other 2 were treatment groups
Each instructor received an AAR three times during
the semester:
Intervals were 25%, 50% and 75% into the semester
Prediction Results
Spring ’12 Portability Findings
Fall ’12 Portability Findings
Conclusion
1. Predictive models
are more “portable”
than anticipated.
2. It is possible to
create generic
models that are
then “tuned” for use
at specific types of
institutions.
Intervention Research Findings Final Course Grades
Analysis showed a
statistically significant
positive impact on final
course grades
No difference between
treatment groups
Saw larger impact in
spring than fall
Similar trend amount
low income students
50
60
70
80
90
100
Awareness OASE Control
Fin
al G
rad
e (%
)
Mean Final Grade for "at Risk" Students
Intervention Research Findings Content Mastery
Student in intervention
groups were statistically
more likely to “master
the content” than those
in controls.
Content Mastery = Grade
of C or better
Similar for low income
students.
0
200
400
600
800
1000
Yes No Yes No
Content Mastery for "at Risk" Students
Control Intervention
Freq
uen
cy
Instructor Feedback
"Not only did this project directly assist my students by guiding
students to resources to help them succeed, but as an instructor,
it changed my pedagogy; I became more vigilant about
reaching out to individual students and providing them with
outlets to master necessary skills.
P.S. I have to say that this semester, I received the highest
volume of unsolicited positive feedback from students, who
reported that they felt I provided them exceptional individual
attention!
JAYAPRAKASH, S . M. , MOODY, E . W. , LAURÍA, E . J . ,
REGAN, J . R . , & BARON, J . D . (2014) . EARLY ALERT OF
ACADEMICALLY AT -R ISK STUDENTS : AN OPEN SOURCE
ANALYT ICS IN I T IAT IVE . JOURNAL OF LEARNING
ANALYT ICS , 1 (1) , 6 -47 .
More Research Findings…
Strategic Vision: Open Learning
Analytics PlatformCollectionStandards-based data capture from any potential source using Experience API and/or IMS Caliper/Senor API
StorageSingle repository for all learning-related data using Learning Record Store (LRS) standard.
AnalysisFlexible Learning Analytics Processor (LAP) that can handle data mining, data processing (ETL), predictive model scoring and reporting.
CommunicationDashboard technology for displaying LAP output.
ActionLAP output can be fed into other systems to trigger alerts, etc.
Technology Stack
Learning Analytics Processor (LAP)
JAVA-based web application
Maven for builds
Temporary Storage - H2 in-memory database
Persistence Storage - MySQL
Predictive Model Mark-up Language (PMML)
OAAI Early Alert Pipeline
Pentaho Kettle – Data Integration & ETL
Pentaho WEKA – Data Mining & Predictive Modelling
High-Level Workflow
Sakai
Admin
tool
activities.csv
grades.csv
Learning Analytics Processor (LAP)Student ID,
Course ID,
Risk Rating
Demographics
from SIS
Go!
grabs files
OAAI XML
Kettle pipeline
applies model
outputs results
..
.
.
------------------ EXTRACT -------- TRANSFORM ------- LOAD ---------
RESTful API
LAP Pipeline Architecture
Features
Key pieces of the LAP architecture
Input source management
Data storage – temporary & persistent
Configuration manager
Pipeline processor
Output results management
Extensibility
Supports multiple pipelines
Supports varied pipeline platforms
Demo Overview
● Three core components of a
collection of open source
applications and services that
represent the “Analytics Diamond”
● Can be used individually or
collectively
● Work with a shared infrastructure
and data model
Technologies:
• AngularJS
• Spring-Boot
• Pluggable Datastores
(redis, elasticsearch, mongodb)
OpenLRS
Learning
Analytics
Processor
SakaiOpen
Dashboard
xAPI
LTI
API
API
Demo
Early Alert Insights – Open Dash
Questions?
APEREO LEARNING ANALYTICS INITIATIVE COMMUNITY
• Accelerate the operationalization of Learning Analytics software and frameworks
• Support the validation of analytics pilots across institutions
• Work together so as to avoid duplication
Josh [email protected]
Sandeep Jayaprakashsandeep.jayaprakash1@
marist.edu
Gary [email protected]
Appendix
Early Alert - Kettle ETL Flows
WEKA Predictive Modelling Flows
Learning Analytics Processor
