Embed Size (px)
DESCRIPTION
Towards the Development of a Real-Time Decision Support System for Online Learning, Teaching and Administration
Citation preview
Kerry Rice, Ed.D.Associate Professor and ChairAndy Hung, Ed. DAssistant ProfessorYu-Chang Hsu, Ph. D.Assistant Professor
Towards the Development of a Real-Time Decision Support System for Online Learning, Teaching and Administration
M.S. in Educational Technology
Masters in Educational Technology
Ed. D in Educational Technology
K-12 Online Teaching Endorsement
Graduate Certificates:Online Teaching - K12 & Adult Learner
Technology Integration Specialist
School Technology Coordinator
Online Teacher PD Portal
Game Studio: Mobile Game Design
Learning Technology Design Lab
EDTECH Fast Facts
• Largest graduate program at BSU• Fully online, self-support program• Served over 1,200 unique students last year• Interdisciplinary partnerships with Math, Engineering,
Geoscience, Nursing, Psychology, Literacy, Athletics. • Partnerships with iNACOL, AECT, ISTE, Google,
Stanford, IDLA, Connections Academy, K12, Inc., ID State Department of Education, Discovery Education, Nicolaus Copernicus University, Poland
• First dual degree program – National University of Tainan.
• Save 200+ tons of CO2 emissions annually
Going Virtual! Research Series
Going Virtual! Research Series
• Who delivered/received PD? • When and how PD was delivered? • Content and sequence of PD?
2007: The Status of Professional Development
• Amount of PD?• Preferred delivery format?• Most important topics for PD?
2008: Unique Needs and Challenges
• Program evaluations• Complexities of measuring “effectiveness”
2009: Effective Professional Development of K-12 Online Teachers
• Revisit questions from 2007 & 2008 • What PD have you had? What do you need?
2010: The Status of PD and Unique Needs of K-12 Online Teachers
• Pass Rate Predictive Model• Engagement• Association Rules
2011: Development of an Educational Data Mining model
Going Virtual! Research SeriesG
oing
Virt
ual!
2007 258 Respondents
•167 K-12 online teachers•61 Administrators•14 TrainersOver 40 virtual schools and online programsOver 30 states G
oing
Virt
ual!
2008 884 K-12 Online
Teachers•727 virtual schools•99 supplemental programs
•54 brick and mortar online programs
Over 60 virtual schools and online programsOver 30 states
Goi
ng V
irtua
l 201
1 Traditional• Virtual Charter• Supplemental Program
With DATA MINING• Online Teacher PD Workshops
• Online Graduate Courses
• End of Year Program Evaluation
Goin
g V
irtu
al!
2
01
0830 K-12 Online Teachers•417 Virtual School•318 Supplemental•81 Blended•12 Brick N MortarOver 50 virtual schools and online programsOver 40 states & 24 countries
Des
crip
tive
Eval
uativ
eGoals: • Program evaluation• Develop cloud-based, real-time
Decision Support System (DSS)• Link PD effectiveness to
student outcomes
Traditional Evaluation Systems
Teacher Effectiveness
Highly qualified?
Parent Satisfaction
Annual Performance
Range of implementatio
n
Student Satisfaction
Knowledge of STS
Program
AYP?
Improved Test Scores
Parent Satisfaction
Student Outcomes
Performance
Participation
Attendance
ISAT/DWA
Self-Efficacy
Satisfaction
Leveraging Data Systems
PD Effectiveness
Quality
Usefulness
Engagement
Teacher Effectivenes
s
Change in teaching practice
Quantity AND Quality
of Interaction
Course Design
Student Outcomes
Satisfaction
Engagement
Dropout Rate
Performance
Learning Patterns
Self report
Self report Self report
Low-level data
Low-level data
Data Mining
Data mining techniques can be applied in online environments to understand hidden relationships between logged activities, learner experiences, and
performance. It can be used in education to track learner behaviors, identify struggling students, depict learning
preferences, improve course design, personalize instruction, and predict student performance.
Educational Data Mining
Special Challenges• Learning behaviors are complex• Target variables (learning outcomes/performance)
require wide range of assessments and indicators• Goal of improving online teaching and learning is hard
to quantify• Limited number of DM techniques suitable to meet
educational goals• Only interactions that occur in the LMS can be tracked
through data mining. What if learning occurs outside the LMS?
• Still a very intensive process to identify rules and patterns
DM Applications in Education
• Pattern discovery (data visualization, clustering, sequential path analysis)– Track students’ learning progress– Identify outliers (outstanding or at-risk students)– Depict students’ learning preferences (learner profiling)– Identify relationships of course components (web
mining)• Predictive Modeling (decision tree analysis)
– Suggest personalized activities (classification prediction)– Foresee student performance (numeric prediction)– Adaptive evaluation system development
• Algorithm generation: analysis methods can be integrated into platforms.
Data Preprocessing
• Data Collection• Data Cleaning• Session Identification• Behavior Identification
Data Transformation
3 Data Mining Studies
• Study #1: Teacher Training Workshops 2010– Survey Data + Data Mining + Student Outcomes
• Study #2: Graduate Courses 2010– Data Mining + Student Outcomes (no demographic data)
• Study #3: End of Year K-12 Program Evaluation (2009 – 2010)– Data Mining + Student Outcomes + Demographic Data
+ Survey Data
Study #1: Teacher Training Workshops 2010
• Survey Data + Data Mining + Student Outcomes• Research Goal: To demonstrate the potential
applications of data mining with a case study – Program evaluation of workshop quality for continuous
improvement of design and delivery.– Evaluation of PD impact on both teachers (and
students).
• Blackboard• 103 participants• 31,417 learning logs• clustering analysis, sequential association
analysis, and decision tree analysis • Engagement variables
– Frequency of logins– Length of time online (survey and dm)– Frequency of content access– Number of discussion posts
Study #1: Teacher Training Workshops 2010
Learning Paths
• Association Rule Analysis– Participants tended to switch between content and
discussion within one session. – Different types of interactions (content-participant,
participant-instructor, and participant-participant) were well facilitated in the workshops overall.
Performance
Pass Rate Predictive Model• Decision Tree Analysis
– Improved grades and pass rate (from 88% to 92% and 89% to 94% respectively) when participants’ logged into LMS more than 10 times over six weeks. The average for both is further improved to 98% when frequency of logins increased to 17 times.
Increased logins = Increased performance
Quality of Experience
Engagement• Clustering + Survey Questions
– More time spent online = more time spent offline.– Previous online teaching experience = more hours
spent both online and offline.
DM Conclusions
• Interaction and engagement were important factors in learning outcomes.
• The results indicate that the workshops were well facilitated, in terms of interaction.
• Participants who had online teaching experience could be expected to have a higher engagement level but prior online learning experience did NOT show a similar relationship.
• There is a direct relationship between the amount of time learners spent online and their average course logins to engagement and performance. Specifically, more time spent online and a higher frequency of logins equates to increased engagement and improved performance.
• Two factors influenced expectation ratings:– Practical new knowledge– Ease of locating information
• Three factors influenced satisfaction ratings:– Usefulness of subject-matter– Well-structured website– Sufficient technical supports
• Instructor quality was related to:– Stimulated interest– Preparation for class– Respectful treatment of students– Peer collaboration– Assessments aligned to course objectives– Support services for technical problems
Overall Conclusions
Study #2: Graduate Courses 2010
• Data Mining + Student Outcomes (no demographic data)
• Research Goal: To demonstrate the potential applications of data mining with a case study – Generate personalized advice– Identify struggling students– Adjust teaching strategies– Improve course design– Data Visualization
• Study Design– Comparative (between and within courses)– Random course selection
Study #2: Graduate Course 2010
• Moodle• Two graduate courses (X and Y)• Each with two sections
– X1 (18 students)– X2 (19 students)– Y1 (18 students)– Y2 (22 students)
• 2,744,433 server logs
Study #2: Graduate Course 2010
• Variables– ID’s (user and session)– Learning Behaviors (reading materials, posting disc.)– Time/duration– Grades or pass/fail (independent variables)
Weekday Course PatternsWeekday Student Patterns
Learner Behaviors
Weekday and Time Patterns of Learning Behaviors
• Reading is the major activity; Similar patterns• Sunday => reply discussions • Monday & Tuesday, between 1pm and midnight
Shared Student Characteristics Course X
CLUSTER ANALYSIS
1) LOW ENGAGED – LOW PERFORMING
2) HIGH ENGAGED-HIGH PERFORMING
3) HIGH ENGAGED – LOW PERFORMING
4) LOW ENGAGED – HIGH PERFORMING
Shared Student Characteristics Course Y
CLUSTER ANALYSIS
1) LOW ENGAGED – LOW PERFORMING
2) HIGH ENGAGED-HIGH PERFORMING
3) HIGH ENGAGED – LOW PERFORMING
4) LOW ENGAGED – HIGH PERFORMING
ASSOCIATION RULEPATH ANALYSISCOURSE DESIGN
Learner Behaviors
Predictive Analysis – Course X
Discussion board posts and replies were the most important variable for predicting performance (27+ replies = better performance)
Some lower performers had high reply numbers (> 43)
Cluster analysis revealed that students tended to only read discussions.
Predictive Analysis – Course Y
Number of discussion board posts read was the most important predictor of performance (378+ = better performance)
Fewer discussions read + more replies (54+ = better performance)
The design of course Y improved the quality of discussions and influenced student behaviors.
• Demographics + Survey Data + Data Mining + Student Outcomes
• Research Goal: Large scale program evaluation– How can the proposed program evaluation framework
support decision making at the course and institutional level?
– Identify key variables and examine potential relationships between teacher and course satisfaction, student behaviors, and student performance outcomes
Study #3: End of Year K-12 Program Evaluation
Study #3: End of Year K-12 Program Evaluation (2009 – 2010)
• Blackboard LMS• 7500 students • 883 courses• 23,854,527
learning logs (over 1 billion records)
Total Variables = 22
stuIDAgeCityDistrictGrade_AvgClick_AvgContent_Access_AvgCourse_Access_AvgPage_Access_AvgDB_Entry_AvgTab_Access_Avg
Login_AvgModule_AvgGenderHSGradYearSchoolNo_CourseNo_FailNo_PassPass ratecSatisfaction_AvgiSatisfaction_Avg
• Average frequency of logins per course.• Average frequency of tab accessed per course• Average frequency of module accessed per course• Average frequency of clicks per course• Average frequency of courses accessed (from the
Blackboard portal)• Average frequency of page accessed per course (page tool)• Average frequency of course content accessed per course
(content tool)• Average number of discussion board entries per course.
Engagement
Cluster Analysis - by StudentSpring 2010
Cluster Analysis - by Student
• High engagement = high performance • The optimal number of courses = 1 to 2 per semester • Older students (age > 16.91) tended to take more than two
courses with pass rates ranging from 54.09-56.11%• High-engaged students demonstrated engagement levels
twice that of low-engaged students • Female students were more active than male students in
online discussions (with higher DB_Entry avg frequency)• Female students had higher pass rates than male students
Identified lowest performing courses (Math, Science and English) were analyzed with cluster analysis. • High-engaged + high performance = good design and good
implementation?• High engaged + low performance = bad design and good
implementation?• Low engaged + low performance = bad design and bad
implementation?
Cluster Analysis – by Course
Subject areas in which the level of activity was consistent with student outcomes: – High Performance and High Engagement = Driver
Education, Electives, Foreign Language, Health, and Social Studies
– Low Engagement and Low Performance = English
Subject areas in which the level of activity was inconsistent with student outcomes:– High Engagement and Low Performance = Math and
Science. Why?
Cluster Analysis – by Course
• Regardless of the content area or level of engagement, low performance courses were entry-level
• Most high-engaged, high performance courses were advanced level courses.
• Regardless of Math, Science, or English subject-matter, entry level courses tended to have lower performance whether students were categorized as low-engaged or high-engaged.
• The reasons students enrolled in a course may influence their engagement level and performance. Student survey responses indicated that students who retook courses they have previously failed, tended to demonstrate lower engagement and lower performance.
Cluster Analysis – by Course
• Positive correlation between engagement level and performance (higher engaged => higher performance)
• Engagement level and gender have stronger effects on student final grades than age, school district, school, and city. For most students, high engaged => high performance
• Overall, female students performed better than male students
• Students who were around 16 years old or younger performed better than those who were 18 years or older.
• Compared with other Blackboard components such as discussion board entries and content access, tab access had negative effects on student performance (higher tab access => lower performance)
Predictive Analysis – Pass Rate
• Students with higher average final grades (> 73.25) had higher course satisfaction.
• Students who passed all courses or passed some of their courses had higher course satisfaction than all-failed students.
• Students who took two or more courses in Spring 2010, whether they passed those courses or not, had higher course satisfaction.
• Female students had higher course satisfaction than male students.
• Online behaviors (i.e., frequency of page accessed and number of discussion board entries) had minor effects on course satisfaction (higher frequency/number => higher course satisfaction).
Predictive Analysis – Course Satisfaction
• Students with higher average final grades (> 73.25%) indicated higher instructor satisfaction.
• Students who took two or more courses in Spring 2010, whether they passed those courses or not, showed higher instructor satisfaction.
• Female students indicated higher instructor satisfaction than male students.
• Online behaviors (frequency of module accessed) had minor effects on instructor satisfaction (higher frequency => higher course satisfaction).
• Older students (> 17.5 years old) had higher instructor satisfaction.
Predictive Analysis – Instructor Satisfaction
Regression Analysis
• Spring 2010 – Survey data + Data Mining• Purpose: To identify which variables contributed
significantly toward students’ average final grade. • Positive (higher values, higher average final grade)
– Self-reported GPA (Likert-scale type of response)– Satisfaction toward positive experience (Likert-scale type of
response)– Satisfaction toward course content (Likert-scale type of
response)– Time on coursework (Likert-scale type of response)– Course access (based on LMS server log data)
• Negative (higher values, lower average final grade)– Effort and challenge (based on Likert-scale type of response on
the survey)– Tab access (based on LMS server log data)
Conclusions
• Higher-engaged students usually had higher performance – limited to courses which were well-designed and
implemented. In this study, entry-level courses tended to have lower performance whether students were categorized as low engaged or high engaged high
• Satisfaction and engagement levels could not guarantee high performance
Characteristics of successful students
• Female• 16.5 years or younger• Took one or two courses per semester• Took Foreign Language or Health course• Lived in larger cities
Characteristics of at-risk students
• Male• 18 years or older• Took more than two courses per semester• Took entry-level courses in Math, Science, or
English• Lived in smaller cities
AUTOMATED INSTRUCTIONAL RESPONSE SYSTEM (AIRS)
**We are looking for partners
