Embed Size (px)
DESCRIPTION
Presentation by John Whitmer and Kathy Fernandes to the California State University Academic Technology Steering Committee on December 12, 2012.
Citation preview
Presenters: Kathy Fernandes and John Whitmer
ATSC Virtual MeetingDecember 13, 2012
System-wide LMS Learner Analytics Projects
Slides @ http://goo.gl/DYqJU
Agenda
1. Chico State Learner Analytics Research Study• EDUCAUSE Article (http://goo.gl/tESoi)
2. Current Projects• Moodle• Blackboard
2
1. CHICO STATE LEARNER ANALYTICS RESEARCH STUDY
“Logging on to Improve Achievement” by John WhitmerEdD. Dissertation (UC Davis & Sonoma State)
3
Case Study: Intro to Religious Studies• Redesigned to hybrid delivery through
Academy eLearning
• Enrollment: 373 students (54% increase on largest section)
• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)
• Bimodal outcomes:• 10% increased SLO mastery• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated reporting data
54 F’s
4
Driving Conceptual Questions
1. How is student LMS use related to academic achievement in a single course section?
2. How does that finding compare to the relationship of achievement with traditional student characteristic variables?
3. How are these relationships different for “at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are most useful to answer these questions?
5
Variables
6
Clear Trend: Grade w/Mean LMS Hits
7
Separate Variables: Correlation LMS Use & Student Characteristic with Final Grade
LMS Use
Variables
18% Average(r = 0.35–0.48)
Explanation of change in final grade
Student Characteristic
Variables
4% Average(r = -0.11–0.31)
Explanation of change in final grade
>8
Combined Variables: Regression Final Grade by LMS Use & Student Characteristic Variables
LMS Use
Variables
25% (r2=0.25)
Explanation of change in final grade
Student Characteristic
Variables
+10%(r2=0.35)
Explanation of change in final grade
>9
At-Risk Students: “Over-Working Gap”
10
Discus
sion
Activi
ty H
its
Conte
nt A
ctivi
ty H
its
Asses
smen
t Act
ivity
Hits
Act
ivity
Hits
Admini
stra
tive
Activi
ty H
its0
50
100
150
200
250
300
350
400
450
500
54 5123 36
16
382
151
58 49
26
Raw Average Hits/Student
Filtered Average Hits/Student
Filtering Data – Lots of “Noise”; Low “Signal”
Final data set: 72,000 records (-73%)11
2. CURRENT PROJECTS
12
Moodle and Bb Learner Analytics
What do these have in common?
• Multi-campus CSU groups discussing common analytics questions & query definitions
13
Moodle vs. Bb Learner AnalyticsMoodle CIG (18 months old) Chair: Andrew Roderick, SFSU
DIY, adopt and evaluate solutions from other Moodlers
Starting with technical reporting to build accurate indicators of use
2 rounds of data collection already completed and discussed
Blackboard Learn Group (just starting) CIG Chair: Terry Smith, CSUEB
Bb Learn Analytics product available “off the shelf”; defined and integrated with Peoplesoft
Pre-built Reports and Dashboards to ANYONE on campus (admin. or faculty if authenticated)
Charts available inside LMS for Faculty and Student Views14
Moodle Reporting & Analytics, Round 1
Prioritized Moodle Queries from S&PG governance group
Focused on measures of adoption (% faculty, % students, % course sections)
For expediency, campuses reported using current queries used for reporting
15
CSU_02
CSU_04
CSU_05
CSU_06
CSU_08
CSU_09
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
2,270
553
2,492
2,997
1,098
671
3,911
614
3,687
7,064
1,162
2,191
Active Sections
Inactive Sections
“How many sections are using the LMS (out of all sections offered that term)?”
16
“How many sections are using the LMS (out of all sections offered that term)?”
CSU_02
CSU_04
CSU_05
CSU_06
CSU_08
CSU_09
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
2,270
553
2,492
2,997
1,098
671
3,911
614
3,687
7,064
1,162
2,191
Active Sections
Inactive Sections
Use = “visible”
Use = “visible”+”student activity”
17
Round 2: mCURL (Moodle Common Usage and Learning Analytics)
8 active CSU & 2 UC campuses – Co-chaired: John Whitmer, CO ATS and
Mike Haskell, Cal Poly SLO
Starting with same measures of adoption, prioritizing “wish list” of more advanced analytics
Local database conventions and campus practices make accurate comps. challenging
18
Faculty LMS Adoption
How many faculty are using the LMS in one or more course sections?
19
mCURL Next Steps
Refine queries for accurate comparative course and student adoption measures
Select additional queries: depth and breadth of use – # tools used – # students in each section – frequency of use
Create repositories for campuses to share unique local queries
20
Blackboard Analytics for Learn (A4L)
CSU ATS Co-Lab Agreement – working together– Functionality: from early alerts/course reporting
to institutional-level analytics– Up to 4 campuses participating (3 confirmed)– Period: December 2012-December 2013– Individual campus Scope of Work for setup of
infrastructure and services
Kick-off meeting next week
21
Co-Lab Goals1. Develop methodologies and processes to identify, aggregate,
and transform LMS usage data into information for analytics.
2. Improve campus usage of learning analytics for decision-making for student success, curriculum improvement, and technical services.
3. Create shared measures, database reports, and algorithms, drawing on campus best practices and research innovations.
4. Increase campus awareness of applications and technical tools.
5. Document campus efforts and disseminate to other campuses.
6. Provide professional development in learning analytics.22
Student at a Glance
23
Instructor at a Glance
24
Dean Dashboard
25
