Upload
emma-webster
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Media & Learning Design (M&LD)
Research & Evaluation
Presentation to M&LD Steering Committee
By Christos Anagiotos ([email protected])
& Phil Tietjen ([email protected])
April 4 2012
OUR CHARGE
Review prior approaches to evaluation within M&LD
Incorporate evaluation in new M&LD projects more systematically
Investigate state of the art in media & learning evaluation
Investigate what other Universities are doing in regards to the use of Media in online courses.
Evaluation elements
Learning outcomes Learning experience Usability
Research:Media enhances learning
Retention Transfer Cognitive Flexibility
Course Surveys
Course Surveys
Criminal Justice Public Administration
Sample Questions
By watching the library learning tutorials, I learned things I previously did not know about the library
These tutorials will help make my research easier
Video cases allowed me to relate to the content
Video cases helped me in doing my assignments for the course
Focus Groups
Focus Groups
World Campus orientation videos: focused groups with students
M&LD participants: Focused groups with Instructional Designers
Identified Problems
Problem = Low Response Rate
Surveys
Response =
•Embedded Evaluation
•Learning Analytics
Learning AnalyticsLearning analytics is the measurement, collection, analysis and Reportingof data and their contexts,
for purposes of understanding and optimizing learningand environments in which it occurs
Universities that are using Learning Analytics University of Phoenix Cabelas University Baylor University Sinclair Community College University of Baltimore Purdue University Regis University (Library’s Distance
Learning Department) University of Rutgers-Newark (Law
Library) Khan Academy
POTENTIAL OF LEARNING ANALYTICSA. Compare users (e.g. evaluation)B. Predict student performance
(Predictive Analytics)C. Understand student’s needsD. Identify media flaws E. Personalization of educational
material
What data can we collect from current WC sources
1. ANGEL
1. Outside ANGEL- Google analytics- Flash Media Server
What does ANGEL offer?
All data is connected to the student (PSU ID, IP address)
Individual analytics (very complicated to get group analytics)
Examples: Log in time, Log out time, Time
spend in each website, Items downloaded
Google Analytics (Outside ANGEL)
Collect anonymous information about the user
Data is connected to IP Address Data is NOT to the PSU ID
Records much more data than ANGEL The data is presented in a more user
friendly way
Media Flash server(M&LD Videos, Outside ANGEL)Collect anonymous information about
the user Data is connected to IP Address Data is NOT to the PSU ID
We can currently measure: Log in/ log out time Duration per visit, per visitor Streaming duration Play, pause hits
How to make sense of data collected?
EXAMPLES
Example 1: from Media Flash Server:
Course Ed. Leadership 802: Average Length of videos : 10
minutes Average watch time: 4 minutes
Example 1: Possible explanations The content is not valuable or useful to
the viewers The user already got the info from
other sources (readings, discussions etc)
Users are tired or bored after watching the same person talking for more than 4 min.
The content may not be clear enough to the user
Example 2
A video was watched 46 times by 12 users in 7 days.
Possible Explanations: High relevance to the user (e.g. used
for an assignment) Entertaining Confusing
Comparison of data collectedANGEL: Pros: Data connected to the user PSU ID Cons: Very limited amount of data, tough to use
Google Analytics (Outside ANGEL):Pros: Large amount of data & Great detailCons: Data not connected to the individual users’
PSU ID
Flash Media Server (Outside ANGEL):Pros: Decent amount of dataCons: Data for the videos ONLYData not connected to the individual users’ PSU ID
Combining the data we already collect
We can gather :
Data directly connected to each user (PSU ID) from the 3 sources
Group data Data for every activity in the course
website
OTHER FORMS OF DATA THAT WC DO NOT COLLECT
1. Social Network Analysis (Student networks)
2. Record student screens
1. Social Network Analysis (Student Networks)
Students’ social networks facilitate learning processes (Dawson, 2010).
These tools are making learner networking visible
Able to “see” (identify) students who are network-poor (apply interventions)
Visualization of Social Networks Analysis
2. Record student screens e.g. Team Viewer software
What’s next in Learning Analytics? Personalization of educational material Knewton - Pearsons partnership
(video):
(Knewton Adaptive Learning Platform).
Confidentiality issues
How much data we collect?
Students’ consent
Who has access to the data?
Recommendations
Coordinate with IDs to implement regular evaluations
Establish regular meetings with IDs to discuss and analyze results
Develop internal visualization-reporting tools
Make the connection to student performance
Publicize our findings, let people, outside PSU, know what we are doing in M&LD.
Some other ideas for evaluation
Thank You
Questions?