View
315
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Evaluation of the Current Learning Management System at JMUThe Students’ Perspective on Blackboard
Presented By
Eric JacobsMichael MayJason PooleJessica Robinson
IntroductionReview of Evaluation Process
Background
IntroductionThis evaluation plan was prepared by Eric Jacobs, Michael May, Jason Poole, and Jessica Robinson who are graduate students in the Educational Technology department at James Madison University. Coordinating efforts by Andrea Adams of the Center for Instructional Technology (CIT), and by Dr. Michele Estes of the Educational Technology department at James Madison University have helped advise this evaluation plan.
Blackboard is a learning management system that has the ability to act as the central hub for the curriculum. This has become increasingly apparent as distance learning programs depend on the features of this evaluand to manage course content and interact with students in a web-based environment. Blackboard’s diffusion wide, claiming more than 1,600 institutions using their LMS (Blackboard, 2011).
Purpose of evaluationReview of Evaluation Process
Stakeholders
Purpose
• To determine the needs that students have for learning management system.
• To identify the technologies that faculty and students are currently using in teaching and learning
• To identify the technologies that faculty and students are no long using in teaching and learning.
• Primary Stakeholders• Andrea Adams• Dr. Michele Estes
• Secondary Stakeholders• Evaluation team• Students of JMU
Purpose of evaluationReview of Evaluation Process
Decisions to be addressed
• To what degree does Blackboard meet the expectations and learning needs of undergraduate and graduate students at JMU?
• Do graduate and undergraduate students feel that Blackboard is a viable solution as a learning management system for JMU over the next 5 years?
• To what degree do the features in Blackboard meet the technological needs of the students at JMU?
Evaluation questionsReview of Evaluation Process
Needs related
What are students needs in regards to using a learning management system in their coursework?
What expectations do students have about learning management systems?
What technologies are students using both in Blackboard and in their day to day lives?
Do students feel that the technology in Blackboard is beneficial to their learning?
What features of Blackboard do students find obsolete or in need of updating?
What features in Blackboard do students engage with most often?
What features in Blackboard do students feel are the most beneficial?
Technology related
Features related
Methods of evaluationReview of Evaluation Process
Methods
• Cross-sectional Survey• Census• Interview
The SampleReview of Evaluation Process
• 2 undergraduate• 2 graduate• Purposive sample of Ed Tech Students
• ~20,000 students• Expected return rate ~2%
Sample for interviews
Sample for census survey
• Students in high Blackboard use courses
• 2,000 students or 10% of JMU enrollment
• Expected return rate 50%
Sample for census survey
InstrumentsReview of Evaluation Process
• Created and administered using Qualtrics.
• Recorded using Elluminate Live!
Surveys
Interviews
Evaluation planReview of Evaluation Process
Limitations
• Time Constraints• Return rate of survey data• Proximity of group members from JMU and each other
Logistics & Timeline
Budget
• Interviews and survey data completed in November
InterviewSummary #1
Interviewee DataGender: Female
Program of Study: Communications Major, Educational Media Minor
Class: Junior
General Info: Student has taken hybrid courses and is currently enrolled in an online course. Experience using Blackboard in high school.
+-
Positives
Negatives
• Access to grades• The more information on Blackboard
the better• E-mails of announcements• Organization of content• Discussion boards
• Announcements when viewed on Blackboard
• Instructors that don’t update Blackboard beyond syllabus
InterviewSummary #2
Interviewee DataGender: Female
Program of Study: Educational Technology
Class: Graduate
General Info: All classes were taken on Blackboard.
+-
Positives
Negatives
• Internal Blackboard email tools• Access to everything in central location
• Dislike wiki feature
InterviewSummary #3
Interviewee DataGender: Female
Program of Study: Middle School Education
Class: Junior
General Info: Overall positive experience with Blackboard
+-
Positives
Negatives
• All important documents in one place• Online grade book• Roster for contacting classmates
• Lots of features she doesn’t know how to use
• Inconsistency among JMU faculty
InterviewSummary #4
Interviewee DataGender: Female
Program of Study: Educational Technology
Class: Graduate
General Info: Overall positive experience using Blackboard. Almost all courses using Blackboard.
+-
Positives
Negatives
• Blackboard and Elluminate are perfect coupling
• Greater connection with course and instructor
• E-mail features, discussion board, grade feature
• Links and Resources
• Instructor implementation• Instructor Usage• Collaboration -- Real-time lacking• Group sharing is “clunky”• Synchronous info sharing (IM, chat)
Gender
Census Survey Data
# Answer Response %
1 Male 57 27%
2 Female 151 73%
Total 208 100%
1. Please select one: Male or Female
EducationalStanding
Census Survey Data
# Answer Response %
1 Freshman 51 25%
2 Sophomore 27 13%
3 Junior 41 20%
4 Senior 48 23%
5 Graduate Student 41 20%
Total 208 100%
2. Please select one: Specific Educational Standing
Census SurveyData
Census Survey Data
# Answer Response %
1 College of Buisness 25 12%
2 College of Communications 14 7%
3 College of Education 29 14%
4 College of Health and Human Services 31 15%
5 College of Humanities and Social Sciences
44 21%
6 College of Science, Technology, Engineering and Mathematics
47 23%
7 College of Visual and Performing Arts 8 4%
8 Undecided 10 5%
Total 208 100%
3. Please select one: Program of Study
Course interactions with BlackboardCensus Survey Data
# Answer Response %
1 I have taken a fully online course that used Blackboard
57 27%
2 I have taken a hybrid/blended course that meets online and in the classroom that used Blackboard
54 26%
3 I have taken face-to-face classroom courses that utilized Blackboard
191 92%
4 I have taken face-to-face classroom courses that do not utilize Blackboard
125 60%
4. Based on your course work at JMU, please describe your course interactions with Blackboard.
CommunicationToolsCensus Survey Data
# Answer Response %
1 social networks 177 85%
2 text messaging 196 94%
3 instant messaging 53 25%
4 email 200 96%
5 other 19 9%
5. Which technologies play a role in your daily communications?
Regularly used technologies
Census Survey Data
# Answer Response %
1 smartphone 109 55%
2 tablet 14 7%
3 laptop/desktop 194 98%
4 Other 29 15%
6. What technologies do you regularly use?
Would like social features in Blackboard
Census Survey Data
# Answer Response %
1 yes 64 32%
2 no 57 29%
3 unsure 76 39%
Total 197 100%
7. Would you like to see Blackboard have more social features for learning?
Blackboard fits expectationsCensus Survey Data
# Answer Response %
1 Strongly Disagree 4 2%
2 Disagree 11 6%
3 Neither Agree nor Disagree
46 23%
4 Agree 114 58%
5 Strongly Agree 22 11%
Total 197 100%
8. Blackboard fits my expectations of how a learning management system should work.
Overall experience with BlackboardCensus Survey Data
# Answer Response %
1 negative 3 2%
2 somewhat negative 15 8%
3 neutral 30 15%
4 somewhat positive 89 45%
5 positive 60 30%
Total 197 100%
9. Overall, how would you describe your experience using Blackboard?
How difficult is Blackboard
Census Survey Data
# Answer Response %
1 Very Difficult 2 1%
2 Somewhat Difficult 20 10%
3 Somewhat Easy 113 57%
4 Very Easy 62 31%
Total 197 100%
10. Overall, how difficult is Blackboard to use?
Satisfaction with Blackboard reliabilityCensus Survey Data
# Answer Response %
1 Very Dissatisfied 2 1%
2 Dissatisfied 13 7%
3 Neutral 33 17%
4 Satisfied 122 62%
5 Very Satisfied 26 13%
Total 196 100%
11. Please describe your satisfaction level of Blackboard as a reliable service.
Difficulty locating info in BlackboardCensus Survey Data
# Answer Response %
1 Very Difficult 2 1%
2 Difficult 24 12%
3 Neutral 62 32%
4 Easy 90 46%
5 Very Easy 18 9%
Total 196 100%
12. Please describe the difficulty in locating information within Blackboard.
How often do you access BlackboardCensus Survey Data
# Answer Response %
1 Never 0 0%
2 Less than Once a Month 0 0%
3 Once a Month 3 2%
4 2-3 Times a Month 3 2%
5 Once a Week 11 6%
6 2-3 Times a Week 72 37%
7 Daily 107 55%
Total 196 100%
13. How often do you access Blackboard?
Importance of features
Census Survey Data
# Question Not Very Important
Not Important
Important Very Important
Responses Mean
1 Email 16 13 80 78 187 3.18
2 Course Annoucements
2 5 64 116 187 3.57
3 Social Media
51 86 42 8 187 2.04
14. Please indicate the importance of the listed features for a learning management system
Importance of features
Census Survey Data
# Question Not Very Important
Not Important
Important Very Important
Responses Mean
1 Discussion Boards
16 33 94 44 187 2.89
2 Blogs 34 101 47 5 187 2.12
3 Wiki 41 96 42 8 187 2.09
15. Please indicate the importance of the listed features for a learning management system
Importance of features
Census Survey Data
16. Please indicate the importance of the listed features for a learning management system
# Question Not Very Important
Not Important
Important Very Important
Responses Mean
1 My Grades 2 1 22 162 187 3.84
2 Test 5 18 58 106 187 3.42
3 Rubrics and Learning Outcomes
1 10 59 117 187 3.56
Importance of features
Census Survey Data
# Question Not Very Important
Not Important
Important Very Important
Responses Mean
1 Assignments
2 1 41 143 187 3.74
2 Calendar 6 33 72 76 187 3.17
3 Groups 13 52 90 32 187 2.75
4 Journals 26 79 57 25 187 2.43
5 Learning Module
16 61 76 34 187 2.68
6 Survey 22 72 66 27 187 2.52
7 Podcasts 36 83 52 15 186 2.25
17. Please indicate the importance of the listed features for a learning management system
Most often accessed featuresCensus Survey Data
18. Which features do you access MOST often in Blackboard?
Most often accessed featuresCensus Survey Data
18. Which features do you access MOST often in Blackboard?
Least often accessed featuresCensus Survey Data
19. Which features do you access LEAST often in Blackboard?
Least often accessed featuresCensus Survey Data
19. Which features do you access LEAST often in Blackboard?
Importance of features
Census Survey Data
20. Blackboard meets my needs in my current courses.
# Answer Response %
1 Strongly Disagree 6 3%
2 Disagree 13 7%
3 Agree 108 58%
4 Strongly Agree 59 32%
Total 186 100%
Comments
Census Survey Data
Open comments section
• Received 35 comments
• 16 comments directly related to instructor knowledge and implementation of Blackboard
• 8 comments related to amount of tools available vs. the few tools used.
• Other less frequent comments• Smartphone support• Interface or navigation confusion• Dislike changes from previous
versions
Comments
Census Survey Data
Example comments
“I think that blackboard does not need to be overloaded with features. From my experience, most of the features in blackboard are not used because they are not very important for a classroom setting. Blackboard is useful because it allows an instructor to communicate with his or her students and students to communicate with their teacher. There is no need to improve blackboard or add more features to it because it already does what it does well. In fact, I find that having too many features makes programs stressful and difficult to use because they demand too much of users. I don't want a comprehensive life planner and entertainment center with blackboard. It already does what I ask of it: It provides a way for my teachers to communicate with me about upcoming assignments and grades. It does this very well.”
Comments
Census Survey Data
Example comments
“In my experience, use of blackboard as a learning management tool largely depends upon the instructor. Only a few of my classes have used discussion board and only one has used Elluminate Live! For most classes, blackboard serves as a way to distribute class documents, announcements, and contact information. Therefore, most of my use of blackboard has been with simple tools, which is really all I need blackboard for.”
“Some features are redundant and confusing, such as Journals and Online Journal. There is a lack of uniformity between how different professors use different tools. It's very clunky and clumsy. I wish it had a more seamless user interface, like facebook.”
Conclusions &Recommendations
Recommendations
Conclusions
• Students are overall pleased with Blackboard as a tool.
• Many of Blackboards features are unused in courses.
• Students feel instructors us of Blackboard is primary to its success in courses.
Option 1Keep Blackboard as JMU’s learning management system and focus attention on course implementation and faculty training.
Option 2Investigate alternative tools that better implement the functions most used by instructors in their courses.
Reflections of evaluation team +
-Positives
Negatives
• Recording interviews using Elluminate
• Data collection using Qualtrics
• Mass student email process
• Timing of survey
• Evaluators distance from JMU campus