Usability Test of the Missouri Western R25 Webviewer
Usability Test of the Missouri Western R25 WebViewerYaser KhaniMichael NelsonStewart Hutchison
Reserve 25 (R25) WebViewer The R25 WebViewer is a web-based program widely used by college campuses, primarily used for event management & scheduling, allowing students to:
View course information, events and activitiesAccess the schedule of indoor and outdoor facilitiesReserve campus space for student organizations or personal useBusiness GoalsOur primary goal is to encourage students to become more involved with campus activities. Informed by usability testing and research, our event scheduling system can be modified for a more user-friendly and accessible experience. As a result, students will be encouraged to initiate and join campus organizations, fostering an active and cooperative campus environment.Two Locations:
Calendar of EventsWebViewer / Campus Calendar
A full-time undergraduate student who spends the majority of their time on campus.
A highly motivated and active student who initiates activities with their peers.
A student who belongs to an organization on campus or regularly attends athletic events.These are desired personality characteristics for participants in our studyParticipant ScreenerHow many credits are you currently enrolled? Which of the following age ranges are you in?Are you currently living on campus?Have you ever accessed our online Calendar of Events to view an activity or search course/activity information? Have you ever initiated or created an event on campus with the Center for Student Engagement?Do you currently belong to a student organization?Profile Summary of Five Users
Participant IntroductionThank you for taking the time to meet with us today. Were a graduate student group interested in better understanding how people interact with the R25 WebViewer Scheduling software and have asked for your help. So we can explore the software together, you will receive specific directions for each task scenario. You can withdraw from this study at any time, and will still receive the extra credit points from your instructor.If you have any questions, please feel free to ask the moderator. No personally identifying information is required and only the graduate student team will have direct access to your responses. You are encouraged to think aloud as you complete the scenarios, describing your thoughts and feelings to the extent you feel comfortable.The Missouri Western website is often busy and may take time to load new pages, so there may be some significant clicking-and-waiting. We thank you again for your time and patience.
Task ScenariosYou are curious about todays events on campusYou have decided to initiate an activity on campus and would like to submit an event requestYou are a softball player and would like to participate in a tournament game on February 1, 2014. You will need to contact the event organizer by obtaining their phone numberYou have enrolled in a college writing course (ENG 104 01), which begins Jan. 14 on Tuesday and Thursdays. You still need to know the time and location of your classYou are planning to organize a conference meeting in one of the Fulkerson rooms the last week of April, 2014. You require a room that will seat 125 people
15 Tasks Locate today's date on the calendar to view the most current event listings. Select any event for more information. Locate the option to view all requests that you have already made. Select the option to submit a request for an event. Locate the ability to view only events held at the Griffon Indoor Sports Complex. Navigate to the specified date of the softball tournament. Locate the event for more information and confirm the requestors phone number. Locate the option to display both events and course information. Navigate to your course using the calendar of events. Select your class to determine the time and location. Locate the option to display building locations only. Navigate to the desired week to schedule your conference. Find and select one of the Fulkerson rooms for more information. Select the room again above the calendar to determine the seating capacity. (Repeat this step until you have found a suitable room for your audience).15. Determine an available time and select the time slot to request a conference.
Testing ProcedureTwo minutes anticipated per task; participants given 30 minutes total.One assist was given for every minute spent on a singe task.All testing was completed on a laptop computer (Toshiba Satellite P855).Screen movements and audio were recorded with Camtasia Studio 8.1.Remote testing was conducted via Google Hangouts.
Scenario 1 Scenario 2Scenario 3Scenario 5Time on Task (seconds)Scenario 4
#5: Locate the ability to view only events held at the Griffon Indoor Sports Complex#11: Locate the option to display building locations onlyMean and Variance#4: Select the option to submit a request for an event#14: Select the room above the calendarTask #4
Select the option to submit a request for an event
Video 1/33:06Mean = 4 secondsVariance = 3 seconds
Task #4 SummaryThe first two users began scrolled-down, where the top menu was not visible.Locating My Requests took considerably longer for users #1 and #2 (72 and 103 seconds).Once accessed, all users located New Request in 6 seconds or less.
Locate the ability to view only events held at the Griffon Indoor Sports Complex
Video 2/33:14Mean = 62.8 secondsVariance = 279 seconds
Task #5 SummaryUser #4 took considerably longer to complete task 5 (283 seconds).After selecting Locations, user #4 did not scroll down to view Filter options until prompted by moderator.User #3 also selected Locations first, but recovered quickly by navigating the Filter options (16 seconds).
The ability to view = Filters?Task #11
Locate the option to display building locations only
Video 3/32:26Mean = 63.6 secondsVariance = 189 seconds
Task #11 SummaryAll users began task scrolled-down, where Locations was not immediately visible.Users #2 & #3 required a prompt to look away from the Filters optionUsers #1, #4, and #5 completed the task under 17 seconds. Possible learning effect at this point, where the Locations option had been previously explored.
The option to display = Locations?
XSUS System Usability Scale1. I think that I would like to use this website frequently. 2. I found this website unnecessarily complex. 3. I thought this website was easy to use.4. I think that I would need assistance to be able to use this website.5. I found the various functions in this website were well integrated. 6. I thought there was too much inconsistency in this website. 7. I would imagine that most people would learn to use this website very quickly. 8. I found this website very cumbersome/awkward to use. 9. I felt very confident using this website. 10. I needed to learn a lot of things before I could get going with this website. All questions are scaled 1-5 (1 = Strongly disagree, 5 = Strongly agree)Scoring the SUSFor odd items: subtract one from the user response.For even-numbered items: subtract the user responses from 5.This scales all values from 0 to 4 (with four being the most positive response).Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100.
User 1 = 50User 2 = 47.50User 3 = 67.50User 4 = 70User 5 = 55A score of 68 is considered average (50% rank)http://www.measuringusability.com/sus.phpMeasure of perceived usabilityInterpretation of SUS Scoreshttp://www.measuringusability.com/sus.php
User #1User #5User #2User #3User #4Based on Jeff Sauros 500 studiesConclusion of SUS AnalysisParticipants average SUS Score = 50+47.5+67.5+70+55 = 290/5 = 58% This is below Sauros average: 68%Measure of perceived usability percentile ranking = 22%Severity Scale Dumas & RedishLevel 1: Prevents task completionLevel 2: Creates significant delay and frustration Level 3: Problems have a minor effect on usability Level 4: Subtle and possible enhancements/suggestions
Local (single page) vs. Global (entire website)Strengths The system did not have any major usability issuesThe system was usable for all participantsThe R25 WebViewer has detailed scheduling information including campus events and classesStrength Location of Filters
The Missouri Western R25 WebViewer uses the navigation menu on the left side of the screenAreas to Improve Text to Describe Event
The calendar does not use text to describe scheduled time slots/events on location pages (Global 3) RecommendationsInclude text descriptions of events on the calendar
Event AEvent BEvent HEvent CEvent DEvent FEvent EEvent GEvent IAreas to Improve The Disappearing Menu
Locations and My Requests
A persistent view of the menu bar while scrolling down the calendar of eventsAn example is given by Binghamton UniversityParticipants could not initially see the top menu bar when beginning tasks 5 and 11 (Global 2).
Having scrolled down, menu options remain visible above Areas to Improve Aesthetics Sorting through bolded information is strenuousThe color of the top navigation menu makes it easy to overlookViewing information as a grid(Global 3)
RecommendationsBold times instead of descriptionsChange the color of the text in the menu or the background of the menu so it is easily noticedNote about locations/additional featuresViewing information as a list is easier to use (have it be default setting)
Example Aesthetics N