UMBC Blackboard Usability Testing Spring 2002. Overview  Usability Testing Basics  UMBC Blackboard Usability Testing  Highlights Video  Sneak Peek:

  • View
    214

  • Download
    1

Embed Size (px)

Text of UMBC Blackboard Usability Testing Spring 2002. Overview  Usability Testing Basics  UMBC...

  • Slide 1
  • UMBC Blackboard Usability Testing Spring 2002
  • Slide 2
  • Overview Usability Testing Basics UMBC Blackboard Usability Testing Highlights Video Sneak Peek: Blackboard Version 6 Next Web Usability Study: myUMBC
  • Slide 3
  • Usability Defined Making whatever youre working on easier to use for who ever is going to use it. An element of design, focusing on can this be used when its done? rather than just making it look good. Steve Krug, 2001 WebTalkGuys interview
  • Slide 4
  • Must Read One of Amazons top 200 books. Companion site www.sensible.com Sample chapter Sample script Workshops & updates
  • Slide 5
  • Usability Basics Instead of imagining what users want, you observe, interview them and prototype their needs. Develop content, design and programming strategy by reconciling observed user needs with sponsors desired objectives. Test user tasks and expectations with low-cost prototypes (e.g., paper, PhotoShop) Users dont make mistakes; websites do
  • Slide 6
  • Pros & Cons The alternative of using designers and programmers as proof of concept tools is expensive and demoralizing. May seem to add time (and cost) to a production schedule, but constantly revising is expensive, too.
  • Slide 7
  • Blackboard
  • Slide 8
  • Blackboard @ UMBC UMBC Faculty = 616 full-time, 318 part-time Students = 11,237 Staff = 415 Blackboard Courses & organizations = 699 (238 Sp02) Number of users = 12,141 Number of Instructors & managers = 610 Average hits per day = 35,585
  • Slide 9
  • User/Task Analysis
  • Slide 10
  • Questionnaire
  • Slide 11
  • User : Instructor 1 Interview Date: 2/18/02 Name: Robin Sex: Female Role: Instructor
  • Slide 12
  • Typical Usage & Findings Post syllabus, handouts, announcements Collect and return papers via Digital DropBox Discussion board Would like to be able to do bulk deletes Likes email discussions, students dont use Blackboard forums Grade book inflexible Likes Word editing and feels like she could duplicate many BB features with other tools. Usage Findings
  • Slide 13
  • User : Instructor 2 Interview Date: 2/18/02 Name: Kristie Sex: Female Role: Instructor
  • Slide 14
  • Typical Usage & Findings Posting to and usage of course syllabus Posting to and usage of course schedules Posting of assignments Use of Grade Book Accessing documents in Drop Box Usage of discussion board Frequently uses 2 or more browser windows along with other apps (such as Word) Makes heavy use of her own external site More reliable Easier to back up Always accessible Uses BB to post copyrighted material Uses MS Word for spell checking Wants fewer buttons Sees BB as an enhancement to course. Usage Findings
  • Slide 15
  • User : Instructor 3 Interview Date: 2/18/02 Name: Karin Sex: Female Role: Instructor
  • Slide 16
  • Typical Usage & Findings Posting of assignments Retrieval of completed work Assistance in assigning and collecting work Support of classroom instruction Communication Homework assignment hand-in Wants: More flexible search Check boxes with Digital Drop Box to indicate document status Replication from computer-to-computer Batch processing Additional levels of admin for TAs Usage Findings
  • Slide 17
  • User : Student 1 Interview Date: 2/12/02 Name: Roger Sex: Male Role: Student
  • Slide 18
  • Typical Usage and Findings Checking: Assignments Schedules Announcements Discussion board for assignments Downloads course docs. Checks grades Interface didnt match mental model Primary functions: Discussion boards Digital drop box Not familiar with many features Uses Word for spell checking Backs up replies in Word. Wants to save docs,not look at them in browser. Easy to use, convenient Usage Findings
  • Slide 19
  • User : Student 2 Interview Date: 2/13/02 Name: Chris Sex: Male Role: Student
  • Slide 20
  • Typical Usage/Findings Mandatory use Communication Drop Box Retrieving and submitting assignments Likes interface: User friendly Easy to navigate Quick to learn Got several errors Usage Findings
  • Slide 21
  • User : Student 3 Interview Date: 2/13/02 Name: Anita Sex: Female Role: Student
  • Slide 22
  • Typical Usage/Findings Checks course materials Completes homework Posts questions to discussion board about homework Encountered server errors Confusion about multiple Discussion Boards Clicks on all buttons to find what shes looking for Does not check site frequently/daily Usage Findings
  • Slide 23
  • User : Student 4 Interview Date: 2/13/02 Name: Sara Sex: Female Role: Student
  • Slide 24
  • Typical usage/findings Checks announcements and course material Do homework Use discussion board Maximum use: 5X per week Email Works from own computer Uses multiple windows to avoid going back and forth Confused by multiple discussion boards Missed announcements burried in lots of text. Usage Findings
  • Slide 25
  • User/Task Matrix
  • Slide 26
  • Typical Environments University public computer lab workstation (Students) Home computer (students & instructors) Modem Cable Office computer (instructors)
  • Slide 27
  • User Goals Earn a good grade Learn about course topic Complete requirement Complete course Research topics Use the Web to enhance course Enable learning Design courses Facilitate administration Foster interaction Provide supplementary research Manage large classes with multiple sections Create online archive Organize materials Personal enrichment Students Instructors
  • Slide 28
  • Task Flow
  • Slide 29
  • User Tasks View course work Read notes Ask questions Share information Turn in assignments Do research View Schedules View syllabus Find required readings Get help Download documents Deliver course work Track student progress Monitor participation Track attendance Track grades Archive course info Communicate with students Conduct discussions and lectures Review work Grade work Edit assignments, course work, schedules, etc. Make announcements Post schedules, syllabus, etc. Manage access Students Instructors
  • Slide 30
  • User Tests
  • Slide 31
  • Methodology Conducted at UMBC Volunteer participants 5 students 4 instructors Varying experience Test team: Moderator- lead subject through tasks Observer- noted subject responses/actions Protocol: Subjects in front of computer, asked to complete tasks Subjects encouraged to verbalize thoughts Responses recorded by Observer
  • Slide 32
  • Usability Test Scripts
  • Slide 33
  • Summary Easy to learn and use Consistent navigation helps Operational scalability issues Features for power users Efficiencies Menu wording/functions
  • Slide 34
  • Summary: Task Efficiencies Faculty spend more time than students, complained about repetitive and boring tasks: Enrolling more than one student in course or group Managing documents via the Digital Drop Box Managing own content Inflexibility of gradebook
  • Slide 35
  • Summary: Interface labeling Students main issue: misunderstanding important menus and functions: Tools a vague term Digital Drop Box and Grades sections use unclear Send vs. Drop metaphor problematic Confusion over Assessment & Pool manager Editable navigation
  • Slide 36
  • Subjects 5 students Graduate, Psychology Undergraduate, Computer Science Undergraduate, Anthropology Graduate Student, Information Systems Undergraduate, Computer Science 4 Instructors Geography, expert Modern Languages, advanced beginner English, advanced beginner Chemical Engineering, beginner
  • Slide 37
  • User Testing Results Students
  • Slide 38
  • Digital Drop Box Observation: Difficult for students Confused with submit and send Confirmation/warning ignored. Recommendation: Language should be reconsidered to clear up confusion Submit should send with first click Rating: UNUSABLE
  • Slide 39
  • Tools Language Observation: Confusion over Tools button Frequently ignored When used, not able to locate information Recommendation: Use more descriptive language in labels Allow instructors to move frequently accessed functionality Create more meaningful name based on user insight. Rating: SEVERE
  • Slide 40
  • Discussion Board Observation: Confusing to students Unclear about role of discussion board Overwhelmed with active boards Recommendation: Work on hierarchy Provide multiple levels of configurability. Clearly differentiate Group Discussions from general class discussions Be able to rename Group as Lab Rating: MODERATE to SEVERE
  • Slide 41
  • Navigation Observation: Students go to staff information instead of communications for email Recommendation: Provide clearly-labeled email or contact instructor buttons Rating: MODERATE
  • Slide 42