Usability Test Report -AppendicesUsability Test Report of E-Learning Platform The UA/Pearson Learning Studio (Appendices) Date of Report: April 11, 2012 Last Update: April 15, 2012

1

Usability Test Report of E-Learning Platform

The UA/Pearson Learning Studio

(Appendices) Date of Report: April 11, 2012 Last Update: April 15, 2012 Date of Interviews: March 21, 2012 – April 4, 2012 Location of Test: Akron, Ohio Prepared for: John R. Savery

Director, Instructional Services Instructional Learning The University of Akron

Phone Number: 330-972-8238 Email: [email protected] Prepared by: Kathleen Kennedy

Executive Director, The Taylor Institute and Suarez Labs The University of Akron

Phone Number: 330-972-5758 Email: [email protected]

Confidential Research Report: This report contains personal information which cannot be published in whole or part without written permission from the research participants.

2

List of Appendices

Appendix A: Study Participants .......................................................................................... 3

Appendix B: Interview Guide ............................................................................................... 4

Appendix C: Student Evaluations .................................................................................... 24

Appendix D: Detailed Task Evaluation – Students .................................................... 30

Appendix E: Pre-‐Interview Online Survey Responses – Students ........................ 59

Appendix F: Additional Faculty Verbatims .................................................................. 69

3

Appendix A: Study Participants

This Appendix contains personal identification information about study participants.

This appendix has been removed per disclaimer requirements:

"Confidential Research Report: This report contains personal information which cannot be published in whole or part without written

4

Appendix B: Interview Guide

Pearson E-Learning Usability Study

FINAL

Date prepared: March 7, 2012

Prepared by: Kathleen Kennedy

Representatives of two cohorts will be recruited:

Cohort 1: Adult Students 12 completed interviews

Cohort 2: Instructors 5 completed interviews

For all cohorts: ability to travel to Usability Lab, use a computer mouse, read a computer monitor without medication (enlargement), and basic computer/internet literacy required.

Incentives

Cohort 1: $50 check, mailed to research participant upon completion of study.

Study Design

Multimodal usability analysis of functional website tracking feature visibility, clicks, scrolling, task completion, user feedback and other factors using scripted facilitator guidance.

Study Objective and Goals

E-Learning Platform Assessment by users and key stakeholders to determine if it is:

1. Useful: it does what we want it to do 2. Effective: it allows us to perform our tasks quickly and proficiently 3. Learnable: it enables us to learn how to do our tasks 4. Flexible: it has system integrity and tolerates our errors 5. Satisfying: it encourages us to continue using the interface

The study will focus on a single E-Learning Platform. There will not be comparison to an alternative E-Learning Platform or traditional face-to-face instruction.

5

Usability goals:

1. Determine how users interact with the system. 2. Overall satisfaction with ease of use and navigation. 3. Identify what makes intuitive sense and where users get lost. 4. Identify what is difficult for people to do and options for improvement.

User experience:

1. Identify aspects of the platform that make users feel distrustful or insecure. 2. Determine which aspects of the platform are liked and disliked.

Learner Tasks and Activities:

1. Can the user accomplish a key task? 2. Can the user find something specific?

Note: Specific tasks to be defined. Recommend focusing on tasks associated with core learning activities.

Comparative evaluation:

1. Score Pearson E-Learning.

Interview Participants

Student users 12

Instructor users 5

TOTAL USERS 17

6

Student Participant Requirements: Age: 25 or more years old

Gender: 6 to 8 women and 4 to 6 men in study – quota as needed.

Recruit student participants to represent a range of areas of study:

Major area of study Undergraduate Graduate

College of Arts and Sciences 1 1

College of Business Administration 1 1

College of Education 1 0

College of Creative & Professional Arts 1 0

College of Engineering 1 1

College of Health Sciences & Human Services

1 1

College of Creative & Professional Arts 1 0

Summit College 1 0

TOTAL 8 4

Exclude: Law School students and Foreign Language and Visual Arts majors.

E-Learning Experience: Must have taken an online credit-bearing course in the past 2 years OR have used an online Learning Management System (LMS) to access course content, and turn in assignments or take quizzes online as part of a University or College for credit course (UA or elsewhere; any Learning Management System including Springboard is okay) in the past 2 years.

Exclude students from Postsecondary Technical Education Bachelor and Masters (UA Dept. 5400 courses).

Require: Students who would consider enrolling in an online for credit course in the future.

7

Instructor Participant Requirements: A faculty member who has used an online Learning Management System (LMS) to deliver course content, grade assignments and/or set up quizzes online as part of a University or College for credit course they have taught (UA or elsewhere; any Learning Management System including Springboard is okay) in the past 2 years.

Department Name Contact info

College of Arts and Sciences

Invitation issued

College of Business Administration

College of Education

College of Engineering

Summit College

8

Technical Requirements Browser: Firefox 10.0.2

A media player will be pre-installed and tested for audio and video assets in the SOT Student Orientation Tutorial.

9

STUDENT E-Learning Discussion Guide Pre-interview Welcome Complete Consent Forms. We will be recording your screen through screen-sharing and our audio conversation. These recordings will be viewed by researchers to aid in analyzing the results but will not be made public in any form. Specific comments that you make may be used in published materials but will remain anonymous. If you sign here, you are agreeing that's ok.

Introduction and Setup Thank you for coming in. We're collecting feedback on an e-learning system. Basically, I'll ask you some questions and watch you use this system. This study will take about 45 min to 1 hour.

Explanation of Moderator and Participant Roles: Your job is really easy, you just have to be yourself and act as you naturally would. I did not design this system, and I don’t work for the company who created this system, I’m just collecting feedback. As you interact with the site, please be honest with your positive and negative thoughts. There’s no right or wrong answers. You are not being tested; the system is being tested.

Also, there is a viewing room behind me and some people who are working on this project are observing.

Background Questions Tell me a little about yourself. How far along are you in your college studies?

Do you use the internet regularly? How would you rate your level of web knowledge?

RECORD: Internet usage in hours per week. Internet knowledge on scale of 1 to 10.

Explore:

How many online courses have you taken?

When was the last time you took an online course?

Are you planning to take any online courses in the future?

Can you describe those online courses to me? How involved were you with these courses?

Why did you take these online courses?

How do you feel about online courses based on own personal experience with them?

Did you find the online courses you’ve taken matched your learning and studying style?

Think Aloud: Next I’d like to have you go to the eLearning system to do a few tasks.

10

There’s one thing that I would like you to do differently than normal. Please think aloud as you use the system today. For example, if you are reading something, read it out loud and feel free to say anything that comes to your mind as you read - for example that’s interesting or what in the world are they talking about? Also, talk out loud while you’re doing a task, so I can understand what you are thinking and doing. (e.g.: Now I’m going to try to use the site search.) And, if you get to a point where you would naturally stop reading, let me know.

I’ll let you decide when your task is complete. Just tell me when you’re finished. We’re going to be focusing on navigating through and using the system for normal tasks that a student needs to complete when they are taking online courses.

Do you have any questions before we continue? Okay, let’s go to the system.

STUDENT Tasks for Usability Test Introduction: If you were really signed up for a course this semester on this platform, you’d log in through Zipline using your normal UANet ID.

For the purposes of this session today, we just have a plain/generic login screen and we have set up a generic user name and password for you. You’ll see that we’ve enrolled you in a couple of courses.

We’ll mostly focus on one course—that one is indicated on the card with your login information. Test Scenario 1: Log-In and Navigation STATE = Logged out (Anonymous Role) CLEAR COOKIES

Login page displayed for participant.

Task 1.1: Log-on to site.

URL for login: http://ua.ecollege.com/internallogin.learn

[PROVIDE CARD with test User ID and Password and name of target course to participant]

STATE = Logged in (Student Role)

Explore page. What would you do first? RECORD MENTIONS OF TUTORIAL – unaided, aided.

RECORD Intent to use tutorial before using site.

Task 1.2: Course List page.

11

Reactions to the layout of the course list page.

What do you think this page is? RECORD IF USER RECOGNIZES THAT THESE ARE THE COURSES THEY ARE ENROLLED IN.

RECORD IF USER RECOGNIZES THAT COURSE NEAME ARE LINKS FOR COURSE ENTRY.

Task 1.3: Enter the first course 101…108 UA Online Course

Do not prompt to enter SOT Student Orientation Tutorial or Course – RECORD ACTION TAKEN.

IF STUDENT ACCESSES SOT GO TO Task 7.1

Task 1.4: Explore Course Home 101…108 UA Online Course

What is your initial reaction to this page?

What do you think you can do from this page?

Is there anything missing from this page?

Is there anything you do not understand?

Is there anything in the wrong place?

RECORD MENTIONS OF: Announcements area at top, Content area in center of page, expandable navigation menu at left, and “social learning widget” at right.

Task 1.5: Navigate to Module 1 called “About UA”

While you are on the Course Home screen, how would you navigate to the module called About UA from here?

Can you find any additional ways to get to that module from here?

Prompt if necessary to use the left navigation items or to use the arrows in the central content pane to flip between modules.

Test Scenario 2: Social Interaction - Classmates STATE = Logged in (Student Role)

Course = 101…108 UA Online Course

Task 2.1: Name some of your classmates

Does it look like you are the first student to log in?

Can you name some of your classmates who have completed activities within this course already?

12

Task 2.2: Social Learning Widget to check for activity

Identify who has recently posted to a discussion. Which discussion did they respond to?

Prompt if necessary to use the dropdown in the widget.

Task 2.3: Social Learning Widget to post a chat-type message

Post a quick message like “Hello, it is nice to be here.”

Prompt if necessary to type in the text box near the top of the Activity tab, then click Say it!.

Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course

Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course

Content screens: how natural is it to use the left tree for navigation; what is the reaction to the pages (just general--how attractive and easy to understand; don’t prompt for them but want to know if they have comments about aspects like text size, white space, colors, etc. being undesirable in any way )

Task 3.2: Explore discussion activity

Was it clear that this is a page where you needed to complete a discussion activity?

Task 3.3: Post response

Prompt: Post a response to the discussion board assignment.

Task 3.4: Post response to another student

Prompt: Post a response to the discussion board assignment.

Task 3.5: Submit a document to a dropbox.

Find the course assignment and submit your work. [DOCUMENT TO SUBMIT WILL BE ON DESKTOP.]

Dropbox: they will follow instructions within the module to submit to a dropbox.

Prompt if necessary to walk them through process.

Get reactions to this type of dropbox (how logical is the concept of going there and choosing which basket to place a document in; how difficult is it to use/get used to; how confident are they that the submission “went through”); while they are in there, ask them to check whether any of their previous assignment submissions have been returned to

13

them; ask what score was received and what written feedback was given (prompt as necessary)

Scenario 4: Course Functionality Module 2 called “UA College Life” 101…108 UA Online Course

Task 4.1: Instruct student to go to Module 2 called “UA College Life” 101…108 UA Online Course

Task 4.2: Take Course Quiz

Allow participant to complete quiz. When done explore reaction to the quiz introduction screen and quiz.

Quiz introduction screen: what do they generally think about it (does this page scare or overwhelm, are they comfortable with it, etc.); is all of the information they need found there (deadline, how many times you can take the quiz, etc.); what questions did they have when they looked at that screen (did they wonder if there would be instructions inside the quiz, or were those enough instructions for them); did they know how to start the quiz.

Quiz itself: did they feel confident that they knew what they had to do; was it clear that they submitted their quiz successfully; did they understand how they did/their score; how confident are they that they answered all of the questions/didn’t forget any

Task 4.3: Check grades (Gradebook)

Instruct student to check their grade in this course.

Identify the score you earned on some particular assignment that is showing a score in the gradebook.

Was this easy to use? Is it what you expected? Is there any thing missing?

Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages

How many unread messages do you have?

Read some of the unread messages and reply to one or two

RECORD how they choose to do that/the nav path they take)

Task 5.2: Check 101…108 UA Online Course for returned assignments

Have any of your assignments been graded and returned to you? identify how they are making the determinations about unread messages and returned assignments.

14

RECORD how they find out answer - central pane on the course home or social activity widget, do they access either the Gradebook or Dropbox tool or look for evidence on the Course Home.

Task 5.3: Access your returned assignment

Open your returned assignments and tell me what grade you received.

RECORD if able to locate and interpret grade and feedback by asking about the grade received, and what comments/feedback the instructor provided; get general reaction to the usability here if appropriate – how easy to check for returned assignments; would they check for returned assignments/what path would they normally take if this were a real class; do they usually follow the same path or would they sometimes check the social activity widget and other times click the Dropbox tool directly.

Scenario 6: Communicate with Instructor Task 6.1: Email Instructor

Instruct student to send an email to their instructor to ask a question about their grade.

Task 6.2: Exit course

Can student figure out how to exit the course versus exit the system.

If they click the system logout icon, note whether the warning popup is ignored or prompts the correct action; if necessary prompt them to click the house icon.

Scenario 7: SOT Student Orientation Tutorial

Task 7.1: Complete the SOT Student Orientation Tutorial course

RECORD start-end time – duration.

Allow participant to use the tutorial naturally, don’t prompt to “complete the whole thing”.

If student asks, just tell them to do as much or as little as they’d normally do; if they’d normally do absolutely nothing, ask them to spend a few minutes looking it over; if they are spending too much time, just stop them and ask the questions); how useful is it; what did they think of the structure of the course, format of the content, and subject matter of the content; name the things learned from the tutorial that they hadn’t already figured out on their own; do they think it should be required before getting access to “real” credit course; [note there are a lot of features covered in the SOT that most of our instructors wouldn’t use and we aren’t testing; it will be interesting to see if they flag those as the things they learned in the tutorial since we didn’t include them in the test; it would be good to ask about whether they would use those features and for what (would they initiate a chat with a group project member) and whether they think the instructor should incorporate some of those features regularly in their courses

15

Task 7.2: Exit SOT Student Orientation Tutorial

Instruct to return to the Courselist.

Record if they remember it is the house icon and prompt if necessary.

IF SOT WAS ACCESSED FIRST AT BEGINNING OF STUDY, GO TO Task 1.2

Scenario 8: Module 3/Tech Support If you needed help with your E-Learning course system what would you do?

How do you typically get assistance with university systems? chat, email or call. How do you prefer to get assistance?

Task 8.1: Locate Help Desk information

RECORD if student can locate the Help Desk info via the Tech Support link.

Note: the Pearson Help Desk is not located within ZipSupport);

Overall Experience Let’s go back to the entire system. Overall, how was your experience?

Is there anything that you would want to see changed/done differently in this system?

Particularly related to the tasks you completed today? If so explain.

Any other likes or dislikes that you’d like to mention?

Anything else you’d like to share with me or ask me?

Would it be okay for us to contact you if we have any follow up questions or information?

Close and Follow up: Well [participant name] that does it for us. I have a brief follow-up questionnaire I would like you to complete that will assist us in classifying the data and provide us with the information we need to send you your $50.00 check for participating in this study.

Follow-up questionnaire will include basic demographics, education and internet usage.

16

INSTRUCTOR E-Learning Discussion Guide Pre-interview Welcome Complete Consent Forms. We will be recording your screen through screen-sharing and our audio conversation. These recordings will be viewed by researchers to aid in analyzing the results but will not be made public in any form. Specific comments that you make may be used in published materials but will remain anonymous. If you sign here, you are agreeing that's ok.

Introduction and Setup

Thank you for coming in. We're collecting feedback on an e-learning system. Basically, I'll ask you some questions and then watch you use this system. This study will take about 45 min to 1 hour.

Explanation of Moderator and Participant Roles: Your job is really easy, you just have to be yourself and act as you naturally would. I did not design this system, and I don’t work for the company who created this system, I’m just collecting feedback. As you interact with the system, please be honest with your positive and negative thoughts. There’s no right or wrong answers. You are not being tested; the system is being tested.

Also, there is a viewing room behind me and some people who are working on this project are observing.

Background Questions

Tell me a little about yourself. What is your current position?

Do you use the internet regularly? How would you rate your level of web knowledge?

RECORD: Internet usage in hours per week. Internet knowledge on scale of 1 to 10.

Explore:

How many online courses have you taught (if any)? When/how long ago?

Do you intend to teach courses online in the future?

17

How many online courses have you taken, as a student (if any)? When/how long ago?

Do you intend to take courses online in the future?

How do you feel about online courses based on own personal experience with them?

How well do online courses match your own teaching style?

How important is the e-learning platform on the e-learning course quality and your ability to teach?

Think Aloud: Next I’d like to have you go to the eLearning system to do a few tasks.

There’s one thing that I would like you to do differently than normal. Please think aloud as you use the system today. For example, if you are reading something, read it out loud and feel free to say anything that comes to your mind as you read - for example that’s interesting or what in the world are they talking about? Also, talk out loud while you’re doing a task, so I can understand what you are thinking and doing. (e.g.: Now I’m going to try to use the site search.) And, if you get to a point where you would naturally stop reading, let me know.

I’ll let you decide when your task is complete. Just tell me when you’re finished. We’re going to be focusing on navigating through and using the system.

Do you have any questions before we continue? Okay, let’s go to the system.

INSTRUCTOR Tasks for Usability Test Introduction: If you were really signed up to teach on this platform this semester, you’d log in through Zipline using your normal UANet ID.

For the purposes of this session today, we just have a plain/generic login screen and we have set up a generic user name and password for you.

We’ll mostly focus on one course—that one is indicated on the card with your login information.

We have set up an account for you and enrolled you in a few courses.

General scenario is that instructor is logging in to this site to participate in an online course they’re teaching this semester; in real life, they would have received training so they know how to make edits in their course, do their grading, and all of the other typical teaching and course management tasks

Test Scenario 1: Log-In and Navigation STATE = Logged out (Anonymous Role) CLEAR COOKIES

18

Login page displayed for participant.

Task 1.1: Log-on to site.

URL for login: http://ua.ecollege.com/internallogin.learn

[PROVIDE CARD with test User ID and Password and name of target course to participant]

STATE = Logged in (Instructor Role)

Explore page. What would you do first? RECORD MENTIONS OF TUTORIAL AND TRAINING – unaided, aided.

RECORD Intent to use tutorial before using site.

STATE = Logged in (Instructor Role)

Task 1.2: Course List page.

Reactions to the layout of the course list page.

What do you think this page is? RECORD IF USER RECOGNIZES THAT THESE ARE THE COURSES THEY ARE TEACHING.

RECORD IF USER RECOGNIZES THAT COURSE NAMES ARE LINKS FOR COURSE ENTRY.

Advise that in real life courses they are teaching will appear under a heading named for the current semester.

Task 1.3: Explore Course Home 101…108 UA Online Course

What is your initial reaction to this page?

What do you think you can do from this page?

Is there anything missing from this page?

Is there anything you do not understand?

Is there anything in the wrong place?

RECORD MENTIONS OF: Announcements area at top, Content area in center of page, expandable navigation menu at left, and “social learning widget” at right.

19

Scenerio 2: SOT Student Orientation Tutorial Task 2.1: Review and Evaluate SOT

RECORD comments and prompt for what would improve this student tutorial. Is there anything missing? What else should be done to support student success.

Scenerio 3: IDT Instructional Design Tutorial Task 3.1: IDT Instructional Design Tutorial

Would you normally use something like this before you attempt to work on a new system? Or when you run into things they don’t know how to do?

Prior to using a system like this would you prefer online or live instructor training? Or both?

Ask instructor to access the IDT Instructional Design Tutorial (If needed prompt to click a course name to get started.) Look at IDT for approximately 10 minutes.

GUIDE participant through the Using Course Tools module with the intent to learn how to use these tools (if necessary, prompt them to click the module heading Using Course Tools in the left nav bar).

Prompt participant to watch one video on the Dropbox Tool screen For Gradebook Tool, view transcripts (link provided below each video).

Which parts of the tutorial would you personally use?

Do you prefer videos or written instructions?

How long do they estimate it would take them to complete the whole course?

How long do you think it should require to complete a tutorial for this e-learning system?

Explore how do instructor feels about the aesthetics, difficulty level of the content, etc.

Just based on what you have seen, how would you rate the overall value of this course?

Task 2.2: IDT Instructional Design Tutorial – One-Page Resource Repository

Ask participant to find the one-page resource repository that contains task-specific instructions in video and written form. [target module heading is VIDEO TUTORIALS]

Is this useful? Would you use this?

Task 2.3: Exit IDT and return to course list.

If they incorrectly click the system logout icon, note whether the warning popup is ignored or prompts the correct action; if necessary prompt them to click the house icon.

Scenario 3: Target Course that you are teaching Task 3.1: Access the target course – name on card

20

If necessary prompt them to click course name.

What is the first thing you would do normally?

Check for understanding of the structure of the course as well as structure of individual modules; check for understanding of layout of this course home screen/what each section of the screen has to offer (Announcements area at top, Content area in center of page, expandable navigation menu at left, and “social learning widget” at right); get reactions (what do they like, what will be useful, what will take some getting used to, if they compare to Facebook or Twitter ask whether they currently use those tools themselves and if incorporated into instruction.

Task 3.2: Check for recent Student Activity – discussions

Identify which students have recently posted to a discussion.

Prompt if necessary to do this using the Social Activity widget and/or the central pane.

Task 3.3: Check for recent Student Activity – reply to student discussion

Reply to a student who recently posted to a discussion.

If necessary prompt by explaining several choices: click through from a post displaying in the activity widget, central pane where unread messages are indicated, and/or left nav to manually check for unread posts); get general reactions to the process of reading and replying as they are doing it

Task 3.4: Check for recent Student Activity – submitted assignments

Identify which students have recently submitted an assignment.

If necessary, prompt to filter the social activity widget using the dropdown to look for certain kinds of activities.

Task 3.5: Post chat-type message in social activity widget on Course Home

Task 2.3: Social Learning Widget to post a chat-type message

Post a quick message like “I am thrilled with the quality of the work you have all been doing” or “don’t forget the deadline for your group project is this Friday.”

Prompt if necessary to go back to the Course Home to find the widget at the upper right, then type the message in the text box near the top of the Activity tab, then click Say it!.

Task 3.6: Navigation to Discussion Boards

Ask Instructor to identify as many ways as they can to navigated to discussion boards that contain unread messages (if necessary prompt by explaining several choices: click through from a post displaying in the activity widget, central pane where unread messages are indicated, and/or left nav to manually check for unread posts)

What do you think of all of the choices for monitoring what activity has been taking place within your course?

21

Do you feel like this will enhance interactions with/between students?

Does it seem like the system is set up so that the things you need to do while teaching can be accomplished efficiently?

Scenerio 4: Announcements

Task 4.1: Edit the existing announcement.

Task 4.2: Add a new announcement.

[If necessary, prompt to switch to “author” mode by clicking the author tab at the top of the left nav pane; you can even assist by directing to the “add announcement” and item “edit” icons if necessary.]

Scenario 5: Course Content and Schedule

Task 5.1: Edit the text content of one or more screens in the Target Course.

Task 5.2: Change the dates associated with one of the content screens.

Prompt to click the Author tab to get into editing mode, then Toolbox tab at top then they can change the dates associated with this screen of content (if necessary prompt to click Edit Schedule, put in some dates and save); get reactions and comments.

Task 5.3: Send an email to your entire class

Instruct participate to prepare an email to all class members (select “All Class Members” as the recipient and Add, then write in a subject and message).

No need to send email but it won’t hurt anything if they do.

Scenario 6: Receiving and grading online assignments Probe for their preferred method for receiving and grading online assignments.

Task 6.1: Find and open the Target Course Dropbox

Do you find the Dropbox screen easy to understand?

Task 6.2: Find the assignments that need to be graded.

Is it easy to determine what needs to be graded?

22

Overall Dropbox evaluation – What do they like/dislike? Any questions about what is happening here?

If necessary point out the 3 tabs/buttons at top and have them explore different views (they are able to see submission.

Task 6.3: Grade an assignment

How easy was graded this assignment?

Task 6.4: Add a comment to an assignment

Task 6.5: Return the assignment to the student

Task 6.6 Find and open a Quiz

Prompt if needed (nav to a Quiz screen, click Author tab).

Task 6.7: Add an item to the quiz

Ask instructor to add an item to quiz (if necessary, prompt to switch to Author tab then near top of item list choose an item type from dropdown and click Add button)

Task 6.8: Find Gradebook and explore all views.

Prompt if needed to select Gradebook from top nav; use dropdown to choose a view then click Go; ask them to explore EVERY view in the dropdown (Grades to Date; Course Home; one of the modules; Item Summary; and Show All)

Task 6.9: While in Gradebook, explore the User Activity tab.

Prompt if needed to select a view from dropdown and click Go. They should explore both available views and click some of the clickable links (student name or module heading) to see what happens.

Note:

It is ok if they want to look at the Setup Gradebook tab, too, but it could take some training to use that. If they do access that tab, the best thing to do is probably to advise that when setting up the gradebook you essentially progress step-by-step fashion through the “sub tabs” labeled Current Setup through Preferences.

Task 6.10: Exit course/return to Courselist.

23

RECORD If participant remembers how to exit the course using the house icon versus exiting the system using the door. If they get the system logout dialogue, do they ignore it or does it actually direct them back to the right way of exiting the course?

Overall Experience Let’s go back to the entire system. Overall, how was your experience?

Is this something that you would use?

Is there anything that you would want to see changed/done differently in this system?

What would make this system more useful to you? To your students?

Which of the tasks would you normally perform and which tasks would you assign to a TA or GA?

Any other likes or dislikes that you’d like to mention?

Anything else you’d like to share with me or ask me?

Would it be okay for us to contact you if we have any follow up questions or information?

Close and Follow up: Thank you for your help.

24

Appendix C: Student Evaluations

Wednesday, March 21, 2012

8:00am Graduate Student, College of Business Administration (EL-01)

EL-01 had minimal difficulties with the site. He found it easy to navigate and felt that it was not very different from other online courses. He liked the messaging system and suggested that the site could be improved through an increase in visual aids (pictures, videos, etc.). “For what I’m seeing, it’s not very different from what’s out there presently for most courses but what I like about it is the way that it loads pictures and graphics. And I think that the graphics are heavy, you know some of the applications tend to have that heavy graphics. You might find it difficult to maneuver around or navigate but this looks ok.” (46:04-46:34) “The messaging system… I think that is actually unique. Most times you just have the names… you click on the messager’s name and it brings up a pop-up of where you can type but this one is not like that.” (46:56-47:15)

“I think there should be more graphics…pictures, video things like that. More color, it captures the student.” (47:47-4803)

Thursday, March 22, 2012

8:00am Graduate Student, College of Arts and Sciences (EL-02)

EL-02 had few difficulties with the site. He found it somewhat easy to navigate and felt that it would be ok to take a course from the site. He felt that problem areas included the dropbox and the discussion board.

“It appears that there is an extra step. On Springboard… you would click on dropbox and you would see the assignment…For this one you have to click on dropbox… it has like an extra step. For Springboard’s dropbox procedure you just click on dropbox and you just click on the assignment and you just browse the file and you just post and submit. This one has an extra step.” (21:20-22:16) “I could (visualize using a course on this system). Of course, it would take a couple days to get used to where everything is located. I just wish that there would be a discussion like here, icon, that’s how it’s mapped out on Springboard. Other than that, I could see myself using this.” (32:28-32:55) “This would help me. I don’t think it would be a problem at all… The way everything is mapped on here I wouldn’t have a problem taking an online course.” (33:37-33:53)

25

“Discussion, for example, is obscured in this module here. I mean, how many people are going to click on ‘College Life’ to see discussion on it. It would be best to make it readily visible in this area here.” (35:29-35:46)

9:00am Undergraduate Student, College of Engineering (EL-03)

EL-03 had minimal difficulties when using the site. She felt that it was user friendly and would not be opposed to taking an online course like the site. She suggested that the site could be improved through increasing the presence of the professor or at least indicating that the professor could be easily contacted if needed.

“I would rather click through a step by step than watch someone else do it. I feel like I would learn how to do it better if I’m actually going through.” (In regard to the online tutorial) (28:36-28:48) “Overall I think it would be a good way (to take an online course)... It’s pretty user friendly.” (29:29-29:33) “Moving where the buttons are or adding links so you can easily contact your professor.” (29:46-29:51) “I think that online because you don’t have that contact it’s really important it’s as many places it can be so that if you do need it it’s easy to find.” (Presence of instructor/link to instructor) (30:35-30:45)

10:00am Undergraduate Student, College of Creative and Professional Arts (EL-04)

EL-04 had some difficulties with the site. She felt that with a few changes she would be more inclined to using the site. She found the problem areas to include the back button and the help menu. She suggested that the site could be improved through more resources for the student (including video).

“It shows you the tool menu but I’m not sure if I understand anything about the tools menu from that part of ‘help.’ And I don’t like that you can’t click back on it… Especially if you need help. You’re already confused and then you want to go back and you can’t go back to where you were.” (27:30-27:56) Disc 2 Would you like to take an online course using this system?: “If it had some changes, yes, to the things that are confusing including the back button and, you know, the discussion…I guess if things were less confusing, then yes, I think it would be conducive because once you actually got into the part where-- here’s this topic, here’s your information, and a picture. --Here’s this topic and a quiz. And everything like that is very self explanatory. But getting help and kind of the messaging part of it seems most confusing to me. And as well as the saving thing- to automatically save during a quiz-- just in case you forgot.” (30:57-31:33) Disc 2 “The only other thing I would say is… maybe the instructor will do a …lecture that they tape or something of that nature. If there are any other things of that nature

26

that can help. I mean, I think the more resources the better since you are online and don’t have a live teacher to ask questions to.” (32:03-32:22) Disc 2

10:45am Undergraduate Student, College of Arts and Sciences (EL-05)

EL-05 encountered some difficulties using the site. He felt that with a few changes the site would be easier to navigate and said that he could learn to use it. He found problems with the email section, the icons/buttons, and felt that the site was not intuitive for students who were not internet savvy. He suggested that the site could be improved through changing the email, icons, and visual aids.

“I would definitely change the mail. It would be clear….Maybe color coded what was read and not read. …The fact that there were messages like one two, and then another one was pushed in and then another one was back here… I don’t understand why it was that way.” (30:16-30:36)

Overall- would you take an online course using this system? “I would… even though it’s… cumbersome. I would probably learn to use it and learn to live with it. I think one other thing I would really enjoy more is a lot more icons. Like, a little house. A grade book- show me a pad of paper… it’s easier to gravitate to my first instincts of my visual..” More graphics so he could move more quickly and a better understanding of what was going on with discussion tabs and felt that chat was useful because it was sitting right there and can simply look at it at his leisure and maybe learn about something he didn’t think of. (30:55-31:22) “I’d say a majority of the people that use this, at least when they first approach it, and anybody who really has problems with it, probably are not very well versed in online applications. So, the more pictures, the simpler…the better…. When I first started using Springboard, it was confusing until I got the hang of it.” (33:10-33:30)

1:00pm Graduate Student, College of Engineering (EL-06)

EL-06 found minimal difficulties with the site and felt that it was easy to navigate. She had difficulty with visualizing an actual online course but felt that she would enjoy taking an online course from the site. She suggested that the site could be improved through grouping the discussion section and the email section together with separate folders to delineate the different sections.

Would you enjoy taking an online course like this: “Yeah, but actually I couldn’t see the real course yet.” (36:18-36:26) “I would probably put the discussion part together… or somehow.. When you have the email you just want to go to one or two emails and check all those.” Email and discussion should be on the same thing but published into a separate folder. All the discussion together so it’s not all broken into pieces. It would be more convenient for her. She also mentioned that the house icon should be moved to the top of the screen. (36:49-37:12)

2:00pm Undergraduate Student, College of Health Sciences and Human Services (EL-07)

27

EL-07 encountered minimal difficulties with the site. She found it easy to navigate and felt the layout of the site was appropriate for an online course. She liked that the site included a grade book and suggested that the site could be improved through changing the house icon.

“I don’t think I’d really change anything at all. The layout is really good. Just maybe… well, the little house… I would just change that. But after that I think it would be really helpful.” She feels that it is a good system. (34:48-35:01)

“I would say as far as like taking a class, it’s nice to have the grade book there and it’s nice to be able to submit everything online so you don’t have to have paper copies.” (35:24-35:35)

Monday, March 26, 2012

1:00pm Undergraduate Student, College of Arts and Sciences (EL-08)

EL-08 encountered minimal difficulties with the site. He felt that the site was very similar to the “perfect site” he would use for an online class. He suggested that the site could be improved by moving the icons to an area where their function was more apparent and felt that the chat system could use some way that students could convey tone in their text. He suggested that increasing the professor’s presence would also greatly improve the site.

How does this compare to a perfect online course: “Very very similar. The only disparity I saw up to it was having the one set of buttons below in the bottom left hand corner… [It would make it] tremendously better. Because when I am looking for overall actions, I’m looking toward the top whereas I’m expecting more course related things to be toward the bottom.” Felt like he wanted to know the instructor’s expectations and what they are specifically looking for. (28:37-28:55)

How to make it better: “The only thing I could see that would be of interest to me would be not just a text chat but a video chat- especially having the ability to have a video chat with the instructor. Get real time- get expressions so I could get a better understanding of what the instructor was looking for.” The instructor would be out front and have more of a presence. (29:18-29:37)

“If I had the facial cues to explain the emotion behind the text, then I could understand it better.” He wants some way to add tone to the chat. (30:19-30:28)

Did you find anything difficult? “Not difficult… being the menu driven, it would take me a little bit of time to get used to how it is set up. But I would think within the first couple days time on the class that would be mastered quickly.” (31:18-31:32)


8:00am Undergraduate Student, College of Business Administration (EL-10)

28

EL-10 encountered some difficulties with the site. She felt that the site had elements of a good online course and would be adequate for her needs. She suggested that the site could be improved through opening it up to students a week in advance in order for students to navigate because online courses do take some time to understand.

Overall, took an online course, would it work for her? “Before the class I would… I mean hopefully it was up early enough that I could like look at it and see what may be a problem so that I would know what I need to know before.” About a week to navigate around would make her feel more comfortable. Changed or done differently? She thought it was adequate for her needs. (38:49-39:06)

10:00am Undergraduate Student, College of Education (EL-11)

EL-11 encountered few difficulties with the site. She felt that if the site was more like Springboard it would be easier to navigate. She liked that the chat section was on the first page and felt that it was accessible. She found problems with the navigation and felt that getting back to the home screen was difficult.

Would this be a good experience? “There are a few things that I would tweak to make it more like Springboard which we’re already used to. But, overall, I think it’s pretty easy to work through.” (44:46-45:00)

Change anything? “I would just have it be along in my Springboard already as one of my courses that are listed that I could just go to and then it takes me to the more in-depth site for that class but I’d rather it just all be together. “ She wouldn’t need to go to another site. (45:33-45:47)

“I really don’t like how you can’t click the thing and go home. And I really don’t like how you can’t get back from discussion board. And the middle thingy- it’s too much stuff.” (46:39-46:50)

“I do really like the chat thing right there on the first page. That’s handy” It’s simple and easy. (47:27-47:30)

12:00pm Graduate Student, College of Health Sciences and Human Services (EL-09)

EL-09 encountered few problems with the site. She felt that she would rather take a live course rather than a course online. She found problems with the navigation and felt the buttons were not big enough. She suggested that the site could be improved by increasing the size of the buttons.

“If I had to choose between this and what we are using now, I would stay with what we are using now- not just for the familiarity factor, but I didn’t have to look anything up to use it.” (43:22-43:30)

29

“If there were a live class and this- I would definitely take the live class. I’d probably get through it but it’s just not real intuitive so I don’t really like it.” (44:04-44:12)

Would you change anything?: “Nice big chunky buttons and no extra clicking. I want it to go where I need it to go with one click not click into a menu and click into a menu.” (44:21-44:32)

1:00pm Undergraduate Student, Summit College (EL-12)

EL-12 encountered minimal problems when using the site. She felt that any online course would take time to get used to navigating. She liked the grade sheet and felt it was useful because the calculations were already done for her. She said she would probably prefer this grade sheet (over Springboard) if it were more organized.

“Yeah, it would take me a while to become familiar with the layout of everything.” (a couple of days) (32:06-32:14)

“Well, I mean, everything is going to take time to learn. I’m just used to what I am working with now, but I wasn’t used to it when I first started it and I had to figure it out all for myself… I think it was fine it is just going to take time to learn.” (33:21-33:36)

“On the grade sheet- I did like that you didn’t personally yourself have to do the calculations- it had those for you. I don’t think it was as organized as it could have been.” Would probably prefer this grade sheet if it were more organized. (33:43-33:53)

30

Appendix D: Detailed Task Evaluation – Students

Note: The scale being used in this study measures the participant’s ability to complete a certain task. A ranking of one indicates a failed attempt while a ranking of five indicates a passing attempt, and a ranking of zero indicates “no attempt”.

Scale: Fail=1 Pass=5 Not Attempted=0 Wednesday, March 21, 2012

EL-01—8:00am Graduate Student, College of Business Administration

Test Scenario 1: Log-In and Navigation

Task 1.1: Log-on to site. No usability issue. Rank=5. Task 1.2: Course List page. No usability issue. Rank=5. Task 1.3: Enter the first course 101…108 UA Online Course. No usability issue. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course. Slight confusion. Rank=4. “I am seeing the course page. I don’t actually know what all of these are. Is this the outline of the course?... Putting one and two together it looks like the outline of the course but it is not explicitly stated.” (15:04-15:24). Anything missing? “Looking at it I don’t see, at least for every course, there is going to be an instructor.” (16:35-16:42). Task 1.5: Navigate to Module 1 called “About UA.” No usability issue however was slightly confused that the course was about UA. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity. No usability issue. Rank=5. Task 2.3: Social Learning Widget to post a chat-type message. Could not find the chat area. Did complete the task with help. Rank=2.

31

“It’s not coming up the way I expected it to come up.” (23:11-23:14). “Because until I clicked ‘main’ that was when it loaded the Java outlet….Maybe because I was used to chat systems and I felt that something that needed to load before being able to use it.” (23:50-24:03) Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. Required help to find the module. Rank=4. “Yeah, yeah. It’s straightforward… But I think this part is kind of hard… discussion.” (26:14-26:31). Task 3.2: Explore discussion activity. No usability issue. Rank=5. “I think for most visual …applications like this, I think it is best if you have lots of graphics. I think pictures really capture the imagination better than letters a lot of times. The mix is not very good. I think there should be a bit more (pictures).” (26:55-27:15) Task 3.3: Post response. No usability issues. Rank=5. Task 3.4: Post response to another student. Not attempted. Rank=0.

Task 3.5: Submit a document to a dropbox. No usability issues. Rank=5. (30:02) Found how to type the paper on the site and then quickly found how to upload it.

Test Scenario 4: Course Functionality Module 2 called “UA College Life” 101…108 UA Online Course Task 4.1: Instruct student to go to Module 2 called “UA College Life” 101…108 UA Online Course. No usability issues. Rank=5.

Task 4.2: Take Course Quiz. No usability issues. Rank=5. Task 4.3: Check grades (Gradebook). No usability issues. Rank=5. “I thought it was straightforward…Because it’s a quiz… correct questions -I got 3. So there is no point at- to show which of the questions you actually got wrong? (34:31-34:47). Would like to be able to see which questions he got wrong and comparative grading. “It also helps to know that ok, general performance was bad or average- I didn’t do badly.” (35:33-35:40). Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

32

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. Was not 100% sure if the message was read or un-read. Rank=4. “I don’t know why this is colored.” (39:25-39:27). Task 5.2: Check 101…108 UA Online Course for returned assignments. No usability issues. Rank=5. Task 5.3: Access your returned assignment. No usability issues. Rank=5. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issues. Rank=5. “It’s good... I like… this concept. You don’t …actually have to type instructor’s email address. Cause it seems by listing them here it’s already embedded and by just clicking it goes. And this is lovely. This is very good.” (42:13-42:31)

Task 6.2: Exit course. No usability issues. Rank=5.

Test Scenario 7: SOT Student Orientation Tutorial

Task 7.1: Complete the SOT Student Orientation Tutorial course. No usability issues. Rank=5. “The good thing about most applications is that it has what you call an FAQ.” (9:40-9:45)

Task 7.2: Exit SOT Student Orientation Tutorial. Could not find the exit button. Suggested that there should be an easy way to find home. Rank=1. About the home button: “It’s not very visualizing. It’s not very eye-catching.” (13:43-13:46)

Test Scenario 8: Module 3/Tech Support Task 8.1: Locate Help Desk information. No usability issues. Rank=5.


EL-02—8:00am Graduate Student, College of Arts and Sciences


Task 1.1: Log-on to site. No usability issue. Rank=5.

33

Task 1.2: Course List page. No usability issue. Rank=5. “I would look at the student orientation link first.” (5:50-5:54) Task 1.3: Enter the first course 101…108 UA Online Course. No usability issues. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issues. Rank=5. “It’s more personalized… the website is trying to tailor the content more to my preferences. I see what’s neat here is the chat messenger. There’s no need to look for it in one of the tabs up top.” (9:58-10:30). Task 1.5: Navigate to Module 1 called “About UA.” No usability issues. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity. Not attempted. Rank=0. Task 2.3: Social Learning Widget to post a chat-type message. Not attempted. Rank=0. Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. Task 3.2: Explore discussion activity. Was unsure if clicked the right link. Rank=4. Would prefer the discussion tab at the top. Task 3.3: Post response. No usability issues. Rank=5. Task 3.4: Post response to another student. Not attempted. Rank=0. Task 3.5: Submit a document to a dropbox. Slight difficulties. Rank=4.

(20:50-21:18) “I would click on this icon here: dropbox… I would look for the assignment… See submit assignment …and this field I would select.”

(21:20-22:16) “It appears that there is an extra step. On Springboard… you would click on dropbox and you would see the assignment…For this one you have to click on dropbox… it has like an extra step. For Springboard’s dropbox procedure

34

you just click on dropbox and you just click on the assignment and you just browse the file and you just post and submit. This one has an extra step.”


Task 4.2: Take Course Quiz. No usability issues. Rank=5. “I thought it was straightforward. Nothing confusing about it. It gave you the option to save your answers later if you… had to take a break which I know is important. And it was easy to… submit my answers.” (24:11-24:37) Task 4.3: Check grades (Gradebook). No usability issues. Rank=5. “I would have expected to see the weight of the grade.” (25:12-25:14) Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. No usability issues. Rank=5. Task 5.2: Check 101…108 UA Online Course for returned assignments. Could not find. Failed task. Rank=1. Task 5.3: Access your returned assignment. Not attempted. Rank=0. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issues. Rank=5.

“One drawback that I have with Springboard is that when you send email via Springboard it does not keep track of emails that you sent.” (29:11-29:25)



Task 7.1: Complete the SOT Student Orientation Tutorial course. No usability issues. Rank=5.

35

“I did not expect …FAQ’s to be posted…. I don’t see that on Springboard… (useful?) Absolutely. Because there will be some questions on ok how do I access certain information on the interface…” (8:00-8:22).

Task 7.2: Exit SOT Student Orientation Tutorial. No usability issues. Rank=5.


“You know, I was expecting to… in the left pane here to see the tabs listed like for example grade book, email. …Show each tab and there would be a link… of each tab where you would click on and it would it would list off the issues you may be having. Like, for example, doc sharing-I have a question about that but I’m having trouble looking up assistance for that.” (11:58-12:41).

“Now, I would change the table of content list here. I would… I would list the …names of the tabs on the tab bar here.. readily available for students to access.” (12:54-13:18)

EL-03—9:00am Undergraduate Student, College of Engineering


Task 1.1: Log-on to site. Slight usability issue- was frustrated that it was case sensitive. Rank=4. Task 1.2: Course List page. No usability issue. Rank=5. Task 1.3: Enter the first course 101…108 UA Online Course. No usability issues. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issues. Rank=5. “It looks pretty user friendly so far. It looks like I can see what my assignments would be.” (7:42-7:51) Anything missing? “I think that the professor’s information at the top of the page would be helpful just so that right away you can see if you need help who to contact.” (8:49-8:59) Task 1.5: Navigate to Module 1 called “About UA.” No usability issues. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5.

36

Task 2.2: Social Learning Widget to check for activity. No usability issue. Rank=5. Task 2.3: Social Learning Widget to post a chat-type message. Site was not functioning. Not attempted but understood the procedure. Rank=5. Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. No usability issue. Rank=5. “It would be nice if there would be a next button down here and you could just click an arrow… I would put it down in the right-hand corner/side somewhere along the bottom.” (12:51-13:06) Task 3.2: Explore discussion activity. No usability issue. Rank=5. Task 3.3: Post response. No usability issues. Rank=5. Task 3.4: Post response to another student. Not attempted. Rank=0. Task 3.5: Submit a document to a dropbox. Attempted to submit an assignment, but failed. Rank=1.

(14:23 –14:43) “I would click on the dropbox link, and then put in the appropriate basket, which I’m not sure what that is, I guess I will go back and look and see what it says.”

(14:52 - 14:55) “There is no dropbox labeled very short paper.”


Task 4.2: Take Course Quiz. No usability issues. Rank=5. “It looks a little bit confusing to me… I don’t understand what the ‘many multiple choice’ means. (16:39-16:48) Would it be useful to be able to see what you got wrong?: “At this point, yes, it might be nice to just have a link so you could go back and see. On a short quiz like this maybe not.” (17:36-17:48) Task 4.3: Check grades (Gradebook). No usability issues. Rank=5. Would like to see “If the grades are weighted, possibly the weighting.” (18:36-18:40)

37

“I don’t know how easy it is to contact the professor from here, but it would be nice to have a link that… if you had a question on a grade you could just hit email a professor or something like that.” (19:11-19:22) Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. Not attempted. Rank=0. Task 5.2: Check 101…108 UA Online Course for returned assignments. She said she was only guessing and would be frustrated that she would have to guess. Slight usability issue. Rank=4. Task 5.3: Access your returned assignment. No usability issue. Rank=5. What did you get on the assignment? “An 18. What that means I’m not sure. So it doesn’t really tell me out of how many, it doesn’t tell me what percent.” (21:24-21:34) Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issues. Rank=5.

“I don’t necessarily love the have to add him to the box. I’d rather like, when you click on it just like a little check mark appear or something.” (22:33-22:41)

Task 6.2: Exit course. No usability issues. Rank=5. “It was ok. The home button was clear. It should be at the top.” (24:58-25:03). She knew to look for a little house but felt that it was hidden.


Task 7.1: Complete the SOT Student Orientation Tutorial course. No usability issues. Rank=5. She was glad that she didn’t have to go through the whole tutorial. She could easily look through FAQs.

“Yeah, this is helpful.” (26:09-26:10)

“I think all the course tools seem pretty helpful. I don’t love videos inside of tutorials but… it seems like they would be helpful because it’s kind of just explaining things you’ll use every day. I don’t think achieving online success will be used.”(26:58-27:28)

She also felt that the taking exams and checking grades would be useful.

38

“’Ensuring you’re ready’ should maybe should have a different name. If it’s for your student profile… maybe it should just have a different name. ‘Your success’ doesn’t really say to me that this is where you can update your profile.” (27:55-28:11).



“I think first there should be like a… link you would click for frequently asked questions or troubleshooting and then maybe IM with a live person.” (23:50-24:05).

EL-04—10:00am Undergraduate Student, College of Creative and Professional Arts


Task 1.1: Log-on to site. No usability issue. Rank=5. Task 1.2: Course List page. No usability issue. Rank=5. “It seems pretty straightforward.” (6:17-6:20 Disc1) Task 1.3: Enter the first course 101…108 UA Online Course. No usability issue. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issue. Rank=5. Task 1.5: Navigate to Module 1 called “About UA.” No usability issue. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity. No usability issue. Rank=5. “It almost seems a little bit unprofessional to me to have…in a course setting it seems more like a social site rather than a class.” (16:14-16:24 Disc 2). Task 2.3: Social Learning Widget to post a chat-type message. Not attempted. Rank=0.

39

Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. Task 3.2: Explore discussion activity. No usability issues. Rank=5. Task 3.3: Post response. No usability issues. Rank=5. Task 3.4: Post response to another student. Not attempted. Rank=0. Task 3.5: Submit a document to a dropbox. She attempted to submit an assignment, and did submit it, however had many difficulties doing so. Rank=3.

(09:53 – 10:01 Disc 2) “It wasn’t easy but it wasn’t really that big of a deal to fix. I guess after doing it one time you figure that out.”

“Again, I feel like I need to be clicking the back button to get back to the assignment but in fact I cannot. I have to click on this assignment… (would like to be able to use the back button?) yeah, especially if you don’t know where you came from it makes it a little easier.” (8:24-8:38 Disc 2)

Test Scenario 4: Course Functionality Module 2 called “UA College Life” 101…108 UA Online Course Task 4.1: Instruct student to go to Module 2 called “UA College Life” 101…108 UA Online Course. No usability issue. Rank=5.

Task 4.2: Take Course Quiz. No usability issues. Rank=5. “I almost wish there was a way to automatically save it for you so you wouldn’t have to keep clicking on ‘save answers’… because if it does it automatically for you, you wouldn’t have to worry.” (20:4321:02 Disc 2) “I actually kind of like this. The true/false, multiple choice or the many multiple choice kind of breaks down the questions- how many you got correct. … I guess way better than I would have expected. (22:13-22:25). Task 4.3: Check grades (Gradebook). No usability issues. Rank=5. Was really impressed with the “grade to date.” Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. Unsure if she found unread discussions. Rank=3.

40

“I’m gonna guess six since they are all bolded kind of like in an email system if they’re bolded you haven’t read them yet... That was a guess.” (5:17-5:28 Disc 2) Task 5.2: Check 101…108 UA Online Course for returned assignments. Unsure if she found if they are returned or not. Rank=2. Task 5.3: Access your returned assignment. Not attempted. Rank=0. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issues. Rank=5. “I really like that feature. I wish we had that on Springboard.” (25:52-25:55).



Task 7.1: Complete the SOT Student Orientation Tutorial course. No usability issues. Rank=5.

“Oh I like the grade book items not yet graded, things not entered, extra credit- that would be helpful.” (7:05-7:16).

“Good I like it. It actually reminds me of Springboard a lot, which I like.” (7:42-7:45 Disc 1)

Task 7.2: Exit SOT Student Orientation Tutorial. Could not find how to get back to course home page without help. Failed task. Rank=1.

“The icons are like really small in the bottom left corner. They should maybe be at the top. If they were at the top and I saw a house I would probably be inclined to click on it. (3:44-3:52 Disc2)


EL-05 – 10:45 am Undergraduate Student, College of Arts and Sciences Test Scenario 1: Log-In and Navigation

Task 1.1: Log-on to site. No usability issue. Rank=5. Task 1.2: Course List page. No usability issue. Rank=5.

41

Task 1.3: Enter the first course 101…108 UA Online Course. No usability issue. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issue. Rank=5. Task 1.5: Navigate to Module 1 called “About UA.” No usability issue. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity. Task 2.3: Social Learning Widget to post a chat-type message. No usability issue. Rank=5. What would you use chat for? “Basically to talk about my grades and ask other students questions that I might not understand.” (9:48-9:54) Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. No usability issue. Rank=5. Task 3.2: Explore discussion activity. No usability issue. Rank=5. “It would be for people to interact about The University of Akron in general versus the other thing was for a class… specific.” (14:46-14:57). Task 3.3: Post response. No usability issue. Rank=5. Task 3.4: Post response to another student. Not attempted. Rank=0. Task 3.5: Submit a document to a dropbox. Submitted assignment with no difficulty. Rank=5.


Task 4.2: Take Course Quiz. No usability issue. Rank=5.

42

“The only thing I think that I might want is the correct answer to the one I got wrong.” (22:29-22:33). Task 4.3: Check grades (Gradebook). No usability issue. Rank=5. “I mean, maybe perhaps if this was completely online, which I think you said it was, an area where I could go and view my assignment and find out what was wrong -why I didn’t get 20 out of 20.” (18:52-19:03). Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. No usability issue. Rank=5. “I don’t know what this means, to have this envelope and this additional sign further in. I would think maybe that means I read it or something but I don’t know.” (11:01-11:12) Task 5.2: Check 101…108 UA Online Course for returned assignments. Failed the task. Could not find a graded assignment. Rank=1. Task 5.3: Access your returned assignment. Not attempted. Rank=0. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issue. Rank=5.

Task 6.2: Exit course. No usability issue. Rank=5.


Task 7.1: Complete the SOT Student Orientation Tutorial course. No usability issue. Rank=5.

“I would use the video for sure… Yeah, I’d say, you know, this is basically teaching me how to use the system. It’s very thorough. A lot of it is review for me but I think for some people this is totally necessary.” (7:01-7:21).

Task 7.2: Exit SOT Student Orientation Tutorial. No usability issue. Rank=5.

About the house button. “It would say something like, on Zipline, my home.” (8:03-8:08).

Test Scenario 8: Module 3/Tech Support

43

Task 8.1: Locate Help Desk information. No usability issue. Rank=5.

EL-06—1:00pm Graduate Student, College of Engineering



Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issue. Rank=5. Task 1.5: Navigate to Module 1 called “About UA.” No usability issue. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity. Not attempted. Rank=0. Task 2.3: Social Learning Widget to post a chat-type message. No usability issue. Rank=5. Who would you probably chat with? “Probably classmates or professor.” (17:05-17:07) Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. No usability issue. Rank=5. Task 3.2: Explore discussion activity. No usability issue. Rank=5. “Probably homework for class. So, probably more like formal than the one under the box earlier.”(19:51-20:03). “Actually, I think it should be just only one page so that I can look at it so I can see that it’s… that’s all my mails.” (20:20-20:27) Task 3.3: Post response. No usability issue. Rank=5.

44

Task 3.4: Post response to another student. No usability issue. Rank=5. Task 3.5: Submit a document to a dropbox.: She attempted to submit an assignment, and passed with no difficulty. Rank=5

(23:35-23:37) “Yeah, it’s pretty easy”


Task 4.2: Take Course Quiz. No usability issue. Rank=5. She said she would like to see which answers she got right and which ones she got wrong. Task 4.3: Check grades (Gradebook). No usability issue. Rank=5. Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. No usability issue. Rank=5. Task 5.2: Check 101…108 UA Online Course for returned assignments. No usability issue. Rank=5. Task 5.3: Access your returned assignment. Slight confusion. Stumbled upon it but did not know exactly how to get there. Rank=4. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issue. Rank=5.




Task 7.2: Exit SOT Student Orientation Tutorial. Could not find how to get back to course home. Failed task. Rank=1.

45

Test Scenario 8: Module 3/Tech Support Task 8.1: Locate Help Desk information. No usability issue. Rank=5.

EL-07 – 2:00 pm Undergraduate Student, College of Health and Human Services Test Scenario 1: Log-In and Navigation

Task 1.1: Log-on to site. No usability issue. Rank=5. Task 1.2: Course List page. No usability issue. Rank=5. “Well, these are the courses I signed up for so I would probably look into those.” (5:37-5:41) Task 1.3: Enter the first course 101…108 UA Online Course. No usability issue. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issue. Rank=5. “You can read the syllabus. It asks who you are. There’s five responses and why you are here. There’s five more responses. There’s announcements on the homepage. There’s… different pages where you can learn about UA, learn about UA college life or learn about research at UA.” (7:19-7:46). “This seems just about right to me.” (8:32-8:34). Task 1.5: Navigate to Module 1 called “About UA.” No usability issue. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity. No usability issue. Rank=5. Task 2.3: Social Learning Widget to post a chat-type message. No usability issue. Rank=5. “No, I think that it’s quick and easy and there’s easy access to it.” (18:11-18:17) Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. No usability issue. Rank=5.

46

Task 3.2: Explore discussion activity. Was not sure where to find the discussion activity. Could make a logical guess. Rank=3. Task 3.3: Post response. No usability issue. Rank=5. Task 3.4: Post response to another student. No usability issue. Rank=5. Preferred that the ‘how to participate’ would be taken away after the first response. Task 3.5: Submit a document to a dropbox. No usability issues. Rank=5.

(16:51-16:58) “Yes, where it says attachments with the little paper clip you can click add and remove and you can attach the file from your hard drive.” She knew exactly how to attach a document.


Task 4.2: Take Course Quiz. No usability issue. Rank=5. Task 4.3: Check grades (Gradebook). No usability issue. Rank=5. “I guess it says already the grade book key …I think maybe it could be at the top of the page… I guess I don’t really read top to bottom of the page.” (23:10-23:24). Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. No usability issue. Rank=5. Task 5.2: Check 101…108 UA Online Course for returned assignments. Slight confusion. Was not what she was expecting to see. It was hard for her to find without help. Rank=3. “I expected to see the actual assignment.” (26:37-26:40) Task 5.3: Access your returned assignment. Not attempted. Rank=0. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issue. Rank=5.


47

“Those icons were confusing because they were really small and it’s hard to tell. I would think if anything they would have been in that yellow bar that was at the top of the page.” (28:52-29:05)

“The one I thought it would be was the one with the little arrow but that was the log out.” (29:40-29:45).



“This is good. This reminds me of kind of like a FAQ’s page…Yeah I like to read them just …to get an idea just in case I run into any of the same problems.” (30:18-30:37)



Monday, March 26, 2012 EL-08— 1:00pm Undergraduate Student, College of Arts & Sciences


Task 1.1: Log-on to site. No usability issues. Rank=5. Task 1.2: Course List page. No usability issues. Rank=5. Task 1.3: Enter the first course 101…108 UA Online Course. No usability issues. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issues. Rank=5. Task 1.5: Navigate to Module 1 called “About UA.” No usability issues. However, slight confusion. Rank=5. “At this point I think would probably have gone back and started looking at the tutorial” (10:08-10:12)

48

Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. Failed task. Could not name students. Rank=1. Task 2.2: Social Learning Widget to check for activity. No usability issues. Rank=5. “Just by being short snippets you can take a quick look to see something you might be further interested in. And I could see where that could be a benefit … It would be something that would take me some time to navigate.” (16:43-16:56) Task 2.3: Social Learning Widget to post a chat-type message. No usability issues. Rank=5. Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course. No usability issues. Rank=5. Task 3.2: Explore discussion activity. No usability issues. Rank=5. “Looks like a straightforward discussion board with main subjects and people’s replies to the main subjects. I like the way it’s organized- easy to read- who started and who is responding.” (11:17-11:28). Task 3.3: Post response. No usability issues. Rank=5. Task 3.4: Post response to another student. No usability issues. Rank=5. “I like the way the button is right below it and you don’t have to go looking for it.” (12:32-12:35).

Task 3.5: Submit a document to a dropbox. Found this section confusing and needed help. Did not actually complete. Rank=0.

“Interesting. That seemed a bit awkward.” (14:13-14:14).


Task 4.2: Take Course Quiz. No usability issues. Rank=5.

49

“Now that I understand how that’s set up that’s very user friendly now that I understand that it’s in modules and subcategorized. But at first it was a little confusing for me.” (19:32-19:44) Task 4.3: Check grades (Gradebook). No usability issues. Rank=5. Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. Not attempted. Rank=0. Task 5.2: Check 101…108 UA Online Course for returned assignments. No usability issues. Rank=5. Task 5.3: Access your returned assignment. No usability issues. Rank=5. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issues. Rank=5.

Task 6.2: Exit course. Could not exit without help. Rank=1.


Task 7.1: Complete the SOT Student Orientation Tutorial course. No usability issues. Rank=5. Would prefer text to be black text on a white background vs. blue text on white background.



EL-09—12:00pm Graduate Student, College of Health Sciences and Human Services


Task 1.1: Log-on to site. No usability issue. Rank=5. Task 1.2: Course List page. No usability issue. Rank=5.

50

Task 1.3: Enter the first course 101…108 UA Online Course. No usability issue. Rank=5.

Task 1.4: Explore Course Home 101…108 UA Online Course . Wants bigger icons, prefers springboard home page. No usability issue. Rank=5. “It looks like Springboard.” (8:19 – 8:24). “The first thing my eyes go to are the who are you and I hate that about online classes, it’s like you’re starting in a brand new class and your instructor wants you to fill out all this information about you to get to know you and its none of their business.” (8:27 – 8:40). “I do love drop boxes, it makes my heart happy.” (10:04 - 10:08). Task 1.5: Navigate to Module 1 called “About UA.” No usability issue. Rank=5. Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. Could not find classmates. Found people on the site but did not know who they were or if they were students. Rank=1. Task 2.2: Social Learning Widget to check for activity. Could not do. Rank=1. Task 2.3: Social Learning Widget to post a chat-type message. Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course . No usability issue. Rank=5. “ I think its kind of…Like the little icons over here are crappy because they’re not self explanatory.” (18:17 – 18:22). Task 3.2: Explore discussion activity. No usability issue. Rank=5. Task 3.3: Post response. Didn’t know what to click to get the respond option. Guessed with some certainty at the icon. Rank=3. Task 3.4: Post response to another student. Is used to new responses being at the top. No usability issue. Rank=5. Task 3.5: Submit a document to a dropbox.

Rank=2: she attempted to submit an assignment, however, had some a lot of difficulty doing so, and was unsure even after she had done it.

51

(20:25-20:29) “I don’t know if this is their inbox or my inbox.”

(21:34-21:35) “What’s a basket?”

(21:41-21:45) I didn’t know what a basket is, let alone find a tab for it.”

(22:04-22:09) “Yea, I don’t know if my assignment went in or not, or who it went to or where.”

(23:06-23:17) “It was confusing, we probably would have been calling each other, and saying “Hey how did you do that, no this is how you do it- …. ‘cause most graduate students are not as tech savvy as undergrads.”

Test Scenario 4: Course Functionality Module 2 called “UA College Life” 101…108 UA Online Course Task 4.1: Instruct student to go to Module 2 called “UA College Life” 101…108 UA Online Course. Would like different layout with better headings. No usability issue. Rank=5.

Task 4.2: Take Course Quiz. Liked option to rejoin the quiz. No usability issue. Rank=5. Task 4.3: Check grades (Gradebook). Struggled to navigate the page. Rank=3. “Why all the clicking to get here?” (30:23 – 30:27). Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages Task 5.2: Check 101…108 UA Online Course for returned assignments

Task 5.3: Access your returned assignment. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. Wants less clicks and difficulty to get to email. Rank=5.

Task 6.2: Exit course. Struggled to find icon. Rank=3. “Why don’t they have a nice big button at the top to get me back to my…” (33:52 – 33:58).

52


Task 7.1: Complete the SOT Student Orientation Tutorial course. Too much random information scattered around. No usability issue. Rank=5.

Task 7.2: Exit SOT Student Orientation Tutorial. Struggled to find a way out. Rank=4.

Test Scenario 8: Module 3/Tech Support Task 8.1: Locate Help Desk information. Couldn’t find search option. Rank=4.

Thursday, March 29, 2012 EL-10— 8:00am Undergraduate Student, College of Business Administration



Task 1.4: Explore Course Home 101…108 UA Online Course . Had trouble getting back to course home once it was left. Rank=4. Task 1.5: Navigate to Module 1 called “About UA”. No usability issue. Rank=5 “Easy to read.” (26:23-26:29) Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity Task 2.3: Social Learning Widget to post a chat-type message. No usability issue. Rank=5. Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course

53

Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course Task 3.2: Explore discussion activity. No usability issue. Rank=5. Task 3.3: Post response. Struggled to find which question to respond to. Rank=4. Task 3.4: Post response to another student Task 3.5: Submit a document to a dropbox. Rank=1.

(29:08) Subject failed to upload the assignment and could not find the correct area. The subject found it confusing and different than the methods she was used to.


Task 4.2: Take Course Quiz. Save answers vs. save all is confusing. Rank=4. Task 4.3: Check grades (Gradebook). No usability issue. Preferred it to Springboard. Rank=5. Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages Task 5.2: Check 101…108 UA Online Course for returned assignments. Confused between inbox and outbox. Rank=4.

Task 5.3: Access your returned assignment. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. Struggled to establish email recipients. Rank=4.

Task 6.2: Exit course. Difficulty exiting and finding way back home. Rank=3. Test Scenario 7: SOT Student Orientation Tutorial

54


Task 7.2: Exit SOT Student Orientation Tutorial. Needed significant guidance to find the way out and back to the home page. Rank=3.

Test Scenario 8: Module 3/Tech Support Task 8.1: Locate Help Desk information

EL-11—9:00am Undergraduate Student, College of Education



Task 1.4: Explore Course Home 101…108 UA Online Course. No usability issue. Wants options to be to the side or on top and out of the way. Rank=5. “It kind of seems temporary there, and the more permanent stuff is usually like on bar.” (14:58 – 15:05). Task 1.5: Navigate to Module 1 called “About UA.” No usability issue. Rank=5.

Test Scenario 2: Social Interaction – Classmates Task 2.1: Name some of your classmates. No usability issue. Rank=5. Task 2.2: Social Learning Widget to check for activity. Task 2.3: Social Learning Widget to post a chat-type message. Slight problem finding the option to post. Rank=4. Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course Task 3.2: Explore discussion activity. Was unsure if the posts were by her or someone else and was unfamiliar with available options. Rank=4.

55

Task 3.3: Post response. No usability issue. Rank=5. Task 3.4: Post response to another student. No usability issue. Rank=5. Task 3.5: Submit a document to a dropbox.

Rank=3: she attempted to submit an assignment, however, had trouble finding where to submit an assignment, but once found it was easy to submit.

(24:03-14) “The dropbox that I’m used to has them listed, and I click … and I saw that, but I was like it’s not here.”

Test Scenario 4: Course Functionality Module 2 called “UA College Life” 101…108 UA Online Course Task 4.1: Instruct student to go to Module 2 called “UA College Life” 101…108 UA Online Course . No usability issue. Rank=5.

Task 4.2: Take Course Quiz. No usability issue. Rank=5. Task 4.3: Check grades (Gradebook). No usability issue. Rank=5. Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages. Didn’t know what was meant by discussion board messages, unsure if they were email messages, announcements, etc. Rank=3. “I hate that you can’t just hit back.” (35:15 – 35:30). Task 5.2: Check 101…108 UA Online Course for returned assignments. No usability issue. Rank=5.

Task 5.3: Access your returned assignment. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. No usability issue. Rank=5.

Task 6.2: Exit course. Took extensive searching to find a way out of the course. Rank=3.


56

Task 7.1: Complete the SOT Student Orientation Tutorial course. Just skimmed over rather than going through the entire tutorial. No usability issue. Rank=5.

“I wouldn’t read it, there’s so much.” (43:15 - 43:25).



EL-12—1:00pm Undergraduate Student, Summit College


Task 1.1: Log-on to site. “Does it have to be caps.” First try was unsuccessful but worked on second try. Rank=4. Task 1.2: Course List page. No usability issue. Rank=5. Task 1.3: Enter the first course 101…108 UA Online Course. No usability issue. Rank=5

Task 1.4: Explore Course Home 101…108 UA Online Course. Needs better heading on page and instructor name. Rank=5. “Confusing to see where I’m at on the page” when submitting response to other students. (9:20 – 9:32). “It doesn’t really say back to course home.” (10:08-10:14). Task 1.5: Navigate to Module 1 called “About UA.” Confused about course content. Required guidance to find key functions. Rank=3. “It’s very vague; it doesn’t have a lot of information on…” (11:03 – 11:08). After long pause, “Do I just click on one?” (12:20-12:30). “Umm that’s not what I was expecting, I’m not sure exactly what is in the basket of my inbox because I don’t have anywhere I can go to submit an assignment I guess. (12:30-12:42). Test Scenario 2: Social Interaction – Classmates

57

Task 2.1: Name some of your classmates. Required significant guidance to find classmates, she went to where springboard has the class list and didn’t know where to look elsewhere. Rank=2. Task 2.2: Social Learning Widget to check for activity. Task 2.3: Social Learning Widget to post a chat-type message. Test Scenario 3: Course Functionality Module 1 “About UA” 101…108 UA Online Course Task 3.1: Instruct student to complete module 1 “About UA” 101…108 UA Online Course . Task 3.2: Explore discussion activity. No usability issue. Rank=5 Task 3.3: Post response. No usability issue. Rank=5 Task 3.4: Post response to another student. Confused on dropdown options. Rank=5. Task 3.5: Submit a document to a dropbox. Rank=0: Not Attempted Test Scenario 4: Course Functionality Module 2 called “UA College Life” 101…108 UA Online Course Task 4.1: Instruct student to go to Module 2 called “UA College Life” 101…108 UA Online Course. Couldn’t find right away. Rank=4

Task 4.2: Take Course Quiz. No usability issue. Rank=5 Task 4.3: Check grades (Gradebook).No usability issue. Would like to have direct discussion to instructor. Rank=5 “Its different than what Im used to… This gives more information than what I’m used to like it will calculate your percentage, my one that I’m used to does not.” (21:15 – 21:30). Test Scenario 5: Checking Discussion Board and Returned Assignments 101…108 UA Online Course

Task 5.1: Check 101…108 UA Online Course for unread discussion board messages Task 5.2: Check 101…108 UA Online Course for returned assignments. No usability issue. Rank=5.

58

Task 5.3: Access your returned assignment. No usability issue. Rank=5. Test Scenario 6: Communicate with Instructor Task 6.1: Email Instructor. Struggled to find the email option and required it to be pointed out. Rank=2.

Task 6.2: Exit course. Required some guidance to find home icon. Rank=3.



Task 7.2: Exit SOT Student Orientation Tutorial. Couldn’t find right away. Rank=3.


59

Appendix E: Pre-Interview Online Survey Responses – Students

Prior to the interviews, the student participants completed a brief, online survey.

1. What is your gender?

# Answer

Response % 1 Male

4 33%

2 Female

8 67%

Total 12 100%

Statistic Value Min Value 1

Max Value 2

Mean 1.67

Variance 0.24

Standard Deviation 0.49

Total Responses 12

60

2. What is your race? (Select one or more boxes)

# Answer

Response % 1 White/Caucasian

9 75%

2 Black, African American

2 17%

3 Asian

1 8%

4 Other, please enter description below:

0 0%

Other, please enter description below:


Max Value 3

Total Responses 12

61

3. Are you of Hispanic, Latino, or Spanish origin?

# Answer

Response %

1

Yes, I am of Hispanic, Latino, or Spanish origin

2 17%

2

No, I am not of Hispanic, Latino or Spanish origin

10 83%

Total 12 100%


Max Value 2

Mean 1.83

Variance 0.15


Total Responses 12

62

4. What is the highest degree or level of education you have completed?

# Answer

Response %

1 High School Diploma

1 8%

2 GED or alternative credential

0 0%

3 Some College credit, but less than 1 year

0 0%

4 1 or more years of college credit, no degree

4 33%

5 Associate's degree (Ex. AA, AS)

2 17%

6 4-year College Degree

5 42%

7

Masters Degree (Ex. MA, MS, MEng, MEd, MSW, MBA)

0 0%

8

Professional Degree (Ex. JD, MD, DDS,LLB,DVM)

0 0%

9 Doctoral Degree (Ex. PhD, EdD)

0 0%

Total 12 100%

63


Max Value 6

Mean 4.75

Variance 2.20


Total Responses 12

5. What is your current employment status?

# Answer

Response %

1

Full time (work more than 35 hours a week)

3 25%

2

Part time (work less than 35 hours a week)

5 42%

3 Retired

0 0%

4 Not employed

4 33%

Total 12 100%

64


Max Value 4

Mean 2.42

Variance 1.54


Total Responses 12

6. On average, how many hours a day do you spend...

# Answer Min Value Max Value

Average Value

Standard Deviation

Responses

1 Watching broadcast or cable TV

1.00 5.00 2.50 1.31 12

2 Listening to broadcast radio

0.00 4.00 1.33 1.15 12

3

Accessing internet from personal computer

1.00 8.00 4.83 2.48 12

4

Internet from mobile phone/PDA

0.00 4.00 1.42 1.08 12

Statistic Value Total Responses 12

65

7. When were you born?

# Answer

Response % 1 1996

0 0%

2 1995

0 0%

3 1994

0 0%

4 1993

0 0%

5 1992

0 0%

6 1991

0 0%

7 1990

0 0%

8 1989

0 0%

9 1988

0 0%

10 1987

1 8%

11 1986

1 8%

12 1985

0 0%

13 1984

1 8%

14 1983

3 25%

15 1982

0 0%

16 1981

0 0%

17 1980

1 8%

18 1979

0 0%

19 1978

1 8%

20 1977

0 0%

21 1976

0 0%

66

22 1975

1 8%

23 1974

0 0%

24 1973

0 0%

25 1972

1 8%

26 1971

0 0%

27 1970

0 0%

28 1969

0 0%

29 1968

1 8%

30 1967

0 0%

31 1966

1 8%

32 1965

0 0%

33 1964

0 0%

34 1963

0 0%

35 1962

0 0%

36 Before 1962

0 0%

Total 12 100%

67


Max Value 31

Mean 18.25

Variance 49.30


Total Responses 12

8. How would you rate yourself in the following areas

# Answer Min Value Max Value

Average Value

Standard Deviation

Responses

1 Knowledge about the Internet

4.00 10.00 7.33 1.87 12

2

Use of online learning systems

1.00 9.00 6.17 2.72 12

9. What type of device do you primarily use to access the Internet?

# Answer

Response %

1 Personal computer

8 73%

2 Tablet

1 9%

3 Smart phone

2 18%

Total 11 100%

68


Max Value 3

Mean 1.45

Variance 0.67


Total Responses 11

10. Are you willing to participate in future studies?

# Answer

Response % 1 Yes

12 100%

2 No

0 0%

Total 12 100%

69

Appendix F: Additional Faculty Verbatims


4:15pm Faculty, College of Business Administration (FEL-01)

When logged on first:

“It looks like there is a lot to read here, so I would need to take some time.” (04:34-04:38)

“I use and don’t use tutorials, kind of a combination of both I guess.” (05:48-05:51)

Look at UA Online Course (course homepage):

“You know it’s a little busy, it’s a… what you’re supposed to do next isn’t all that clear, and in general it’s got sort of the look and feel of Springboard, you with the way this is organized here and here … but I think what’s missing is -okay what’s my next step –“ (06:50-07:07)

Student Orientation Tutorial

Yeah I think people are impatient and they like to get to something right away and so this is maybe taking a little bit longer and the fact that there wasn’t any video on there seemed like sort of the screen part of it was kind of basically that stuff flying around, even if it were somebody’s face and just the top of the head would have been better than just the abstract part of whatever that was.” (13:40 – 14:09)

Overall

“I really like when I’m grading students’ papers to be able to see a picture of them ... and, so … it doesn’t seem to have that…” (57:33 – 57:42)

“I use a touch sensitive screen, so another think that would be interesting … and I don’t think I can do this on Springboard, if so I haven’t figured out how to do it yet easily…. is to just be able to write something, you know, with my light pen or my finger, or whatever, on my touch system screen on my laptop computer and have that sort of, you know, appear to the student when I return the paper to them.” (58:04 – 58:30)


70

6:30pm Faculty, Summit College (FEL-02)

“I like it. I mean, it’s just something you’d have to get used to.” (1:01:07

“Some of the, you know, little icons down here again would definitely take getting used to.” (1:01:25-1:01:30)

“The only thing I can tell you is if you’re going to make it look like a Springboard, make it look like a Springboard, if it’s going to be something totally new, then make it something just a little bit different than Springboard.” (1:02:33 – 1:02:42)

“And this an awful a lot like Springboard, so my suggestion would be if you’re going to make it different, let’s make it different.” (1:02:53 – 1:03:00)

Monday, April 2, 2012

10:00am Faculty, Speech Lang-Path (FEL-03)

Tuesday, April 3, 2012

5:30pm Faculty, Anthropology (FEL-04)

“I didn’t get a clear enough sense of how that “discussion exchange” would actually work.” (1:02:33-1:02:40)

“As a useful tool, looks pretty straight forward.” (1:03:51-1:03:55)

“Everybody’s pet-peeve about Springboard, and they only just fixed it surprisingly this year, is activating your course … when the students sit there and say “it’s not there.” … So I am assuming that this does not have that problem.” (1:03:56 - 1:04:20)

“It’s not that you would look at it all that often, but sometimes even on springboard, I write something into the announcement page, and then I’ll just flip over and see from the student point of view, looking at all of their stuff, what are they seeing.” (1:05:14 – 1:05:27)

Documents

Usability Test Report -AppendicesUsability Test Report of E-Learning Platform The UA/Pearson Learning Studio (Appendices) Date of Report: April 11, 2012 Last Update: April 15, 2012