CITERS2003 Presentation 5 July 2003CITERS2003 Presentation 5 July 2003 11

A Usability Study of a A Usability Study of a Language Centre Web Site Language Centre Web Site

By Andy MorrallBy Andy Morrall

22

Why, in general?Why, in general?

Teachers write educational web pagesTeachers write educational web pages

Students use them outside the Students use them outside the classroomclassroom

Teachers don’t always see them being Teachers don’t always see them being usedused

Hard to see if the pages work properlyHard to see if the pages work properly

Students rarely give detailed feedbackStudents rarely give detailed feedback

33

Why, specifically?Why, specifically?

I am in charge of an educational I am in charge of an educational websitewebsite

Very little feedback from usersVery little feedback from users

Need to write reports about quality Need to write reports about quality controlcontrol

44

How?How?Need a tried-and-tested, reliable and valid Need a tried-and-tested, reliable and valid way to test the siteway to test the site

Can’t test if it is good for learning, as there Can’t test if it is good for learning, as there are too many other ways users could be are too many other ways users could be learning Englishlearning English

Users opinions of the user-friendliness of Users opinions of the user-friendliness of the site may be too nice and non-specificthe site may be too nice and non-specific

So, a need to watch students using the So, a need to watch students using the site and analyse the results systematicallysite and analyse the results systematically

55

What is Usability?What is Usability?Rubin (1994, p.22) says that usability testing Rubin (1994, p.22) says that usability testing “employs techniques to collect empirical “employs techniques to collect empirical data while observing representative end data while observing representative end users using the product to perform users using the product to perform representative tasks”. He describes an representative tasks”. He describes an approach to usability testing, which approach to usability testing, which “employs an iterative cycle of tests intended “employs an iterative cycle of tests intended to expose usability deficiencies and to expose usability deficiencies and gradually shape or mould the product in gradually shape or mould the product in question.”question.”

66

What?What?

77

Research QuestionsResearch Questions1.1. Do the improvements to the site made Do the improvements to the site made

during the usability study result in during the usability study result in improvements in test participants’ improvements in test participants’ success rate in completing tasks on the success rate in completing tasks on the CILL site? CILL site?

2.2. Do the improvements to the site made Do the improvements to the site made during the usability study result in during the usability study result in participants being able to carry out these participants being able to carry out these tasks in a shorter time? tasks in a shorter time?

3.3. Do the participants say that they believe Do the participants say that they believe that the site is easier to use?that the site is easier to use?

88

Who and what?Who and what?To learn if the site is usable, I needed to know To learn if the site is usable, I needed to know what it was used for.what it was used for.I asked 5,100 CILL members to take part in the I asked 5,100 CILL members to take part in the survey by e-mailsurvey by e-mail215 replied filled in the questionnaire at 215 replied filled in the questionnaire at http://elc.polyu.edu.hk/CILL/research/cill_site_imprhttp://elc.polyu.edu.hk/CILL/research/cill_site_improvement_questionnaire.htmovement_questionnaire.htm

The survey asked users to:The survey asked users to:– Identify the right siteIdentify the right site– Say how long they had been using the InternetSay how long they had been using the Internet– When they last used the CILL siteWhen they last used the CILL site– What did they use the site forWhat did they use the site for– If they had any suggestions for improving the siteIf they had any suggestions for improving the site

99

Survey ResultsSurvey Results78 users correctly identified the right site78 users correctly identified the right site

Analysis of Responses to Question 7 of the Initial Questionnaire

Grammar 22 Pronunciation 12 Listening 9

Vocabulary 9 Exercises 7 Dictionary 6

Reference Machine 4 Job Application Letter

4 Reading 3

Writing 3 Oral 2 Referencing 2

Idioms 2 Newspapers 2 Academic Writing 2

English in the Workplace

2 Reports 2 Interviews 2

Finding Materials 2 Testing Level 2 CILL Information 2

Research 1 Purdue On-line Writing Lab.

1 CILL teachers’ e-mail 1

Example Assignment 1 Staff information 1 Opening Hours 1

Games 1 Resumes/CV 1 Discussion 1

Exit Test 1 Memos 1 Speaking 1

1010

TasksTasks1.1. Find a grammar exercise on the difference Find a grammar exercise on the difference

between ‘staff’ and ‘staffs’.between ‘staff’ and ‘staffs’.2.2. Find out how to pronounce ‘thorough’.Find out how to pronounce ‘thorough’.3.3. Find a page where you can learn vocabulary Find a page where you can learn vocabulary

about the news.about the news.4.4. Find a page where you can learn English by Find a page where you can learn English by

listening to pop music.listening to pop music.5.5. Find a page with some common job interview Find a page with some common job interview

questions and advice and an exercise about how questions and advice and an exercise about how to answer them.to answer them.

6.6. Look up the meaning of the word ‘discharge’.Look up the meaning of the word ‘discharge’.7.7. Make a reference (Details provided)Make a reference (Details provided)8.8. Find a page which helps you to write a job Find a page which helps you to write a job

application letter.application letter.9.9. Find a page where you can read about e-mail Find a page where you can read about e-mail

pen-pals you can write to.pen-pals you can write to.10.10. Find a page that helps you improve your Find a page that helps you improve your

newspaper reading skills.newspaper reading skills.

1111

ParticipantsParticipants2 groups:2 groups:– Re-test group – did the test tasks both before Re-test group – did the test tasks both before

and after the changesand after the changes– New group – only did the tests after the New group – only did the tests after the

changeschanges

Both groups included:Both groups included:– CILL students and teachersCILL students and teachers– native speakers of English and non-nativesnative speakers of English and non-natives– participants with varying degrees of participants with varying degrees of

computer experience and skillscomputer experience and skills

1212

How to do a usability testHow to do a usability test1.1. Welcome the participant and check their identity and Welcome the participant and check their identity and

profileprofile2.2. Explain the purpose and duration of the test (1 hour in Explain the purpose and duration of the test (1 hour in

total, including post-test questionnaire).total, including post-test questionnaire).3.3. Elicit any questions.Elicit any questions.4.4. Ask for consent for the participant to take part.Ask for consent for the participant to take part.5.5. Ask the participant to sign the consent form. Ask the participant to sign the consent form. 6.6. Participant chooses the browser and version they usually Participant chooses the browser and version they usually

use to access the CILL web site.use to access the CILL web site.7.7. The participant attempts to carry out the tasks, with the The participant attempts to carry out the tasks, with the

tester watching and taking notes.tester watching and taking notes.8.8. The participant either completes the final task or runs out The participant either completes the final task or runs out

of time.of time.9.9. Give the participant a short break.Give the participant a short break.10.10. Ask the participant to complete a post-test questionnaire.Ask the participant to complete a post-test questionnaire.11.11. Chat about the test and the siteChat about the test and the site12.12. Thank the participant for taking part.Thank the participant for taking part.13.13. Participant departs.Participant departs.

1313

Task 1Task 1

1.1. Find a grammar exercise on the Find a grammar exercise on the difference between ‘staff’ and difference between ‘staff’ and ‘staffs’.‘staffs’.

Easy task first to give users Easy task first to give users confidence.confidence.

1414

Task 1Task 1

1515

Task 1 ChangedTask 1 Changed

1616

Results of Task 1Results of Task 1Question 1 Statistics

0

50

100

150

200

250

First Second (Re-test)

Second(New)

Test

Sec

on

ds

Mean

Minimum

Maximum

StandardDeviation

1717

Summary of ImprovementsSummary of Improvements  Task

NumberTask Success Mean Task Duration

 Improvement or remained at maximum

Improvement in percent

Improvement or remained the same

Improvement in seconds

 Re-test group

New group

Re-test group

New group

Re-test group

New group

Re-test group New group

  1. 20% 20% 52 15

  2. 0% 10% 26 10

  3. 0% 0% 71 76

  4. 0% 10% 37 12

  5. 40% -5% 118 111

  6. 0% 0% 0 21

  7. 0% 5% 139 -12

  8. 0% 0% 3 3

  9. -20% 0% 11 24

  10. 0% 10% 42 40

 Total / Mean

Total = 9

Total =9

Mean =

4%

Mean =

5%

Total =10

Total =9

Mean = 50 Mean =30

 

1818

Results – Research Question 1Results – Research Question 1

1.1. Do the improvements to the site Do the improvements to the site made during the usability study made during the usability study result in improvements in test result in improvements in test participants’ success rate in participants’ success rate in completing tasks on the CILL site? completing tasks on the CILL site?

Re-test Group:Re-test Group: +4%+4% New Group: New Group: +5%+5%

1919

Results – Research Question 2Results – Research Question 2

2. Do the improvements to the site 2. Do the improvements to the site made during the usability study made during the usability study result in participants being able to result in participants being able to carry out these tasks in a shorter carry out these tasks in a shorter time? time?

Re-test Group: 50 seconds fasterRe-test Group: 50 seconds fasterNew Group: New Group: 30 seconds faster 30 seconds faster

2020

Results – Research Question 3Results – Research Question 3

3. Do the participants say that they 3. Do the participants say that they believe that the site is easier to believe that the site is easier to use?use?

Re-test group only:Re-test group only:Strongly agree: 1Strongly agree: 1Agree: 1Agree: 1Neither agree nor disagree: 3Neither agree nor disagree: 3

2121

Limitations of the Study Limitations of the Study The Small Number of Tasks in Relation to The Small Number of Tasks in Relation to the Size of the Site (10 tasks, 234 pages)the Size of the Site (10 tasks, 234 pages)The Small Number of Design Iterations The Small Number of Design Iterations (I did 3, iterative design principles (I did 3, iterative design principles recommend repeated iterations)recommend repeated iterations)Debriefing Participants – it’s embarrassing Debriefing Participants – it’s embarrassing to talk about why they didn’t do or see to talk about why they didn’t do or see things - the worse the performance of a things - the worse the performance of a participant is, the more valuable their participant is, the more valuable their debriefing feedback and further test debriefing feedback and further test participation is, but the more difficult they participation is, but the more difficult they are to get.are to get.

2222

Recommendations for Further Recommendations for Further Study Study

Further study could be done into:Further study could be done into:using more authentic tasks such as a text using more authentic tasks such as a text correction task, with the answers on the sitecorrection task, with the answers on the siteinvestigating the effects of testers and investigating the effects of testers and participants reading the task instructions aloud, participants reading the task instructions aloud, because some participants didn’t read all the because some participants didn’t read all the instructionsinstructionsthe effects of participants’ levels of computing the effects of participants’ levels of computing skills on their level of task success and task skills on their level of task success and task duration duration the optimum number of tests to run between site the optimum number of tests to run between site design iterations design iterations

2323

Usability StudiesUsability StudiesDiscount Usability Testing Discount Usability Testing Nielsen, J. (2000). Nielsen, J. (2000). Designing web usability: The practiceDesigning web usability: The practice

of usability. of usability. Indianapolis: New Riders.Indianapolis: New Riders.

Nielsen, J. (1994). Guerilla HCI: Using discount usability Nielsen, J. (1994). Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In engineering to penetrate the intimidation barrier. In Bias, R.G. & Mayhew, D. (Eds.) Bias, R.G. & Mayhew, D. (Eds.) Cost-justifying Cost-justifying usability. usability. Boston: Academic Press. Boston: Academic Press.

Nielsen, J. (1993). Nielsen, J. (1993). Usability engineeringUsability engineering. Boston: AP . Boston: AP Professional. Professional.

http://www.useit.comhttp://www.useit.com

Rubin, J. (1994). Rubin, J. (1994). Handbook of Usability TestingHandbook of Usability Testing. New . New York, NY. John Wiley & Sons. York, NY. John Wiley & Sons.