Newspaper in Education Web Site (NEWS)Usability Evaluation

Conducted by Terry Vaughn

School of InformationThe University of Texas at AustinNovember 29, 2004

NE

WS

Usa

bilit

y E

valu

atio

nWhat we’ll talk about today

• Newspaper in Education Program Overview• Test Description• Participant Background• First Impressions Interview• Task Performance• Problems and Their Sources• Final Reactions• Conclusions• Recommendations

• Questions from the audience

NE

WS

Usa

bilit

y E

valu

atio

nNewspaper in Education Program Overview

NIE• Program of the Orlando Sentinel (major newspaper)• Promotes literacy in the central Florida area• Supplies newspapers and teaching materials to area

schools

NEWS• Public can learn more about NIE programs• Register for NIE programs• Purchase newspapers and other classroom materials• http://extra.orlandosentinel.com/nie/

NE

WS

Usa

bilit

y E

valu

atio

nTest Description

• Research Questions• Goals• Participants• Environment and Equipment• Test Procedure• Data Collection

NE

WS

Usa

bilit

y E

valu

atio

nParticipant Background Data

• 4:1 female to male• 30 year median age• All college educated• 10 - 21 years of computer use• 8 - 10 years of Web use• All used the Web in their work• All claimed the Web was important - very important part of

their work

ID Age Sex Education

ComputerUsage(years)

WebUsage(years)

Web Usagefor work

Importanceof Web to work

1 26 F Graduate 10 8 Every day Important2 30 F Undergraduate 10 9 Every day Very important3 29 F Undergraduate 10 9 Every day Important4 53 F Graduate 21 10 Every day Very important5 32 M Graduate 20 10 Every day Very important

NE

WS

Usa

bilit

y E

valu

atio

nParticipants’ First Impressions

Positive• “Visually appealing”• Appropriate colors and

graphic• Grouping of left-side

navigation• Clear audience

Negative• Headline looks like a

run-on sentence• Multiple layers of

navigation on top may confuse

NE

WS

Usa

bilit

y E

valu

atio

nTask Set

Tasks 1 - 5• Finding information• Example:

You are interested in taking a workshop to learn how to incorporate the news into your curriculum. How many participants must you have for a Teacher Workshop?

Tasks 6 - 10• Filling out forms• Example:

You decide to order a newspaper subscription for your classroom starting after the Christmas break (1/4), suspending service over Spring Break (3/14 – 3/18), and ending on the last day of school (5/25). Find the page where you can order a subscription and fill out the form.

NE

WS

Usa

bilit

y E

valu

atio

nTask Performance (ranked according to difficulty)

Most problematic tasks:• Task 6. Find the page where you can order a classroom

subscription and fill out the form.• Task 2. What is the minimum order of newspapers per

classroom?• Task 1. In addition to daily newspapers, what else will you

receive with your classroom subscription?

Task Errors Assistance Time (min) CompleteComplete without

assistance6 3.0 1.8 5:50 40% 0%2 3.2 2.0 2:23 100% 0%1 2.4 2.2 2.42 100% 20%7 0.8 0.4 2:11 100% 60%3 1.8 0.4 1:17 100% 60%5 1.2 0.4 1:00 100% 60%4 0.6 0.4 0:51 100% 60%9 1.0 0.4 2:25 100% 80%8 0.4 0.2 2:10 100% 80%10 0.2 0.2 1:19 100% 80%

NE

WS

Usa

bilit

y E

valu

atio

nMost Problematic Tasks

Task 6. Find the page where you can order a classroom subscription and fill out the form.

• Errors: invalid, incorrect, and omitted data• Sources: vague labeling and directions; customers had to

manually calculate data; unintelligible server error page

• Quotes:"I don't get this at all.”"I'm supposed to calculate all this myself?”

Classroom Subscription Order Form

NE

WS

Usa

bilit

y E

valu

atio

nMost Problematic Tasks

Task 2. What is the minimum order of newspapers per classroom?

• Errors: navigation - clicked wrong page• Sources: vague labeling; information disorganization -

relevant information is located on the FAQ

• Quotes:“An FAQ is my last resort.”

Newspaper in Education FAQ

NE

WS

Usa

bilit

y E

valu

atio

nMost Problematic Tasks

Task 1. In addition to daily newspapers, what else will you receive with your classroom subscription?

• Errors: navigation - clicked wrong page; wrong answer• Sources: poor information organization

• Quotes:"I have no idea where to go.”"How do I begin?”

Teacher Home Page

NE

WS

Usa

bilit

y E

valu

atio

n

Errors Ranked According to Criticality

Identifying Problems

Error Frequency Severity Criticality

Entered invalid data into form (unrecoverable) 60% 4 2.40Clicked wrong page

100% 2 2.00Entered incorrect data into form (recoverable) 100% 1 1.00Searched for information that could not be found 40% 2 0.80Omitted required form data

60% 1 0.60Gave incorrect answer

20% 2 0.40

Severity Scale1 - Minor inconvenience, easy to circumvent234 - Major problem prevents user from completing task

NE

WS

Usa

bilit

y E

valu

atio

nIdentifying Problems

Sources of ErrorsError Source

Entered invalid data into form(unrecoverable)

• Directions did not adequately describe the necessaryform information

• Text box labels confused participants• Unnecessary form information confused participants

Clicked wrong page • Links inconsistently labeled• Link labels confused participants• Site organization confused participants

Entered incorrect data into form(recoverable)

• Text box labels confused participants

Searched for information that couldnot be found

• Search engine searched Orlando Sentinel newspapercontent, not NIE site

Omitted required form data • Form layout• Labels to distant from their text boxes• Labels look like they are associated with nearest form

elements• Text boxes are too close vertically

Gave incorrect answer • Format and style of copy• Information is not grouped• Headings do not stand out

NE

WS

Usa

bilit

y E

valu

atio

nFinal Reactions

Overall grade: C+

Most appreciated: visual designLeast appreciated: information disorganization

“Will you return to the site?”Yes - 2Maybe - 2No - 1

“Would you recommend the site to a colleague?”Yes - 2Maybe - 1No - 2

NE

WS

Usa

bilit

y E

valu

atio

nConclusions

Q: Is there enough information for customers to make informed decisions about the purchase of NEWS content and services?

A: Yes, but it is difficult to gather because relevant information is not adequately grouped and labeled.

Q: Can customers navigate the site and find the information they need?

A: Yes, but sometimes customers get disoriented because of the poor labeling and information organization.

Q: Can customers find the information in an acceptable time frame?

A: Yes, but not without difficulty and/or assistance.

NE

WS

Usa

bilit

y E

valu

atio

nConclusions

Q: What problems do customers encounter performing tasks on NEWS?

A: Customers mainly encounter problems completing the order forms.

Q: Can customers complete the online subscription forms. If so, how many?

A: None without assistance.

Q: How do users perceive NEWS?A: Due to the flaws in the site and the problems participants

encountered, they we’re left questioning the professionalism of the NIE program. Since the forms were so difficult, participants questioned NIE’s commitment to customer service.

NE

WS

Usa

bilit

y E

valu

atio

nRecommendations

Labeling• plain English - no jargon• consistent• descriptive

Information Organization• group relevant information according to service offering• don’t scatter related information around

Hyperlinks• change the Contact link to direct to a web page• no “click here”

NE

WS

Usa

bilit

y E

valu

atio

nRecommendations

Form Design• improve layout - spacing and grouping• clear, descriptive labeling of text boxes• don’t make customers do work that a computer could to

e.g. calculating data• break Classroom Subscription Form into two parts, with an

interactive calendar to help customers with start/stop dates

Writing style• clear and concise• employ a copy editor

Feedback• order form confirmation

NE

WS

Usa

bilit

y E

valu

atio

nQuestions

Thank you!