UX-Research 3 Scenarios v2019 - HS Augsburg€¦ · UX Research test scenarios KP Ludwig John Usability Test Preconditions Already existing ... This encourages the user to explore

UX Research test scenarios

KP Ludwig John

UX Researchtest scenarios


KP Ludwig John

Usability TestPreconditions

Already existing

• PersonasDefinition of target groups

• Context of usewhere, when, how long, equipment

• Test goalsWhat do we want to learn?!


KP Ludwig John

• What do you want to learn by the test?Which elements of the IA design have been discussed controversial?Which elements seam to be critical for the general success of the application?

• Which elements of your application have to be coveredMain functions / use casesCrucial functions of the applicationUse Cases with many Alternate Flows

Usability-Test PredefinitionsTest goal


KP Ludwig John

Usability-Test

Comparison of 3 print Layouts Brochure with information about companies

IMS.UX - 2017

Test goal

Source: Team UXperts, IMS.UX 2017

Which Layout is best?


KP Ludwig John

Usability-Test

Source: Team UXperts, IMS.UX 2017

Aspects to consider

Extracting Information How easy is it to get the information? What information can you get?

Memorizing InformationCan you memorize the information after reading? Is it easy to remember

ClarityIs the structure obvious to the user? (Naming, Positioning, Design of elements)

ComprehensibilityDoes the user understand the content?

Readability - Is the text easy to follow?

Look & Feel - How is the first impression? Do you have a good feeling reading the paper?



KP Ludwig John

Usability-Test

Source: Team „Group Title“, IMS.UX 2017

Aspects to consider

How well is information communicated?


How fast can information be perceived and processed?

How much attention does relevant information get?

Measuring units!

Related questions


KP Ludwig John

Usability-Test


Aspects to consider

How well is information communicated?


How fast can information be perceived and processed?

How much attention does relevant information get?

Measuring unit: Time to complete tasks.

Measuring unit: Time a test subject’s focus rests on relevant information.

Measuring unit: Amount of misunderstandings and missed information

Measuring units


KP Ludwig John

paper based software based

Usability-Test

• Which functionality has your prototype to providein order serve your test goals?

PredefinitionsTest object

• What kind of Prototype is needed in order serve your test goals


KP Ludwig John

Usability-Test

• Diary study (remote) • Thinking aloud (with note taking) • Video observation • Screen recording • Eye tracking • Post test interview

PredefinitionsTest methodWhich test method do you want to apply?


KP Ludwig John

Usability-Test

• What kind of equipment serves best the test goals? • Consider testing under of real context of use conditions

(Screen size, Space, time, mobile, distraction, sound)

PredefinitionsTest equipment


KP Ludwig John

Usability-Test PredefinitionsTest scenario

Create a test scenario which defineseach single step of the test in detail,

including

•Introduction •Tasks •and Expression of Thanks


KP Ludwig John

Test location • Where do you want to test? • Availability, Influence of Weather or Light?

Usability-Test PredefinitionsTest space

Test equipment • Where do you get it? • How to handle it?


KP Ludwig John

• Testleader • Technical manager • Observer • Analyst

Usability-Test PreparationTest team


KP Ludwig John

• Where can the test persons be recruited from? (do they represent the target group, are they available (time) ?) • Do they get some expense allowance (Aufwandsentschädigung)? (which?) • recruit test persons

Usability-Test PreparationTest persons

Find test persons make appointments and a test schedule

• How many Test persons are needed? (about 7-8 per Test version (!) )

UX Research research methods

overview

KP Ludwig John

UsabilityTest persons


overview

KP Ludwig John

It’s best to test with representatives of your main target group.But don’t be obsessed about it!If its difficult to recruit them, test with people you can get hold of. Take it in account when evaluating the test results:Would our users have that problem? Or was it only a problem because the participant wasn’t familiar with the field?

Rather few but frequentlystart early in the development processand repeat and repeat and repeat

Who do the test with?

Usability Test Test persons

How many test persons?


overview

KP Ludwig John

Quelle: Steve Krug, Don‘t make me think! Redline Heidelberg, 2006

Example:

Test personsUsability Test

Test with 8 usersfound problems: 5 major ones



overview

KP Ludwig John

Example:

Quelle: Steve Krug, Don‘t make me think! Redline Heidelberg, 2006


2 Tests with 3 users each = 6 Test personsfound problems: 5 major + 4 minor = 9 problems



overview

KP Ludwig John

Wen testen?

Datenquelle: Faulkner (2003): Beyond the five-user assumption: Benefits of increased sample sizes in usability testing; in Sarodnik/Braun: Methoden der Usability Evaluation (Huber Verlag 2011)



overview

KP Ludwig John


source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF, page 19

http://www.nngroup.com/reports/how-to-conduct-eyetracking-studies


overview

KP Ludwig John

Most significant problems will be discovered by small number of test users already.

Finding three participants is easy to handle:test and debrief (Nachbesprechen) the same day! Design small tests with focussed tasks.Learn, move on and test again.


Reference: Steve Krug: „Rocket Surgery Made Easy“, New Riders 2010, chapter 5


Can you use the same participants again?Not for the same task or application.They know to much already.


KP Ludwig John

Test scenarios in detail


KP Ludwig John

1. Test person works on given tasksoften under direct observation(screen recording software, video, human observer)

2. Interview with the userto better understand critical situations during the test.Possibility to discuss improvements together

Usability-TestTwo Steps


KP Ludwig John

Content and style

Test scenario

Identify at least 3 essential activitiesto be covered by the test

Create 1 or more tasks per activity to be followed by the test person

Example:

„Become a new member of this online communityand create your individual account!“

„Which of the dishes offered at Mensa next Monday you prefer? Note it on paper.“


KP Ludwig John

Content and style

• complex enough (not to simple) • not too complex either (shouldn’t take too long to work on each single task) • cover all important issues of your test

Test scenario

Tasks have to be


KP Ludwig John

Content and style

• create realistic tasks Users should try to do the same things in Lab as they would do at home or at work

Test scenario

Tasks determine looks!

Nielsen/Pernice: Eyetracking Web Usability, New Riders, Nielsen Norman Group 2010

Important!


KP Ludwig John

Tasks determine looks!The Monkey Business IllusionSelective attention test by Daniel Simons 2010

https://youtu.be/IGQmdoK_ZfY

Count how many times the players wearing white

pass the ball


KP Ludwig John



pass the ball

?Result


KP Ludwig John


https://youtu.be/IGQmdoK_ZfY


pass the ball


KP Ludwig John

Tasks determine looks!

Reference: http://www.theinvisiblegorilla.com

These experiments reveal two things:

• we are missing a lot of what goes on around us

• we have no idea that we are missing so much

The Monkey Business IllusionSelective attention test by Daniel Simons 2010

A variation of one of the best-known experiments in psychology

Book first published in 2010

The Invisible Gorilla experiment from 1999


KP Ludwig John

Test scenario Tasks determine looks!

Same content


Website of Usability Week Conference

Task 1:Decide which of two conferences rather to attend

Task 2:Decide which seminar to choose

from that conference


KP Ludwig John



People look differently at the same information,depending on what they are trying to do.

In both cases: • Heatmap visualisation of Eyetracking Recordings

(objective measurement) • identical content

But: • Different users interest (determined by the task)

Same content - Different Tasks


KP Ludwig John


source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF page 145

Task: Find the US population and that of the state Texas Looks: Menus, content headings, population clock at the right

Screenshots from us census bureau (statistics)Same content - Different Tasks



KP Ludwig John



Task: What do you think of this page? Comment on it! Looks: All over … This will not happen in real user situations!

Screenshots from us census bureau (statistics)Same content - Different Tasks



KP Ludwig John



If testing just one page

Don’t open the page and ask the user to “Please look at the page and tell us what you think.”This encourages the user to explore a page in a way they probably never would.

! Do ask the user to do a task, then launch the page

Don’t open the page, let the user examine it, then give him a task. Doing these steps in this order would encourage a user to examine the page when he might have never done this on his own.



KP Ludwig John

Test scenarioAvoid Priming

source: https://www.nngroup.com/articles/priming/

Fill in the blanks to form an english word:

B_ _ T_RWhat comes to your mind spontaneously??


KP Ludwig John

Test scenarioAvoid Priming


Fill in the blanks to form an english word:

B_ _ T_RWhat comes to your mind spontaneously?

“The aroma of warm bread spread tantalizingly out of the bakery.”

Read the following sentence:


KP Ludwig John

Test scenario


The mention of “bread” is likely to have activated concepts that are associated to it and thus may have made you think of the word “butter” over the many other fitting alternatives.

… “barter,” “banter,” “beater,” “bitter,” or even “better.” …

Avoid Priming

Chances are that your word was “butter.”

B_ _ T_R


KP Ludwig John

Test scenario


Exposure to one stimulus (initial sentence about the aroma of bread)

influences a subsequent response to a different stimulus (template to fill in)

Avoid Priming


KP Ludwig John

Test scenario


When tasks use terms already present in the interface, participants are primed by the task formulation and tend to look for those exact terms.

When asking people to search for an “iPad keypad.” Although the word “keypad” is not as frequent as “keyboard”, most users typed “keypad” instead of “keyboard” in the search box.

… the test may have yielded the wrong conclusion that the product was easy to find.

Task wordingAvoid Priming in User Research


KP Ludwig John

Test scenario


A question such as “What do you think this button does?” can warn users that the indicated UI element is a button, although they may not have realized that it was active.

The facilitator can elicit (hervorlocken) a certain behavior by demonstrating it to the user.

In a children study, we redirected a lost user back to the browser’s homepage by pressing the browser’s Back button several times.

Although before that event the child had never used the Back button, afterwards she used it in almost every task.

Facilitator’s behaviorAvoid Priming in User Research


KP Ludwig John

First steps

Determine:

What do you want to find out?Which elements of the application are crucial?

Formulate the tasks straight forward, short and precise

Each task should have a defined goal(that defines, when the task is done)

Test scenario Content and style

!Start with simple, easy to handle tasks!More difficult (or more complex) tasks are possible,but not good at the very beginning of the test)


KP Ludwig John



KP Ludwig John

Test leader although positioned next to the proband,remains silent / interferes only in critical situations.

Don’t get into conversations with the test person during the test itself. That can be done after the test.


Let the test person act rather independently and on its own


KP Ludwig John


Try out the tasks with team members or random usersbefore starting the official test

Set up test conditions close to the real context of use


KP Ludwig John

Test scenario Structure


• My name is Martin and I am guiding you through this test • First, we would like to thank you for participating in our test, we really appreciate your help and the time you took for this.

• Very important before we get started: We are not going to test you in person, we try to test different layouts for a magazine, so don’t worry, you can’t do or say anything wrong.

1. Welcoming and short introduction by test leader


KP Ludwig John

Test scenario


• If there are any questions or misunderstandings, just ask.

• Vivi, Kate, Daniel and Chris are here to observe the whole testing sequence. Ruben is our technical manager.

• You will have to do some tasks here in the UX-Lab, which I am going to explain later in detail.

• Bastian is a member of Academica. He will guide you to another room where you are going to get your last task,which is answering some questions in a questionnaire.

2. Introduction of the team members and their functions

Structure


KP Ludwig John

Test scenario


For the following test here in the UX-Lab, my team members will be observing the test.

We would like to do audio recording during the test. We are not going to provide this data to 3rd parties, but if you don’t want your voice being recorded, feel free to tell us.

3. Declaration of consent

Structure


KP Ludwig John

Test scenario


There will be 3 different PDFs which contain several information.

• You have to do some tasks, they will be displayed on the screen. Pressing space on the keyboard will start the first question,

• We would like to encourage you to tell us what you are thinking while doing the tasks. It helps us analyzing your final answers, but don’t feel forced to comment if you don’t feel like it.

• If there are no more questions from your side about the test procedure itself, you can start by pressing the SPACE key

• After a short calibration of the eye-tracking software the actual test is going to start immediately.

4. Definition of tasks

Structure


KP Ludwig John

oder:

Testszenarien entwickeln

Aufbau

Beispiel: IA6 2013 A Serious Game (Kinect)

Begrüßungstext


KP Ludwig John

Tasks

Source: IMS.UX 2017, Compare 3 Print-Layouts, Team UXperts


What’s the address of the company?Layout is shown, subject starts task.Subject gives answer

Does the company offer topics for Bachelor Thesis?Layout is shown, subject starts task.Subject gives answer

How many employees work at the company?Layout is shown, subject starts task.Subject gives answer

Each task should have a clear goal(that defines, when the task is done)


KP Ludwig John

Tasks

Source: IMS.UX 2017, Compare 3 Print-Layouts, Team Xperience


Questions indicates meaning of UI element

Look for the rating points in the pdf. Find one criteria which has a bad rating and tell us.

Subject gets 2-3 minutes to look closer at the layout.We ask the user to talk about the parts he is looking at.

- Some critical examples

No clear goal, unnatural situation

Why critical?


KP Ludwig John

Expression of Thanks

Post test Interview to collect impressions and to discuss critical situation observed during the test

probably:


or probably

Thank you very much for participating in our test.

We would like to offer you this little gift to thank you.

After completing the tasks


KP Ludwig John

Test scenarioDuration

Rule of Thumb:Between 10 -30 minutes per person is OK

• less than 10 minutes seems rather short as well to the test person as for the team

• longer than 30 minutes requires a lot of effort from both sides

depends on test goal and circumstances


overview

KP Ludwig John

PracticePrepare your test

Formulate your test scenario in detail!

Plan and organize your test:

Appoint Functions and responsibilities in the test team Find and appoint test persons Schedule the test (incl. equipment and location)


overview

KP Ludwig John

PracticePrepare your test

• Test goals • Criteria of Success and Measurements for that criteria • Target group • Prototype • technical equipment • scenario • post test interview (if applicable)

Summarize your test design into one document, including

Be prepared to discuss it in detailnext week

hs-augsburg.de/homes/john

Documents

UX-Research 3 Scenarios v2019 - HS Augsburg€¦ · UX Research test scenarios KP Ludwig John Usability Test Preconditions Already existing ... This encourages the user to explore