UX-Research 3 Scenarios IAM v2019 - HS Augsburgjohn/IAM-UX/Material/... · UX Research test scenarios KP Ludwig John Hypothesis + Description of MVP Criteria Description of individual

UX Research test scenarios

KP Ludwig John

UX Researchtest scenarios


KP Ludwig John

Before we startDelivery

Hypothesis + Description of MVP


KP Ludwig John

Hypothesis + Description of MVPCriteria

Description of individual Functions • specific and detailed• testable (Put it In or Leave it Out)

Clearly related to the needs and goalsof one or more of your Personas

Traceable (measurable) measurements

based on a clear research question?Which one?

Simplicity of the MVP focussed on research question / hypothesis

MVP

Hypothesis


KP Ludwig John

Feedback

• Hypothesis functions: specific, detailed, testable (Put it In or Leave it Out)

• Measurements traceable?

• Research question clear, focussed and understandable?

• Simplicity of the MVP Could it be done even simpler?

MVP

Hypothesis

Provide your fellow students withRecommendations and Suggestions


KP Ludwig John

Delivery

Hypothesis + Description of MVP

PDF per E-Mail

Deadline: Tonight midnight


KP Ludwig John

Usability TestPreconditions

Already existing

• PersonasDefinition of target groups

• Context of usewhere, when, how long, equipment

• Test goalsWhat do we want to learn?!


KP Ludwig John

• What do you want to learn by the test?Which elements of the IA design have been discussed controversial?Which elements seam to be critical for the general success of the application?

• Which elements of your application have to be coveredMain functions / use casesCrucial functions of the applicationUse Cases with many Alternate Flows

Usability-Test PredefinitionsTest goal


KP Ludwig John

Usability-Test

Comparison of 3 print Layouts Brochure with information about companies

IMS.UX - 2017

Test goal

Source: Team UXperts, IMS.UX 2017

Which Layout is best?


KP Ludwig John

Usability-Test

Source: Team UXperts, IMS.UX 2017

Aspects to consider

Extracting Information How easy is it to get the information? What information can you get?

Memorizing InformationCan you memorize the information after reading? Is it easy to remember

ClarityIs the structure obvious to the user? (Naming, Positioning, Design of elements)

ComprehensibilityDoes the user understand the content?

Readability - Is the text easy to follow?

Look & Feel - How is the first impression? Do you have a good feeling reading the paper?



KP Ludwig John

Usability-Test

Source: Team „Group Title“, IMS.UX 2017

Aspects to consider

How well is information communicated?


How fast can information be perceived and processed?

How much attention does relevant information get?

Measuring units!

Related questions


KP Ludwig John

Usability-Test


Aspects to consider

How well is information communicated?


How fast can information be perceived and processed?

How much attention does relevant information get?

Measuring unit: Time to complete tasks.

Measuring unit: Time a test subject’s focus rests on relevant information.

Measuring unit: Amount of misunderstandings and missed information

Measuring units


KP Ludwig John

paper based software based

Usability-Test

• Which functionality has your prototype to providein order serve your test goals?

PredefinitionsTest object

• What kind of Prototype is needed in order serve your test goals


KP Ludwig John

Usability-Test

• Diary study (remote) • Thinking aloud (with note taking) • Video observation • Screen recording • Eye tracking • Post test interview

PredefinitionsTest methodWhich test method do you want to apply?


KP Ludwig John

Usability-Test

• What kind of equipment serves best the test goals? • Consider testing under of real context of use conditions

(Screen size, Space, time, mobile, distraction, sound)

PredefinitionsTest equipment


KP Ludwig John

Usability-Test PredefinitionsTest scenario

Create a test scenario which defineseach single step of the test in detail,

including

•Introduction •Tasks •and Expression of Thanks


KP Ludwig John

Test location • Where do you want to test? • Availability, Influence of Weather or Light?

Usability-Test PredefinitionsTest space

Test equipment • Where do you get it? • How to handle it?


KP Ludwig John

• Test conductor • Technical manager • Observer • Analyst

Usability-Test PreparationTest team


KP Ludwig John

• Where can the test persons be recruited from? (do they represent the target group, are they available (time) ?) • Do they get some expense allowance (Aufwandsentschädigung)? (which?) • recruit test persons

Usability-Test PreparationTest persons

Find test persons make appointments and a test schedule

• How many Test persons are needed? (about 7-8 per Test version (!) )

UX Research research methods

overview

KP Ludwig John

UsabilityTest persons


KP Ludwig John

It’s best to test with representatives of your main target group.But don’t be obsessed about it!If its difficult to recruit them, test with people you can get hold of. Take it in account when evaluating the test results:Would our users have that problem? Or was it only a problem because the participant wasn’t familiar with the field?

Rather few but frequentlystart early in the development processand repeat and repeat and repeat

Who do the test with?

Usability Test Test persons

How many test persons?


KP Ludwig John

reference:Nielsen (2000) https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Test personsUsability TestHow many test persons?

The best results come from testing no more than 5 users and running as many small tests as you can afford.


KP Ludwig Johnreference:Nielsen 2018: https://www.youtube.com/watch?time_continue=31&v=RhgUirqki50

Test personsUsability Test

Jacob Nielsen N/N Group


KP Ludwig John

Reference: Steve Krug, Don‘t make me think! Redline, 2006


Test with 8 usersfound problems: 5 major ones



KP Ludwig John

Test personsUsability TestHow many test persons?

2 Tests with 3 users each = 6 Test personsfound problems: 5 major + 4 minor = 9 problems

Reference: Steve Krug, Don‘t make me think! Redline, 2006


KP Ludwig John

Wen testen?

Datenquelle: Faulkner (2003): Beyond the five-user assumption: Benefits of increased sample sizes in usability testing; in Sarodnik/Braun: Methoden der Usability Evaluation (Huber Verlag 2011)



KP Ludwig John


source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF, page 19

http://www.nngroup.com/reports/how-to-conduct-eyetracking-studies


KP Ludwig John

Most significant problems will be discovered by small number of test users already.

Finding three participants is easy to handle:test and debrief (Nachbesprechen) the same day! Design small tests with focussed tasks.Learn, move on and test again.


Reference: Steve Krug: „Rocket Surgery Made Easy“, New Riders 2010, chapter 5


Can you use the same participants again?Not for the same task or application.They know too much already. ?


KP Ludwig John

Test scenarios in detail


KP Ludwig John

1. Test person works on given tasksoften under direct observation(screen recording software, video, human observer)

2. Interview with the userto better understand critical situations during the test.Possibility to discuss improvements together

Usability-TestTwo Steps


KP Ludwig John

Content and style

Test scenario

Identify at least 3 essential activitiesto be covered by the test

Create 1 or more tasks per activity to be followed by the test person

Example:

„Become a new member of this online communityand create your individual account!“

„Which of the dishes offered at Mensa next Monday you prefer? Note it on paper.“


KP Ludwig John

Content and style

• complex enough (not to simple) • not too complex either (shouldn’t take too long to work on each single task) • cover all important issues of your test

Test scenario

Tasks have to be


KP Ludwig John

First steps

Determine:

What do you want to find out?Which elements of the application are crucial?

Formulate the tasks straight forward, short and precise

Each task should have a defined goal(that defines, when the task is done)

Test scenario Content and style

!Start with simple, easy to handle tasks!More difficult (or more complex) tasks are possible,but not good at the very beginning of the test)


KP Ludwig John



KP Ludwig John

Test conductor although positioned next to the proband,remains silent / interferes only in critical situations.

Don’t get into conversations with the test person during the test itself. That can be done after the test.


Let the test person act rather independently and on its own


KP Ludwig John


Try out the tasks with team members or random usersbefore starting the official test

Set up test conditions close to the real context of use


KP Ludwig John

Test scenario Structure


• My name is Martin and I am guiding you through this test • First, we would like to thank you for participating in our test, we really appreciate your help and the time you took for this.

• Very important before we get started: We are not going to test you in person, we try to test different layouts for a magazine, so don’t worry, you can’t do or say anything wrong.

1. Welcoming and short introduction by test leader


KP Ludwig John

Test scenario


• If there are any questions or misunderstandings, just ask.

• Vivi, Kate, Daniel and Chris are here to observe the whole testing sequence. Ruben is our technical manager.

• You will have to do some tasks here in the UX-Lab, which I am going to explain later in detail.

• Bastian is a member of Academica. He will guide you to another room where you are going to get your last task,which is answering some questions in a questionnaire.

2. Introduction of the team members and their functions

Structure


KP Ludwig John

Test scenario


For the following test here in the UX-Lab, my team members will be observing the test.

We would like to do audio recording during the test. We are not going to provide this data to 3rd parties, but if you don’t want your voice being recorded, feel free to tell us.

3. Declaration of consent

Structure


KP Ludwig John

Test scenario


There will be 3 different PDFs which contain several information.

• You have to do some tasks, they will be displayed on the screen. Pressing space on the keyboard will start the first question,

• We would like to encourage you to tell us what you are thinking while doing the tasks. It helps us analyzing your final answers, but don’t feel forced to comment if you don’t feel like it.

• If there are no more questions from your side about the test procedure itself, you can start by pressing the SPACE key

• After a short calibration of the eye-tracking software the actual test is going to start immediately.

4. Definition of tasks

Structure


KP Ludwig John

oder:

Testszenarien entwickeln

Aufbau

Beispiel: IA6 2013 A Serious Game (Kinect)

Begrüßungstext


KP Ludwig John

Tasks

Source: IA6 2018, Team Gravecare


Each task should have a clear goal(that defines, when the task is done)

More specific = better


KP Ludwig John

Tasks

Source: IMS.UX 2017, Compare 3 Print-Layouts, Team UXperts


What’s the address of the company?Layout is shown, subject starts task.Subject gives answer

Does the company offer topics for Bachelor Thesis?Layout is shown, subject starts task.Subject gives answer

How many employees work at the company?Layout is shown, subject starts task.Subject gives answer

Each task should have a clear goal(that defines, when the task is done)

More specific = better


KP Ludwig John

Test scenario

source: https://www.nngroup.com/articles/priming/

When tasks use terms already present in the interface, participants are primed by the task formulation and tend to look for those exact terms.

When asking people to search for an “iPad keypad.” Although the word “keypad” is not as frequent as “keyboard”, most users typed “keypad” instead of “keyboard” in the search box.

… the test may have yielded the wrong conclusion that the product was easy to find.

Task wording

Content and style


KP Ludwig John

Tasks

Source: IMS.UX 2017, Compare 3 Print-Layouts, Team Xperience


Questions indicates meaning of UI element

Look for the rating points in the pdf. Find one criteria which has a bad rating and tell us.

Subject gets 2-3 minutes to look closer at the layout.We ask the user to talk about the parts he is looking at.

- Some critical examples

No clear goal, unnatural situation

Why critical?


KP Ludwig John

Expression of Thanks

Post test Interview to collect impressions and to discuss critical situation observed during the test

probably:


or probably

Thank you very much for participating in our test.

We would like to offer you this little gift to thank you.

After completing the tasks


KP Ludwig John

Test scenarioDuration

Rule of Thumb:Between 10 -30 minutes per person is OK

• less than 10 minutes seems rather short as well to the test person as for the team

• longer than 30 minutes requires a lot of effort from both sides

depends on test goal and circumstances


overview

KP Ludwig John

PracticePrepare your test

Formulate your test scenario in detail!

Plan and organize your test:

Appoint Functions and responsibilities in the test team Find and appoint test persons Schedule the test (incl. equipment and location)


overview

KP Ludwig John

PracticePrepare your test

• Test goals • Criteria of Success and Measurements for that criteria • Target group • Prototype • technical equipment • scenario • post test interview (if applicable)

Summarize your test design into one document, including

Be prepared to discuss it in detailnext week

hs-augsburg.de/homes/john

Documents

UX-Research 3 Scenarios IAM v2019 - HS Augsburgjohn/IAM-UX/Material/... · UX Research test scenarios KP Ludwig John Hypothesis + Description of MVP Criteria Description of individual