Exploring the UX Research Toolbox

Exploring the research toolbox: what to use, when and why.

Calgary - Vancouver - Toronto

Anthony HempellDirector, User Experience

Tara Franz,Director, Research

Part 1: Why do research, anyway?Part 2: Understand the landscapePart 3: Pushback & pitfallsPart 4: Exploring the toolboxPart 5: Case Study: ATB

Part 1:

Q: Why do research, anyway?

Part 1:


“If we knew what it was we were doing, it would not be called research, would it?” -Albert Einstein

Part 1:


“If we knew what it was we were doing, it would not be called research, would it?” -Albert Einstein

“I believe in innovation and that the way you get innovation is you fund research and you learn the basic facts” - Bill Gates

Why Research?

Why Research?

measurements, facts, observations

“know nothing”words, numbers, symbols

Why Research?

descriptions, correlation, comparison, relationships

“know what”

meaning & purpose

Why Research?

framed experiences, values, context, expertise, intuition;

organizational routines, processes, practices and norms

understanding & capability

“know how”

Why Research?confidence to act

judgement, ethical and aesthetic values

“know why”

So how do we do that?

gather server logs, conduct surveys, perform usability tests

comparison & analysis of data

communicate and evangelize findings

turn insights into policies, methods and actions

Part 1: A: We want to fill our gaps in the data to create useful information, knowledge & wisdom which can be used to make the world a better place.

Part 2: Understand the landscape

Part 2: Understand the landscape

“If you know the enemy and know yourself, you need not fear the result of a hundred battles.” - Sun Tzu

Some practical questions before we

begin...What do we already know?

Who are the users? Who are the stakeholders?

Where are they located?What are we trying to discover?When do we need our findings?

How are the findings going to be used?

What do we already know?

What research results do we already have access to?

How recent is it?

How was it collected?

Is it valid?

What does it tell us?

Has it been converted into organizational assets like policies, procedures, shared knowledge?

Who are the users?

What research has been done on current audience?

Is any of it qualitative?

How has the current audience / user / customer base been determined?

Are there demographic aspects to customer base that are of interest -- language, age, mobility, cultural, technology use, etc.

Who are the stakeholders?

Who has ultimate accountability (financial and otherwise)?

Who is the primary point of contact / responsibility for coordination?

Who else needs to be consulted? What are their roles?

Who needs to be informed and when?

Who would be most affected if the project goes poorly?

What are the desired outcomes + effects?

Where users + stakeholders located?

Local? Provincial?

National? International?

Urban / Suburban / Rural

Regional attitudes / behaviours?

When do we need our findings?

Deadlines -- what is driving them?

Possibility of future research?

How are our findings going to be used?

Creating requirements?

Concept creation or validation?

Setting policies / product direction?

Within small project team or communicating across departments?

Part 2: Gather as much data about your surroundings as possible.

Part 3: Pushback & Pitfalls


“You can go anywhere you want if you look serious and carry a clipboard” - Murphy’s Law


“You can go anywhere you want if you look serious and carry a clipboard” - Murphy’s Law

“We don’t devote enough scientific research to finding a cure for jerks” - Bill Watterson

Some common pitfalls“We already have a research department

that does that”

“Marketing did some market research last quarter, we can use that”

“Let’s put a survey up on our website”

“That sounds like it will take too long / be too expensive -- let’s talk to some of my

friends / co-workers / kids instead”

“Let’s get the users to tell us what they want” (a.k.a “Rainbow Unicorn”)

Some common pitfalls“We already have a research department

that does that”

“Marketing did some market research last quarter, we can use that”

“Let’s put a survey up on our website”

“That sounds like it will take too long / be too expensive -- let’s talk to some of my

friends / co-workers / kids instead”

“Let’s get the users to tell us what they want” (a.k.a “Rainbow Unicorn”)

Some common pitfalls - why they happen

Organizational pressure for concrete results, yesterday

Research not seen as valuable / waste of time

Lack of knowledge about research methods

Lack of understanding of research benefits

Quote

Research

Strategy

Design

Culture

Testing

Novice Beginner Intermediate Advanced Enlightened

NoneAd-hoc,

AnecdotalData-driven

Qualitative + quantitative

Triangulation: data & observation;

passive & active.

Not used

Ad-hoc

Never

No knowledge

Reactive; HIPPO

Creative Chaos

Rarely

Awareness of value

Focus on business requirements

"Best practices"

Occasionally

Individual champions;

traction dif!cult

Balance needs of business & users

Style guides + processes de!ned; ideation & creativity

Sometimes

Some managerial advocates +

success

Business is customer-driven

Agile / Lean: small teams, rapid design sessions, iterative

Always

Empowered by C-level executive

Adapted from: Shane Morrishttp://blogs.msdn.com/b/shanemo/archive/2006/12/18/user-experience-maturity-model-microsoft-style.aspx

"We don't have time / money for

that!""Just do it."

"Think before you act."

"Use lessons learned to predict, plan and create opportunity."

"Review, learn, adjust, execute, review, learn…"

Jakob Nielsenhttp://www.nngroup.com/articles/usability-maturity-stages-1-4/

0 1 2 3 4

UX Maturity Model

Quote

Research

Strategy

Design

Culture

Testing


NoneAd-hoc,




passive & active.

Not used

Ad-hoc

Never

No knowledge

Reactive; HIPPO

Creative Chaos

Rarely

Awareness of value


"Best practices"

Occasionally


traction dif!cult



Sometimes


success



Always




that!""Just do it."





0 1 2 3 4

Quote

Research

Strategy

Design

Culture

Testing


NoneAd-hoc,




passive & active.

Not used

Ad-hoc

Never

No knowledge

Reactive; HIPPO

Creative Chaos

Rarely

Awareness of value


"Best practices"

Occasionally


traction dif!cult



Sometimes


success



Always




that!""Just do it."





0 1 2 3 4

Company A

Company B

Company C

Company D

Some common pitfalls - how to avoid

Know your research question. Always have concrete, measurable goals for your research that you can refer to.

Build a research strategy and plan.

Be willing to listen to alternative points of view, but don’t deviate from sound research practices.

Protect the validity of your methods. It is your basic currency.

If you’re getting pushback, take baby steps.

Focus on the value.

Don’t force it. If the glass slipper doesn’t fit, it doesn’t fit.

Part 3: Observe and understand the organization you’re working with. Propose measured improvements. Make friends.

Part 4: Exploring the toolbox

“If all you have is a hammer, everything looks like a nail.” - Abraham Maslow

Focus groupsIn-depth interviews (in person / remote)EthnographyCard sortingUsability testingUsability benchmarkingSurveysObservationMessage board miningCoding customer feedback emailsA/B testingWeb AnalyticsDiariesEye trackingOnline Panels / Communitiesand more...

http://nform.com/cards/

Let’s look in the toolbox

Research dimensionsBasic vs AppliedCross-sectional vs LongitudinalResearch purpose: Exploratory, descriptive, explanatory

Data collection techniques:Quantitative vs QualitativeSmall vs Large sampleModerated vs UnmoderatedAttitudinal vs Behavioural

http://qualitative.wikidot.com/dimensions-of-research

Mod

erated: high eff

ort / high context U

nmoderated: low

er effort / lower context

Moderated• Can observe subtle cues:

non-verbal behaviour, tone of voice, etc.

• More chances for further inquiry based on context

• Harder to eliminate biases• Much higher quality of data

Mod

erated: high eff


nmoderated: low


Unmoderated• Can facilitate much larger sample

sizes• Not as many constraints with

different 9me zones• Limited context• Media biases• Respondents have more 9me to

par9cipate (send photos, diaries, ponder thoughts)

Quan8ta8ve: how many and how much

Qualita8ve: why and how to fix

• Small test group but deeper understanding• Gathering informa9on or themes from texts, conversa9ons or

interviews• Open ended; changeable. There is a maybe. • Opportunity for flexibility; serendipity



• More closed questioning with directed response (pick Yes or No, there is no maybe)

• Gathering data with an “instrument”• Derive measures or variables (operationalization)• Error of measurement



• Hybrid: gather data qualita9vely and then code into variables to make inferences quan9ta9vely.

Mod

erated: high eff


nmoderated: low




Mod

erated: high eff


nmoderated: low




Mod

erated: high eff


nmoderated: low




Discovering mo9va9ons in context

Mod

erated: high eff


nmoderated: low



Qualita8ve: why and how to fix BLUE = measures aUtudes

RED= measures behaviours

Focus groups

In person interviews

Ethnography

Usability tes9ng

Telephone interviews

Card sor9ng

Diaries

Online panels

Mod

erated: high eff


nmoderated: low




Observing and tracking behaviours in context

Mod

erated: high eff


nmoderated: low




Observa9on

Usability benchmarking

Eye tracking

BLUE = measures aUtudes


Online panels

Mod

erated: high eff


nmoderated: low




Discovering general themes and aUtudes

Mod

erated: high eff


nmoderated: low




Message board mining

Customer feedback emails



Mod

erated: high eff


nmoderated: low




Gathering data about behaviours

Mod

erated: high eff


nmoderated: low




Web Analy9cs

A/B tes9ng


Unmoderated usability tes9ng



Mod

erated: high eff

ort / high context

Unm

oderated: lower effort / low

er context



Focus groups

In person interviews

Ethnography

Surveys

Web Analy9cs

Observa9on

Usability benchmarking

Usability tes9ng

A/B tes9ng

Message board mining


Telephone interviews

Card sor9ng

Unmoderated usability tes9ng

Eye tracking



Diaries

Digital Ethnography

Online panels

Opportunities:Triangulation

What data do you currently have? How was it collected? Consider using a method that is complimentary.

Do you have the opportunity to run two or more research methods? Use different methods to look at the same problem from different angles.

Part 4: Know the strengths and weaknesses of every method and pick wisely. If possible, research with more than one.

Part 5: Case Study: ATB Financial

“Our youth now love luxury. They have bad manners, contempt for authority; they show disrespect for their elders and love chatter in place of exercise” - Socrates

• Problem: –Required insight and

understanding of the millennial demographic to increase market share within the group

–Make something

awesome

ATB Case Study

• what do we already know? – Lots of data about ATB customer needs and

current use –Also were aware of Millennial research and

some overarching themes

ATB Case Study: Landscape

• Who are the users? –Target of urban Millennials and drafted up a

demographic framework to contain the sampling

• Who are the stakeholders? –Emerge group at ATB and big wigs –Ensure we had a plan to have them involved


• Where are they located?–Calgary, Alberta

• What are we trying to discover? –Something we can take to market to engage a new

market for ATB


• When do we need our findings? –We had 3 months time

• How are the findings going to be used? – To create a new offering


• The plan:- Secondary research review on Millennials and ATB data - Ethnographic Immersion – be one with the animals

- DEBRIEF (this is where the real insight starts to happen and stakeholders are involved) - User Interviews – understand the ways and the why

- DEBRIEF again – what have we learned? What else do we want to know? Let’s get our persona on!

- Naïve Expert and Extreme Customer Interviews (Kingdon; Science of Serendipity. 2012)

- DEBRIEF – what have we learned about solving problems?- Innovation session - The big write up and vote - In field feedback (this is when you call your friends)

- RDL and prototyping- Concept testing - Finalize and forge ahead!

ATB Case Study: Toolbox

- How was everyone kept in the loop?

Collaboration

Remember the triangle?

- Secondary research review on Millennials and ATB data

- Ethnographic Immersion – be one with the animals - DEBRIEF (this is where the real insight starts to happen and

stakeholders are involved)

Stage One: Foundation

- User Interviews – understand the ways and the why - DEBRIEF again – what have we learned? What else do we

want to know? Let’s get our persona on!

• Semi structured phone and in person ethnographic interviews: we dined, we coffee-d, we chatted, we made life plans together

• Debrief goal: Personas

Stage Two: Patterns

- Naïve Expert and Extreme Customer Interviews (Kingdon; Science of Serendipity. 2012)- DEBRIEF – what have we learned about solving problems?- Innovation session - The big write up and vote

Stage Three: Innovation

• Explored creative ways to solve the problems we had: personal organizers, teachers, support workers, rehabilitation counselors

• Spoke with people who were on the extreme side of everything we discovered: wealthy and homeless

• Debrief goal: 5 solid ideas with a framework. • Opportunity to vote on favorite idea • In field feedback

- RDL: include your stakeholders and sometimes users - Prototyping- Concept testing:

- recruit the right people - Test on the right screen

- Finalize and forge ahead!

Stage Four: Building

Anthony [email protected]

Web: www.uxguys.comTwitter: @uxguys

Thank You!

Tara [email protected]

Web: about.me/tarafranzTwitter: @tarafranz