Exploratory user research (How to figure out what to test)

@DhyanaScarano

Exploratory ResearchHow to figure out what to test

@DhyanaScarano

• To have a drink and meet some cool people?

• To learn some new methods and tools?

• To learn about conversion optimisation and heuristic analysis?

• To hear how other people do explorative research?

• Something else…?

Why are you here?

@DhyanaScarano

Who am I?

• Senior UX Designer at CareerFoundry

• Co-organiser of Service Jam Berlin

• Co-organiser of UXcamp Europe

• Head mentor (product strategy), Google Launchpad startup bootcamp

• Asks a looooot of questions

Dhyana Scarano (Dee)

http://www.jamberlin.org/

http://www.uxcampeurope.org/

https://developers.google.com/startups/

@DhyanaScarano

• A bit of product strategy

• A bit of research methods & tools

• A look at a real exploratory research project underway at CareerFoundry

• A little bit about how all that feeds into conversion optimisation

What’s in this talk?

@DhyanaScarano

• A little bit of product strategy

• A little bit of research methods & tools

• A hands on exercise (yay!)

• Talk about all that feeds into conversion optimisation

How does CO think about research?

@DhyanaScarano

And what about Design Thinking?

Research Underst-anding DefineTesting

ideas

@DhyanaScarano


Research Underst-anding DefineProblem Testing

ideas

@DhyanaScarano


Research Underst-anding DefineProblem The real

ProblemTesting ideas

@DhyanaScarano



Problem SolutionTesting ideas

@DhyanaScarano



Problem Solution

Defining the problem Coming up with a solution

Testing ideas

@DhyanaScarano



Problem Solution

Defining the problem Coming up with a solutionFiguring out what to test Testing options

Testing ideas

@DhyanaScarano


Research Underst-anding

Testing ideas

DefineProblem The real Problem Solution


(what we’ll talk about now) (Conversion optimisation)Figuring out what to test Testing options

@DhyanaScarano

So, it always starts with a problem…

ResearchProblem

Defining the problem

Problem: CareerFoundry needs to increase it’s conversions

@DhyanaScarano


ResearchProblem


So how do we figure out what makes customers click “enrol”?


@DhyanaScarano


ResearchProblem


So how do we figure out what makes customers click “enrol”?


But wait. First, what does CareerFoundry do?

@DhyanaScarano


@DhyanaScarano


How do we figure out what makes people click this?

@DhyanaScarano


How do we figure out what makes people click this?

But then we need to find the “real problem”

We can test different things based on what we already know

We can also try and learn something totally new about our users that we don’t know yet!

@DhyanaScarano


How do we figure out what makes people click this?We can also try and learn something totally new about our users that we don’t know yet!

Tweaking based on what we already know only gets us so far.



@DhyanaScarano


How do we figure out what makes people click this?We can also try and learn something totally new about our users that we don’t know yet!

New insights can lead to bigger jumps in conversion

Tweaking based on what we already know only gets us so far.



@DhyanaScarano

ResearchProblem


Original problem: CareerFoundry needs to increase it’s conversions

How to do the research?

@DhyanaScarano


Testing ideas

DefineProblemThe real Problem


Original problem: CareerFoundry needs to increase it’s conversions more people to click “Enrol”


We can’t easily find the people who decided not to enrol with us.

But we can ask the people who did choose us.

Another challenge: our users are all over the world, from the U.S. to China, and even Australia!

@DhyanaScarano


Testing ideas



What was in their head just before they decided to click?

What were they afraid of?

Did they have any doubt or fears?

What was the thing that helped them decide?


What can we ask that will help us figure out what to put on our landing pages?

@DhyanaScarano


Testing ideas



What to ask?• We’re trying to discover new

information, so…

• Really open questions

• Get them to tell their story


How to ask it?• Need to explore and go deeper

(interviews)

• Also need enough data to draw conclusions (survey)

@DhyanaScarano


The survey (For enough data)

• Lots of free text responses, for thoughtful answers

• Open questions. Who, Why, What, How, Can you tell us…

• Asking them to tell the story of their decision

• Try to discover things we don’t know we don’t know

At the end of the survey we asked if they would be interested in doing a 45 minute interview for a $25 Amazon voucher

@DhyanaScarano


The interviews (For deeper insights)

• 45 mins

• Field guide / conversation points

• Only seen by the interviewer

• Skype or appear.in used for video chat

• Topics tailored to go deeper in to the survey questions

@DhyanaScarano


The interviews (For deeper insights)

• Recorded audio with a field recorder

• Transcribed notes in Google Docs for easy sharing

• Highlighted interesting quotes while typing for easy reference later

Tip: Amazon vouchers were very difficult in multiple currencies. For the second round of interviews we didn't offer an incentive and tried personally messaging users. This took more manual effort but a higher percentage of users agreed to an interview.

@DhyanaScarano

How to do the research?Scheduling interviews

• Services like youcanbook.me and calendly.com let the interviewee book a time that suits you.

• About 2 hours to set up and customise

• Syncs with your calendar and only shows available time slots to the customer

• You are notified when someone makes a booking, and it appears in your calendar

http://youcanbook.me

http://calendly.com

@DhyanaScarano

Underst-anding

Testing ideas

DefineThe real Problem Solution


• Frustration at current job Very high hopes in CF as a life changer

• Fear of failure & lack of confidence

• Time constraints biggest concern Spend more time than anticipated

• Longing for community (also offline) UI & UX students much more so than WebDev and iOS

General insights

Understanding (the user)

@DhyanaScarano

Course specific insights (e.g. WebDev)

From interviews

• Age: 33-53 (older than other courses)

• Diverse professional backgrounds, often don’t live in a TechHub

• Personal projects or business often a motivation to learn coding

• Self starters with free code schools, need in-depth knowledge/skill polishing

From survey

• 80% interested in WebDev for some months or years

• 60% considered a career path as a WebDev before signing up for the course

• 40 % considered another CF course

• 70% want to change career or job, some interested in freelancing


@DhyanaScarano

New insights

“I’m still apprehensive about posting on Slack [...]I’m worried that if I make a stupid mistake I’ll

feel silly.”

“ One of my biggest concerns was that after finishing the course my skills wouldn't be job ready.

Since you added the job guarantee it instilled more confidence that the course is robust.”

“My father was a computer scientist and he taught me how to code, I realized that I always

liked to code.”

“After reading the reviews, watching the YouTube channel and all the great feedback from the staff at CareerFoundry, I've no doubt

that this will be the step I need to take.”

Validation

“I do have a small concern that my work will not be polished enough in six months to be able to

successfully gain employment as a UI Designer.”


@DhyanaScarano

Underst-anding

Testing ideas

DefineThe real Problem Solution


Next?

• Next we’ll update our user personas(different personas for each course)

• Start to apply changes/tests to our home page, course landing pages and other parts of the service based on these new insights.

• E.g. testing different value propositions based on what users have told us are their goals, doubts and fears.

How does this fit into conversion optimisation?

@DhyanaScarano


Testing ideas

DefineProblemThe real Problem Solution


(what we’ll talk about now) (Conversion optimisation)Figuring out what to test Testing options

@DhyanaScarano

Tools and acknowledgements

• Surveys: typeform.com

• Video interviews: appear.in (Click-and-go. No install required.)

• Interview scheduling: youcanbook.me, calendly.com

People harmed in the making of this work

Tools

• Clive K Lavery (freelance)Research lead, chief interviewer, insight gleaner

• Jeff Humble (CF) Research support, transcriber, Amazon gift card victim

• Claire Rackstraw (CF)Research support, transcriber, motivator

@DhyanaScarano

Thank you :)Questions?