Five Ways to Get Better Data From Our Users

Lean User Research: making design research work

for those that ship

Sajid Reshamwala // Jut.io // February 2015

Where do our design problems come from?

A tough question to warm up.

How do we know when we're not solving our

users' problems?

A slightly easier one.

(data) We know how to measure and we do it well.

How do we understand why we're not solving our customers' problems?

This ones tougher, and usually leads to lots of assumptions.

The process is usually like this. It kicks off with a hunch or an insight. Sometimes even an insight from a user.

We then come up with a bunch of solutions for what we can do.

We decide on one that we think is our best bet.

Throw it into a sprint.

Get it out into the world.

And measure the data we get back.

Pretty familiar and close to lean startup methodology.

Especially great when iterating on a problem.

But what problem are we focusing on?

Segway. Brilliant team building brilliant tech.

But, after being built in a closed environment, didn’t account for how people would interact with it in the wild.

Recognize this guy? Color. Talented team with $150mil + funding.

"I thought we were going to build a better Facebook ... within 30 minutes I realized, Oh my God, it's broken. Holy shit, we totally fucked up." -Bill Nguyen, Color

But far from brilliant execution. But this is the risk that comes with being in a startup, right? I mean, right?

Well we all know we need to start with our user. In our case, that’s our dev ops guy, Kyle. But how do we understand his problems?

Well we’ve got all this quant data about him. Why don’t we start from there?

Well, understanding problems with just quant data can be tough. I’ll explain. Lets imagine that we’re making Kyle some dinner.

And so we make him this awesome burrito.

Unfortunately, we learn that he doesn’t like it. So we make it awesomer.

Hmm. Still doesn’t like it. Mk. Well maybe we try and put that salsa underneath instead of on top. Boom.

But that’s not going to help since Kyle just wanted some apple pie.

The tough problem with data is its really great mostly for iterating, and its tough to iterate on a burrito and ever end up with apple pie.

You see this websites all the time. The optimize an element on a page which marginalizes the value of the whole page all together.

And they end up loosing what they’re focus was all along.

Innovating in the

Wrong Direction

In fancy mba talk, they call this innovating in the wrong direction.

"Too many teams perfectly execute the

wrong plan."this guy said this thing.

How do we get closer to perfectly solving the right

problem?

ok, well quant data + growth hacking doesn’t do the full trick…

but we still know we need to get to this guy’s problems

to do that, we need to get in his head

yup.

so we can talk to users, right? I mean, right?

GOOB Get out of the Building

This is what lean startup tells us to do. Get out of the building. And this is totally the right intention.

But we know we’ve all been there. We talk with a user about what we’re making, and they love it.

But they go and play with it, and the results a very different story.

And it’d happen with us. (via Malcom Gladwell) If I were to ask you guys how you like you’re coffee

most of you would probably say you like it dark and robust.

But the truth is, most of us like it light and milky.

"Humans are horrible at predicting their future behavior" -Daniel Khaneman, Thinking Fast and Slow

That’s not because we’re dishonest, its because of this. Just talking with users can be dangerous.

Companies like Walmart have run into this when reacting to customer surveys to tidy up their isles. The result cost them +billion and resulted in them re-cluttering their aisles.

So we can’t just talk with our users. How else can we understand this guy’s problem’s?

Well a more direct way we can understand what’s going on in his head is to watch what he’s doing.

And more precisely, to watch what he’s doing in his natural context.

Whatever context that may be.

User Research Understanding user behaviors, needs, and motivations through observation techniques, task analysis, and other feedback methodologies.

That’s what user research is.

And this is what the most interesting companies in our industry realize is an important skill set in their teams.

"At Airbnb we have user researchers, engineers and designers all within six feet of us." - Andy Orsow, AirBnB

And that’s for companies big and small.

How?

Aight, cool. So how do we do it?

User Research Sandwich

I like to think of traditional user research as kind of a sandwich.


Its easier to imagine it sitting on its side. Give it a sec.


Understand The Problem

The front end of the sandwich is understanding the problem. This is usually when you’ll here of fun ethnographic exploratory research that companies will do.



Understand How to Solve it

Better.

After you start making stuff to solve your problem, the second slice of bread is about understanding how to solve the problem better. You’ve probably heard this referred to as usability.

Lean User Research Sloppy Joe.

In lean user research i like to think of it more as a sloppy joe.




Better.

Both pieces are still there.




Better.

But we let them run together to keep us moving fast.

Lean User Research is Based on 3 Things:

1. Its not a big deal

For us to pull off lean user research, there’s three tenants. We need to stick by. First, its not that big a deal. We keep it light, we keep it frequent.


1. We all smell users

Like everything else in lean, we all need to be a part of it to make it fast. This keeps us from having to deal with tedious reporting and death by docs.


3. There is methodology.

And lastly, while it is light and quick, there is a right way to do it.

On that note...

5 simple ways to get better data

from our customers.

1. Don't Listen, Watch.

The intent of listening to our customers is the right one. But there’s a better way to get better data.

"people greatly overestimate the reaction they will have to both pleasant and unpleasant events that happen in one’s life" - Dan Gilbert, Stumbling Upon Happiness

People are bad at prodicting how they’re going to use stuff in the future and how they’ll react to it.

That’s why these questions, while so tempting, are the most dangerous ones we can ask.

Asking theoretical questions is often considered research theatre because it seems like we’re getting good data when we’re actually just making ourselves feel good.

Even worse, this bad data can be used to give us false confidence. It can be worse than doing no research at all.

The introspection illusionpeople wrongly think they have

direct insight into the origins of their mental states

Another reason that asking theoretical questions is dangerous is because people are great at answering ‘why’, but not great on giving you the real reason why they do stuff.

A great heuristic is that you want people leaning in and doing stuff.

As soon as they’re leaning back, they’re rationalizing. This is a sign that you want to get people back to completing some sort of task.

Can't?

Well that’s great… if you have the chance to watch people do stuff. Well, what if you can’t?

The next best thing to watching people do stuff is to ask about a recent time.

This will get people talking about stories.

This phrase works because it gets people talking about recent events that are specific.

2. Observe in Context

Tip number 2: watch people in their natural context. This is an easy thing to forget when working in tech. Especially when designing for desktop.

It feels like we’re usually just designing for our window.

But that window is often one of many and is effected by all the people and things around it.

3. Features are our Job. Look for the why.

This one’s probably the most contentious of all.

If we made a hierarchy of good user research queries…

In the fiery depths of hell would be theoretical questions.

At the top you would have watching people.

Next up you would have stories about recent times.

And somewhere in the middle you’d have ‘what features do you want.

The reason features or dangerous is because they often just reference part of the problem.

If we do get feature requests, thats our cue to try and understand the problem that drives it.

We can then use our understanding of the problem to better understand what we should build.

Its kind of like the allegory of the cave. Feature requests are like the shadows on the wall. We don’t need to go too deep on this one.

4. Watch for 3 ThingsRoutines

Interrupts Transitions

Tip 4. Watch out for 3 things (a shortened version of Tomer’s list of 5).

What rituals do they have?

What do people do repetitively?

What interrupts them?What disrupts their flow?

Where do they transition?

What are the inbetween moments that connect disparate activities?

5. Watch 5 - 6 People Using Our Product

Number 5, when it comes to understanding how we can better solve the problems we’re trying to solve, watch 5 - 6 people.

Research shows that we can get to 80% of usability problems with this heuristic and its a great starting point. That is, as long as we do a good job defining our user and recruiting the

right participants.

5 Ways to Get Better Data from our customers.1. Don't Listen, Watch. 2. Observe in Context 3. Look for the why. 4. Watch for Routines, Interrupts, and Tansitions. 5. Watch 5 - 6 People Using Our Product

And that’s a good starting list of how we can start getting better data.

One more thing...

Oh yeah.

Apple’s got this great notion of getting to 1000 no’s.

It's scary to be wrong.

And that’s the tough part that this won’t fix. We’re going to be wrong and that feels scary.

But if we focus on finding better problems…

Then we can find a problem that we can fall in love with and come up with many better solutions that we’re ok with throwing away.

Learn fast and perfectly solve the right problems.

And hopefully we’ll get here.

Design

Five Ways to Get Better Data From Our Users