Science and Sensibility: Thoughts on Experimentation and Growth

Hilary Roberts | Senior Product Manager @Skyscanner | @hilcsr

sciencesensibility&

13 years ago

@hilcsr

today

@hilcsr

0"

10"

20"

30"

40"

50"

60"

monthly unique visitors (millions)

@hilcsr

learn buildminimise total time

measure

lean startup

@hilcsr

startup enterprise

L B

M

L B

M

@hilcsr

startup at scale

L B

M

L B

M

L B

M

L B

M

L B

M

@hilcsr

0

75

150

225

300

Jan-15

Mar-15

May-15

Jul-15

Sept-15

Nov-15

Jan-16

Mar-16

May-16

Jul-16

Sep-16

experiment iterations

@hilcsr

0

75

150

225

300

Jan-15

Mar-15

May-15

Jul-15

Sept-15

Nov-15

Jan-16

Mar-16

May-16

Jul-16

Sep-16

experiment iterations hypotheses

@hilcsr

hypothesis

Based on insight, we predict that product change

will cause impact.

@hilcsr

hypothesis

Based on the insight that travellers disproportionately choose certain holiday

destinations, we predict that ordering destinations by popularity instead of price

will cause more people to convert.

@hilcsr

two possible experiment outcomes

successful failedthe hypothesis was correct the hypothesis was incorrect

@hilcsr

0

75

150

225

300

Jan-15

Mar-15

May-15

Jul-15

Sept-15

Nov-15

Jan-16

Mar-16

May-16

Jul-16

Sep-16

experiment iterations hypotheses

@hilcsr

case study No.1

Based on the insight that active users of our flights product are also likely to like our hotels

product, we predict that landing returning travellers on our hotels homepage instead of

our flights homepage will cause more travellers to use both products.

@hilcsr

case study No.120% increase in travellers booking both flight and hotel

Negligible impact on overall flights metrics

@hilcsr

Skyscanner you are really annoying me with your hotels and car hire options! You are not called HotelScanner or CarHireScanner. You are Sky as in aeroplanes!

If you want to diversify into other services then you should have thought of that before calling your website SKYscanner. Imagine Compare The Market had limited themselves by calling it Compare The Car Insurance Market!

They can now ease into the Travel Insurance market with no issue because they didnt specify which market they are comparing. You cant! and please stop defaulting me to Hotels I'm here for flights!!

@hilcsr

case study No. 120% increase in travellers booking both flight and hotel

Negligible impact on overall flights metrics…but some complaints and added friction for >95% of travellers

bottom line Achieved the desired impact, but qualitatively was not the experience we wanted to provide.

@hilcsr

We could not measure the downside in the experiment.It never could have failed.

invalidnot possible to fail

successful failedthe hypothesis was correct the hypothesis was incorrect

possible experiment outcomes

@hilcsr

invalid tests

Observed outcome: surprisingly one-sided results.

Characterised by inability to measure the upside/downside.

Typically occur when you’re in love with your idea.

Can be avoided by taking hard decisions up-front.

Trigger: “If we can’t agree, why don’t we just test it?”@hilcsr

case study No. 2

Based on the insight that our “Aha!” moment is when travellers conduct their

first search on Skyscanner, we predict that adding search controls to our travel articles will cause more readers to become active

users of our product.@hilcsr

@hilcsr

case study No. 2Negligible increase in percentage of travellers who did a search.

Negligible difference between the variants.

bottom line Our solution didn’t match the context.

Why did we run this test?

@hilcsr

The method we chose was not capable of having the impact we wanted.It never could have succeeded.

invalidnot possible to fail

successful failedthe hypothesis was correct the hypothesis was incorrect

flailednot possible to succeed

possible experiment outcomes

@hilcsr

flailed tests

Observed outcome: Zilch.

Characterised by mismatch between desired impact and proposed solution.

Typically occur when over-focusing on ‘MVP’.

Can be avoided through broader brainstorming.

Trigger: “It’s quick and easy. Let’s just try it.”@hilcsr

invalidnot possible to fail

successful failedthe hypothesis was correct the hypothesis was incorrect

flailednot possible to succeed

two four possible experiment outcomes

@hilcsr

“Any time a team attempts to justify its failures by resorting to learning as an excuse, it is engaged in pseudoscience….

We cannot afford to breed a new pseudoscience around pivots, MVPs, and the like.”

The Lean Startup (page 279)@hilcsr

We cannot afford to breed a new pseudoscience around being data driven.

@hilcsr

Invalid or flailed experiments can look and feel data driven, but they’re pure waste.

@hilcsr

We’re improving our experiment design through experiment and hypothesis templates,

thought experiments and a culture of peer-review.

@hilcsr

To avoid waste, we must balance both science and sensibility.

@hilcsr

More about what we’re learning at Skyscannerhttp://codevoyagers.com

UX comic library from@steve_cable

Hilary Roberts | @hilcsr

Thank you