[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

SUBTITLE BELOW

Optimize for €€€HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS

@AM_Klaassen / @tomvdberg1

How we got here…


The quest for female speakers…


Female speakers…

?


Our lovely colleagues

What we do…


Conversion rate optimization

Analytics Psychology

Lots of A/B-tests


Our clients


Adding

direct value

Learning

user behavior




Van der Valk

Unique website for each hotel

1 centralized team

Paid by % of their turnover


Optimization of Valk

Increase revenue through CRO

Reduce costs per experiment

“We’re not in the business of

science, we’re in the business

of making money”

3 Tactics

1.

Free test software


Jorrin Quest

GTM testing

http://gtmtesting.com

http://gtmtesting.com/


GTM testing

Easy setup

Free

Preview mode

2.

Change of statistics


Focus on finding proof


Example test: Order flow


Example test: Set completion

A B


Test result


• No effect on conversion

Don’t implement

Re-test with higher volumes

Conclusion



What’s the alternative?

Frequentist

statistics

Bayesian

statistics


Example test: Set completionA B


Bayesian Test evaluation


Focus on risk assessment

88,4%

A test result is the probability that B outperforms A:

ranging from 0% - 100%


IMPLEMENT B PROBABILITY * EFFECT ON REVENU

Expected risk 11,6% - € 69.791

Expected uplift 88,4% € 213.530

Contribution € 180.733

* Based on 6 months and an average order value of € 175

Make a risk assessment


Implement BB


abtestguide.com/bayesian/

Roy Schieving

Annemarie Klaassen


Conclusion Bayesian

Easier to understand and

communicate

Better suits the business

Don’t throw away good ideas (indicatively significant)

Higher implementation rate and

revenue

€ 0

€ 1.000.000

€ 2.000.000

€ 3.000.000

€ 4.000.000

€ 5.000.000

€ 6.000.000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Expected impact on revenue in 1 year

€ Frequentist € Bayesian

3.

Change of test method


50%

50%

A/B-test


Traffic allocation over time

A

B

Tra

ffic

Time

Profit or loss?


What’s the alternative?

A/B-test Bandits

If you have multiple options:

which bandit

will pay-out the

most?


Explore Exploit

?

Explore / Exploit dilemma


Bandit

100%


Bandit

80%

20%

10%

90%


Traffic allocation over time

A

B

Tra

ffic

Time

More Profit!


Limited time frame Automation

When use Bandits?

Earn over learn

How?


Example 1


Example 1

Older target group

Comparing price on

other sites

Behavior of others


Example 1

Control No message

Variation B Copy default

Variaton C

Variation D

11,1%

10,6%

10,4%

10,5%


Example 1 – Effect on revenue

A

B

Tra

ffic

Time

+ € 6.637


Example 2

Control New control

Variation B Copy control

Variation C

Variation D

9,2%

8,9%

7,8%

8,4%


Example 2 – Effect on revenue

A

B

Tra

ffic

Time

+ € 19.862


Conclusion Bandits

Less regret during testing

No implementation costs

Optimizing per user segment

1. Free test software

2. Bayesian statistics

3. Bandit testing

revenue

Maximize


“We’re not in the business of

science, we’re in the business

of making money”

TÄNAN VÄGA!

Bayesian calculator: abtestguide.com/bayesian/

Slide deck: ondi.me/elitecamp2016

@AM_Klaassen

[email protected]

nl.linkedin.com/in/amklaassen

@tomvdberg1

[email protected]

nl.linkedin.com/in/vandenbergtom

Marketing

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths