A/B Testing with Air Patriots - GDC 2014

1

@PaulCutsinger

Russ Carroll

A|B Testing

2

3

4

Surface the Value

Image from Diner Dash by PlayFirst

Source: Amazon Appstore, January 2013

+75% ARPPU

5

Show Users How to “Consume”

Image from Cut the Rope by ZeptoLab

Source: Amazon Appstore, January 2013

+65%

repeat orders

6

Experiment and react quickly

7 day retention

Time

Active Players

7

A|B Testing

8

This is Air Patriots

Case Study: Store Optimization

10

• Goal – Easier to access store content

• Platform – Android & iOS

• Test – Make it possible to unlock content on Main Menu

STORE OPTIMIZATION

11

LOCKED ON MAIN MENU

UNLOCKABLE >

<

12

STORE OPTIMIZATION

Test Case Conversion to

Customer 7-Day Retention

Locked on Main Menu

(Control) – –

Unlockable on Main

Menu 20.93% 4.17%

13

• Goal – Drive players to our other games

• Platform – Android

• Test – Which image is clicked on more

CROSS PROMOTION

CROSS PROMOTION

15

CROSS PROMOTION

16

Air Patriots: Cross Promotion

B

A 1.77%

0.93%

Clic

k T

hro

ug

h

19.94% 1. 3 day Retention -1.07% +0.83%

8.06% 2. 7 day Retention +0.74% -10.11%

2.05% 3. Units Purchased +0.34% -12.07%

0% 4. Ads Clicked +1.89% +3.07%

43,120

The Producer’s Perspective Ads on menu or in game

19.94% 1. 3 day Retention -1.07% +0.83%

8.06% 2. 7 day Retention +0.74% -10.11%

2.05% 3. Units Purchased +0.34% -12.07%

0% 4. Ads Clicked +1.09% +4.07%

43,120

Ads on menu or in game

In Game On Menu No Ads

The Producer’s Perspective

19.94% 1. 3 day Retention -1.07% +0.83%

8.06% 2. 7 day Retention +0.74% -10.11%

2.05% 3. Units Purchased +0.34% -12.07%

0% 4. Ads Clicked +1.09% +4.07%

43,120

Ads on menu or in game

In Game On Menu No Ads Goals

The Producer’s Perspective

19.94% 1. 3 day Retention -1.07% +0.83%

8.06% 2. 7 day Retention +0.74% -10.11%

2.05% 3. Units Purchased +0.34% -12.07%

0% 4. Ads Clicked +1.09% +4.07%

43,120

Ads on menu or in game

In Game On Menu No Ads Goals

The Producer’s Perspective

19.94% 1. 3 day Retention -1.07% +0.83%

8.06% 2. 7 day Retention +0.74% -10.11%

2.05% 3. Units Purchased +0.34% -12.07%

0% 4. Ads Clicked +1.09% +4.07%

43,120

Ads on menu or in game

In Game On Menu No Ads Goals

The Producer’s Perspective

22

Retrieve a variation

Objective C

23

Record a view and conversion Objective C

bit.ly/ABTestXPlat

bit.ly/ABTestiOS

bit.ly/ABTestJava

More details at

Case Study: Junk Race by GameNug

Junk Race Overview

26

Junk Race Overview

27

Tips: Producers in control: “With two-way services you can request a value from the

server that you control. Two-way services let you make changes to your live game

instantly.

Free: Amazon is the only free two-way service that we have found.”

Multi-variable: “We don't test variables, we test user experiences.”

Watch sample size: “If a couple users can change your conclusion, you need a

larger sample”

Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of

skipping the week long review period ”

-John Goforth

28

Case Study: Difficulty Level

29 Amazon Services Confidential

GAME DIFFICULTY TEST

7 day retention

Time

Active Players

30 Amazon Services Confidential

GAME DIFFICULTY TEST

• Goal – A Dramatic Increase!

• Platform – Android & iOS

• Test – A/B/n Test – 5 Difficulty levels!

OR?

31 Amazon Services Confidential

GAME DIFFICULTY TEST

32 Amazon Services Confidential

Test 1:

A

Control

B

2% easier

C

4% easier

D

6% easier

E

8% easier

7-Day Retention – tbd tbd tbd tbd

Conversion to customer – -1.71% 16.11% 19.23% 4.84%

GAME DIFFICULTY TEST

33 Amazon Services Confidential

GAME DIFFICULTY TEST

• How much would you pay to increase retention by 20%?

• How much time would you invest?

Our Take-Aways!

• Never launch a game in future w/o AB difficulty test!

• Low-hanging fruit!

• Opportunities abound!

34 Amazon Services Confidential

A|B Testing

• Monetization – minimize luck and maximize

skill

• Delight your customers

• test customer experiences

• AB testing tools are easy to use

• Cross Platform

• Deploy changes instantly

• Free

• Low-hanging fruit is waiting to be harvested

Paul Cutsinger Russell Carroll Find me at: • @PaulCutsinger • @AmazonAppDev • developer.amazon.com/blog • bit.ly/ABTestXPlat