Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve

Top 10 Tips for Mobile App Optimization using A/B Testing

Steve CollinsCTO / Cofounder Swrve


“One accurate measurement is worth 1,000 expert opinions.”

Admiral Grace Murray Hopper


A/B Testing

Split Test Measure Choose Deploy


Start simple

0101


#1: Start Simple

■Identify “low hanging fruit”■Choose minimal viable

implementation■Get a win as early as possible

Time to 1st interstitial? Tutorial guide

character?

vs

Force Login?


#1: Start Simple


#1: Start Simple

{variant = A}

UID = ABCD


#1: Start Simple

Event(Tut_A)

A: 1,356 completedB: 1,793 completed


Build a data driven culture

0202


■Data driven != lots of metrics / reports

#2: Data Driven Culture



■Beware the HiPPOs



■Link experiments to bottom line

Time to 1st interstitial?

CTR In-app Revenue


Remove latency

0303


#3: Remove Latency

■Close the loop

Measure

Take actionIterate, fail-fast

TEST


#3: Remove Latency

Measure / Adapt

Measure / Adapt

DevelopDevelopARTENG

PM

00111011100011100001000001010101

Test FrameworkTest Framework


Kill switch

0404


#4: Kill Switch

■3 Schools of thought:1. Fail early / fast


#4: Kill Switch

■3 Schools of thought:1. Fail early / fast2. Thorough QA

DraftDraft

StageStage

PublishPublish

QA Users

All Users



#4: Kill Switch■3 Schools of thought:

1. Fail early / fast2. Thorough QA3. Ramp-up


Minimize variation

0505


#5: Minimize Variation

A B

+102%

www.abtests.com and diythemes.com


Lookout for longitudinal impacts

0606


#6: Longitudinal Impacts

■Clear winning variant:

Conversions up



■But long-term KPIs are negative■Segment your metrics by variant

Test Start Time

KPI



■Beware temporal effects

24 hrs


Treat new and repeat users differently

0707


#7: Treat new and repeat users differently

A

B


#7: Treat new and repeat users differently

■“Primacy Effect”– Test new users only– Test users on a new app-version only– Ensure consistent user experience


Test offers not prices

0808


15 15

#8: Test offers not prices

■Sometimes better to avoid direct testing

20 20


2525

#8: Test offers not prices

■Sometimes better to avoid direct testing

25 25

-20%-40%


Run the math

0909


#9: Run the math

■Null hypothesis testing


#9: Run the math

■Null hypothesis testing


#9: Run the math

■How long to run test?


Save your results

1010


#10: Save your results

■ Name■ When it ran■ Owner■ Hypothesis■ Variants■ Result■ KPIs


Thanks

"Planet Cute" art by Daniel Cook (Lostgarden.com)48 px Web Icons by emey87.deviantart.com

■Some handy links:– http://www.swrve.com/blog/ – http://www.kaushik.net – http://20bits.com/article/statistical-analysis-and-ab-testing

– http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/

Documents

Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve