I’m Jim Williams,

Head of Customer

Analytics

Split Testing to Optimize Digital Sales

www.weeworld.comwww.ju2.comwww.twitter.com/ju2ltd

Weeworld

• Weeworld is a social networking site/

virtual world

• WeeWorld is the home of the

WeeMee

• The world's favourite avatar

• 33M weemees and growing

•Users Express themselves visually by

dressing up the weemee with latest

fashions and accessories

Hi… I’m a WeeMee

Building out community

I’m so popular, I have 10,000 friends!

Rooms: extension of your WeeMee

A new room is created about like every 10 seconds.

******

World: Synchronous interaction

• Doesn’t have to be 3D, that’s not important as far as building a Virtual World is concerned

Hey you got a boyfriend?

Maybe yes, maybe no.

Getting Gold Points to Spend on Site

•Subscriptions•PayPal•Prepaid Cards•PSMS Mobile Payments•Surveys•Offers•Pay by Cash•Launch Credit Cards

•Original Get Points Page too Complicated•Get Points Page Redesign

New Tabbed Design

Credit Card Tab Surveys Tab

Which Page Design gave the highest Revenue per User?

Which Design is Best?

Wanted to answer 2 questions:

• Is the tabbed design better than the original design

• Which tab should be default

Average Revenue per User

Tab or not to Tab that is the question?

Objective: To Identify whether new tabbed design more effective than original.

Challenges:

•All transactions completed on third party sites•Could not guarantee to capture conversion•Unable to use Google Website Optimizer•Multiple micro transactions•User experience must be consistent

Whys life so complicated?

Original Get Points Page

Allocate page versions according to userID

MOD5- 0 Original 20%- 1 or 2 Credit Card 40%- 3 or 4 Surveys 40%

Track average revenue per user ARPU

Statistics 95% confidence IntervalsNo significant differences between groups prior to tests

New Tabbed Design

Credit Card Tab Surveys Tab

Which Page Design gave the highest Revenue per User?

Improvement in Revenue per Login

Version Response(improvement in

revenue per login)

p<0.05

Original - -

Credit Card 7.7% yes

Surveys 7.1% yes

Tabbed design 7% more effective than original one page design no difference between tabs

Difficulties – Thing about your control

• Don’t be impatient• Change only one thing at a

time• Original Page had no credit

card option• Was result due to new design

or just credit card?• Another follow up experiment

showed that the design was better than Original -Credit Card

Change only one thing?

Difficulties – Don’t breakout the champagne too early!

• Cyclical Effects - don’t breakout the champagne too early.

• Improvement in ARPU much higher early in experiment

• Maintaining Interest 28 days to see result.

Difficulties – Avoid Bias

• Site constantly changing• Marketing initiatives• Store Promotions• Avoiding bias• Live in real world

That’s biased it?

Difficulties -

• PayPal Revenue dropped• Cannibalization by Credit Card• Average Transaction Value fell

28%• Changed price points• Reversing option order removed

fall in average transaction value

Lessons Learned

• Using userID for split testing valid – continue to use• Tabbed Design increased revenue per user compared to

original design even with credit card. • Give careful consideration to controls• Change only one thing at a time.• Try to avoid bias with other changes on the site• Average transaction values very sensitive to price points• Maintain stakeholder interest throughout experiment

Questions?