© 2011 Webtrends, All Rights Reserved.

MVT Across Multiple Brands: One approach to successful testing

Martin SheerinHead of Online Product Development, Phones4UMichael Schirrmacher UK Sales Director, Webtrends

Webtrends |

© 2011 Webtrends, All Rights Reserved. |2

Why should we Optimize?

#1. Improve Customer Experience

#2. Improve Conversion & Revenue

#3. Deepen Visitor Engagement

© 2011 Webtrends, All Rights Reserved. |3

© 2011 Webtrends, All Rights Reserved. |4

Where do I start?

Analytics

Identify and quantify opportunities through

insight

Optimization

Test, segment and target to improve performance

© 2011 Webtrends, All Rights Reserved. |5

What Optimisation Involves?

© 2011 Webtrends, All Rights Reserved. |6

Full Factorial….Fractional factorial….?

“ok this looks complex”

© 2011 Webtrends, All Rights Reserved. |7

So, for now…a managed service is still more popular

© 2011 Webtrends, All Rights Reserved. |8

Optimization Maturity Model

Conversions

• Web Analytics• Measurement • No Testing• Ad Network

Targeting

• A/B/n Testing• Basic MVT -1-3

areas• 1 Element Targeting• 1-Project at a Time

• More Advanced MVT• In-session

Segmentation• Multi Element

Targeting • Frequent

• Personalization• Integrated

Optimization• External Segment

Use• Social & Multi-

Channel• On-going

Optimization Maturity Phases

“Crawl ” Phase “Walk” Phase “Run” PhaseOperational Reporting

© 2011 Webtrends, All Rights Reserved. |9

Multivariate Test

£

Control

© 2011 Webtrends, All Rights Reserved. |10

Multivariate Test

Challenger

£

Control

© 2011 Webtrends, All Rights Reserved. |11

Multivariate Test

+£3.2m Lift in special offers sales in UK

Martin SheerinHead of Online Product Development Phones 4U

@martinsheerin1

linkedin.com/in/martinsheerin1

Phones 4u

• Multi-channel retailer• Over 600 stores• Contact centre for sales and service• Website: www.phones4u.co.uk• Mobile site: m.phones4u.co.uk

• Online only brand:– www.dialaphone.co.uk– m.dialaphone.co.uk

Why should you MVT?

Increase Conversion

Why not MVT yourself instead of using a partner?• Time• Skills• Ensuring that a test is statistically significant• Ensures the result is true and not biased to an internal individuals

viewpoint

• You can do this yourself:

What to look for in an MVT provider?

Business understandi

ng

eCommerce insight

Business alignment

Ease of reporting

Our Approach

• Test simultaneously on both sites

• Use results of test on one site and apply to the other (if its possible)

• Be happy to repeat tests annually (or more regularly if you deem necessary)

Our Test Successes

Free Gifts

Op

tim

al

Co

ntr

ol P

ag

e

6.27%lift

Customer Behaviour

Research

Choose Handset

Allowance (Lifestyle)

Select Package

Purchase

Phones 4u

Dialaphone

Phones 4u Review Tab

Control

Winner

+2.99%

Search Results

+16.56%

Simple and Usable

Source: Simple and Usable Web, Mobile, and Interaction Design (Voices That Matter) - Giles Colborne (CX Partners) (2010)

Search Results

Op

tim

al

Co

ntr

ol P

ag

e

Experiment Page Lift

Control ---

Optimal +16.56%

+16.56%

Upgrades - Variants

Op

tim

al

Co

ntr

ol P

ag

e

Experiment Page Lift

Control ---

Optimal +81.64%

+81.64%Lift at next step

Taking this finding we applied it to Phones 4u

Optimal

Lev

el 2

- g

rey

bac

kgro

un

d

and

up

gra

de

pro

mis

e

Lev

el 3

–

wh

ite

bac

kgro

un

d

and

up

gra

de

pro

mis

e

Lev

el 4

–

gre

y b

ackg

rou

nd

Lev

el 5

–

wh

ite

bac

kgro

un

d

con

tro

l

Phones 4u Winning Variant

Op

tim

al

Co

ntr

ol P

ag

e

Experiment Page Lift

Control ---

Optimal +70.64%

+70.64 %Lift at next step

Not satisfied we went with a hunch…

Op

tim

al

Co

ntr

ol P

ag

e

Experiment Page Lift

Control ---

Optimal +0.89%

+0.89%

Moving on with the upgrades test

Multi-device testingC

on

tro

l Pa

ge

Applying this idea to Phones 4u

All Traffic (non-iPad)C

on

tro

l Pa

ge

Op

tim

al

Experiment Page Lift

Control ---

Optimal +4.11%

Optimal

iPad VersionC

on

tro

l Pa

ge

Optimal

Experiment Page Lift

Control ---

Optimal +4.27%

Op

tim

al

Summary

Define what success

looks like

Apply common

sense

Only test what you are willing

to put live

Don’t be satisfied that the result of one test cannot

be improved upon

Re-test if you think its

necessary

Sometimes the simplest things give the biggest

results

Geo targeting

Persona driven

segmentation

Touch commerce

(Multi Device)

Martin SheerinHead of Online Product Development Phones 4U

@martinsheerin1

linkedin.com/in/martinsheerin1