Information Technology Program

Aalto University, 2015

Dr. Joni Salminen

joolsa@utu.fi, tel. +358 44 06 36 468

DIGITAL ANALYTICS

1

Approaching R

• Download R: (google ’download r’)

• Download R-Studio: (google ’download r studio’)

• Install both.

1

Just one question from my friend…

2

What’s the most

important

application of

analytics?

OPTIMIZATION

3

You could have

also answered

”decision-making”,

but this is how

Joni wants it today.

What is optimization?

• A constant process of improving an object or set of

object in order to better the values of key metrics tied

to business performance.

• i.e., making changes to make more money.

4

It’s always about

the green$, baby.

The many faces of optimization

• Google → search-engine optimization

• Facebook → EdgeRank optimization (newsfeed

optimization)

• Website → landing page optimization, conversion

optimization

• etc.

• In each, content is a common denominator.

Systematic testing for finding compatibility between

content and audience.

• (Content = product, message, blog article, social

media post)

5

Whenever there is something to measure, it

can be optimized.

• rank on a website

• popularity of a social media post

• people’s purchase behavior

6

If you can’t

measure it…

In many situations, you seemingly optimize

for an algorithm. However, almost always

the algorithm follows human decision-

making.

• e.g., most popular search results rank higher in

Google, most popular posts in Facebook

• therefore, eventually you always want to optimize for

user experience (no tricks, remember UFO = user-

focused optimization)

7

UFO… That’s

clever!

Conversion optimization

• Measures taken to improve the likelihood of users

taking a desired action after the click.

• In practice, we modify websites: test different value

propositions, visual elements, placements, layouts,

features, or offerings.

• Note:

– Users are active agents in making concious decisions

when browsing a website.

– Conversion optimization is seen to take place mostly

after the click (however, when does this not apply?)

8

Conversion differs from channel to channel

Display Search engine Facebook

Visitors 1000 1000 1000

Cost (€) 1000 1000 1000

CVR (%) 2 5 2

9

Yo! Check this out:

• let’s buy 3,000 visitors for an ecommerce site from three

sources, 1,000 from each

• CPC = 1 €

• CVR varies

• calculate CPA – which channel has the lowest?

Conversion differs from channel to channel

Display Search engine Facebook

Visitors 1000 1000 1000

Cost (€) 1000 1000 1000

CVR (%) 2 5 2

CPA (€) 50 20 50

10

Yo! Check this out:

• let’s buy 3,000 visitors for an ecommerce site from three

sources, 1,000 from each

• CPC = 1 €

• CVR varies

• calculate CPA – which channel has the lowest?

Why do channels convert differently?

A/B testing

11

You think landing page A is better, Joni

says B is better. What do we do?

12

control (A) test (B) +3,6 %

Let’s test and see who was

right.

The structure of the test:

Only change call to action,

what do we learn?

A/B testing in one picture (Lillevälja, 2013)

13

One test version is made, and it is

run in parallel with the original

version (control). The website traffic

will be divided randomly between

the variations.

Some considerations in A/B testing

• keep clear separation between manipulated and non-

manipulated variables: the more variables

manipulated, the more difficult it is to show a cause-

effect relationship (preferably, test one variable (or

design) at a time)

• make sure you have enough data for statistical

significance

• have a hypothesis & theory (what do you think will be

the result? why?)

14

Hypothesis formulation (Lillevälja, 2013)

15

“Problem: Less than one percent of

visitors sign up for our newsletter.

Hypothesis: “Visitors don’t see the value

in signing up for our newsletter. Adding

three bullet points about the benefits will

increase signup rates.”

In this case, we would try placing a good

summary of benefits the newsletter

member would get from joining the

newsletter. Even if the original version

works better in your A/B test, you

learned something about your visitors.

You clearly defined why you did the test

and can draw conclusions based on the

outcome.”

A very common error in optimization

• Toni: ”Oh, look Oliver, the ad version 1 has 50 clicks

and the version 2 only 20 clicks! Let’s stop version 2,

or what do you reckon?”

• Oliver: ”You’re right buddy, let’s stop that sucker

from wasting our impression. Joni would be so proud

of us…”

• Toni: ”Hooray!”

16

The reality…

Version 1 Version 2

Clicks 50 20

Impressions 300 100

17

What are the guys missing?

The reality…

Version 1 Version 2

Clicks 50 20

Impressions 300 100

CTR 17% 20%

18

What are the guys still missing?

A/B TESTING: ONLINE ADVERTISING

• Open the Excel file, and calculate if there is statistical

significance.

19

The reality…

Version 1 Version 2

Clicks 50 20

Impressions 300 100

CTR 17% 20%

Significant NO

20

The method applies to ANY

performance-based marketing:

• Google AdWords

• Facebook Ads

• display advertising

• etc.

A/B testing and confidence interval

(Nanigans, 2014)

21

few clicks, large

chance of error

WEBSITE CONVERSION OPTIMIZATION:

OPTIMIZELY [JONI SHOWS]

22

Stockmann.com is not performing.

Let’s go to:

https://www.optimizely.com/ab-

testing/

Let’s do these changes:

1. remove second menu

2. move ”back to school” up

3. change logo to black and white

How it works in the background

1. JavaScript replaces content dynamically

2. matches the variations with selected audience

3. checks if they perform the desired action

4. calculates statistical significance and reports the

winning design

23

Got it!

Where to apply A/B testing do-it-yourself?

• long sales pages

• home page (hide/remove elements, rearrange,

change fonts, pictures, CTAs…)

• as you can see, many are minor changes. Yet, e.g.

lack of wordings and guiding can have a

proportionally big impact.

24

For anything more complex than that, you

need IT people, a management process,

and a healthy culture of experimentation.

Which is why 99% of companies don’t do

split testing.

25

It’s just too much

effort – data is for

nerds, while golf is

for managers.

Sequential vs. split test

• Sequential: first test A, then B (easier to execute)

• Split: simultaneously A and B (usually harder to

execute)

• In both, traffic is split randomly between A and B.

• Why is split usually considered better?

26

But, sequential testing can be okay as well,

if it’s not mission-critical stuff

• pick a point of change

• implement change (CMS)

• keep change for a period (e.g., 2 weeks)

• measure change before-after (equal periods)

• if improvement, leave it

• if worse, go back

27

Mission-critical

means somebody’s

life depends on it.

Some managerial considerations.

28

Optimization strategy alternatives

1. Large disruption

2. Chained micro-gains

…there’s also a mid-way of first getting large disruption

(close to global maximum) and then decreasing the

magnitude of tests (fine-tuning to global maximum)

29

Optimization strategy

1. Capture low-hanging fruits

2. Go detailed

”as campaigns mature and settle into a steady state, the

amount of “low-hanging fruit” gradually decreases. This

necessitates a shift of focus from speed to accuracy,

which means that it becomes increasingly important to

make solid, data-driven decisions.” (Yamaguchi, 2013)

30

Consider two things!

a. SIGNIFICANCE of change

b. MAGNITUDE of change

• not significant = not reliable

• no big difference = not that important

31

Like any other investment decision, the

potential gains to run a test need to exceed

its costs (time, money, effort)

If small expected gain and cumbersome

implementation, don’t do.

If vice versa, do. Something in the middle,

consider case by case.

32

Optimization process

1. Get a corporate buy-in: secure resources, access,

set the right expectations

2. Identify low hanging fruits. (How? By spotting

anomalies, such as bottle necks.)

3. Choose a very simple KPI you want to optimize for

4. Choose other metrics that support that KPI

5. Then operationalize, i.e. make a plan on how to

improve each metric

6. Don’t forget – it’s okay to make errors; you’re

optimizing many many times, so individual fuck-ups

regress to mean in the long run.

33

Optimization via funnel stage analysis

a. If top of the funnel is the bottleck for conversion,

then optimize for awareness metrics (e.g., reach,

frequency, CPM)

b. If bottom of the funnel is the bottleneck for

conversion, then optimize for direct sales metrics

(conversion-%, avg. basket)

• Ideally, you want to take both into account, but practice has

shown it’s best to focus on one thing at a time. Either first build

good environment for conversion and then start to drive traffic, or

then start to drive traffic and iterate quickly as you get live data.

• (remember, in a given point in time, only select people need your

product.)

34

How to identify points of improvement?

• look at industry best practices: what others have

tested

• identify loopholes and bottle necks with your

analytics:

– pages getting the most traffic

– landing pages with most value per visit

– biggest exit pages

– pages with biggest bounce

35

What can you test?

• website layout

• value propositions (e.g. price vs. selection)

• copy text (message formulation, use of punctuation,

CTAs)

• political slogans

• target audiences

• visuals (images: placement, size, content)

• forms (length, number of fields)

• product contents & pricing

36

Oh boy, sky is the

limit.

As you can see, conversion optimization is

not (only) some geeky stuff like changing

button colors. It is driven from the premise

that ”people are not stupid”, and therefore

involves all the magic tricks marketers have

on their sleeves (e.g., pricing psychology,

bundling, scarcity, social effects).

37

Every day, I admire

marketers more

and more.

PIE: potential, importance, ease

(Goward, 2013)

38

Brainstorm. Give value for every item on PIE

dimensions. Calculate the average of each

item. Prioritize based on averages.

Some best practices

39

The anatomy of a landing page

40

social

media

indicators

testimonials /

customer

feedback

certificates /

referrals

call-to-action

pictures

copy text

Adhering to conventions

41

convention = element and

position to which a customer

is accustomed (i.e., a ruling

practice)

shopping basket top-right

logo top-left

categories left side

products in the middle

campaign banner above

products

search in header

contact info & help in

footer

Do not deviate

from conversions

without a good

reason. (Confuses

users.)

Full of social proof (Amazon, 2012)

42

reviews

indicate

product

quality (”fake

it till you

make it”)

purchase

frequency tells of

the choices by

other customers

targeted recommendation

is based on other users’

behavior

Remember, people are social

animals (Aristotle, ~300 BC)

Trust seals & brand solicits

(reference proof)

43

reference

customers,

brand spill over

effects

”If big boys trust it, I guess so can I”

Authentic pictures

“Marketers often use stock images that imply

nothing about the value of the offer, settling for

‘pretty’ images that make no clear connection to

the offer’s core value. Remember, images that

say nothing are worth nothing. The force of an

image increases with its authenticity. Images can

bring a realism that reduces the ‘virtual

distance’ between an offer’s value and the

recipient’s perception of that value. Therefore,

marketers must attempt to find images that help

the visitor see and experience the core value of

the product.” (Marketing Experiments 2010)

44

Always show

real people, not

some stock

photos!

”Marketing bullshit” – avoid this!

45

• Be specific. ”industry

leader”, ”best data” – BS!

• murky qualitative claims

• information value zero

• Watch the

tone. ”Your

hunt is over!”

Do I know you?

• The customer should

be given choice in

being contacted.

Concretizing value propositions

”To offer an example, commonly used PPC terms such

as ‘biggest’ mean nothing out of context. Instead of

wasting ad space with unsubstantiated generalities,

choose to tell the user, ‘106,000+ new users in 2010.’

Don’t tell the user that something offers ‘fastest

downloads’ when it is more effective to say, ‘Download

time is X seconds.’ Such superlatives offer the user no

information that would encourage a clickthrough.”

(Marketing Experiments 2011)

46

It’s like all research. You have some data,

and then you generalize.

But yeah… All this is just ”best practice”,

something that other people tell it’s the best. It

may appeal to your marketer’s intuition, but

there are books written about how intuition can

fail. (So, let the data tell what’s right in your

case.)

47

An example of marketer’s intuition

48

1,15%

49

0,92%

Survivor bias to do and case studies of A/B

testing

• “An A/B test that worked for another company

isn’t always repeatable. Don’t blindly copy tests

from success stories expecting similar results.

Instead try to understand why it worked for them,

and what lessons you can draw from it.” (Kogan,

2014)

• usually, only the most successful tests are published

(cf. lottery)

• these are a small minority

• therefore, the majority will not get reported at all!

• the result: over-optimistism over A/B testing

(survivor bias).

50

What’s the purpose of testing?

To understand why.

“Marketers should not assume that a popular page

design will be effective for every situation. The problem

with ‘best practices’ or ‘best designs’ is that they rarely

work across the board. It is more important to move

beyond understanding the ‘what’ of page layout, and use

testing to attain the deeper understanding of ‘why’.”

(Marketing Experiments 2010)

51

Discovering, through a reliable test, which

alternative yields the best response,

enables you to generalize that alternative

beyond the test. For example, test five

different copy texts which are ”the nominees”

for new slogan in Facebook, and based on the

results you can say which one resonates best

with the audience (before launching it in mass

marketing).

52

Testing with ”focus group” mentality (small

budget, limited reach), and then expanding

to the mass marketing. (Nothing new here, it

was already done by Ogilvy and Hopkins

starting from 1920s, but now it’s hell of a lot

easier, cheaper and faster. And yet most

copywriters don’t do it!)

53

Testing does not automatically lead into

improvements…

54

Why?

This is why:

• most tests fail to show improvement (survivor

bias)

• small changes → small improvements (garbage

in, garbage out)

• postponing or cancelling the proposed changes

(HiPPO)

55

There are

always

complications.

Smaller differences take longer to clear.

Instead, you would like to make many tests

in a given time period, not only one.

CVR (%) Change (%) Required

sample size

per variant

Days to

clear

Test 1 3 5 72,300 72

Test 2 3 10 18,500 18

Test 3 3 30 2,250 2

56

(Johns, 2015)

”Goodbye Google” (aka 41 shades of blue)

• “Yes, it’s true that a team at Google couldn’t decide

between two blues, so they’re testing 41 shades

between each blue to see which one performs better.

I had a recent debate over whether a border should

be 3, 4 or 5 pixels wide, and was asked to prove my

case. I can’t operate in an environment like that. I’ve

grown tired of debating such minuscule design

decisions. There are more exciting design problems

in this world to tackle.” (Bowman, 2009)

• Is this a case of local maximum problem? What do

you think?

57

How to survive the local maximum

problem?

58

Problem with A/B testing: HiPPO

• ”Listen, I know what works. Let’s do like this and

that’ll be the end of it.”

• ”…okay.”

59

I wish I didn’t

always cave in!

Some theoretical considerations

60

THEORY?? Dude,

who cares!

(Just kidding.)

Compounding benefits of optimization

(Marketing Experiments, 2005)

• Base level in the example:

– monthly sales = 100,000 $

– costs = 85,000 $, monthly profit 15,000 $

• The company implements nine improvements in nine

months (one per month).

61

Compounding benefits of optimization

(Marketing Experiments, 2005)

62

The benefit table

Improvements (1 per month) Gain Profit per

month

Change in

profit

(0. Base line) N/A $15,000 0%

1. Improving PPC ad copy 5% (CTR) $19,500 30%

2. Decreasing CPC 5% (CTR) $19,999 3%

3. Optimizing landing page 5% (CVR) $30,249 51%

4. Optimizing order form 5% (CVR) $35,761 18%

5. Streamlining website copy 5% (CVR) $41,549 16%

6. Adding upselling and cross-selling

options 5% (Sales) $47,627 15%

7. Lowering price 5% (Sales) $49,988 5%

8. Changing shopping cart flow 5% (Sales) $56,487 13%

9. Adding trust indicators 5% (Sales) $63,312 12%

Compounding benefits of optimization:

example (Marketing Experiments, 2005)

• Results:

– Extra profit = 48,312 $

– Compounding benefit = 322 %

– (Simple benefit = 163 %)

• In other words, there is an ”interest to interest” effect

in optimization.

63

Implications for a marketer

• Make systematically small improvements

• Focus on both before and after the click

64

I know it’s boring

at first, but seeing

the fruits of your

labor motivates!

Scaling of optimization: example

• Three companies

A. 1000 visitors per day

B. 10,000 visitors per day

C. 100,000 visitors per day

• Other parameters

– Conversion prior to optimization = 1 %

– Conversion after the optimization = 2 %

– Fixed monthly cost of optimization = 2000 €

– Avg. basket = 50 €

• Let’s optimize. What is the ROI for each company?

65

Scaling of optimization efforts: results

66

0

20000

40000

60000

80000

100000

120000

A B C

myynnin muutos

myynti

kiinteä kust per kk

ROI:

A = -50 %

B = +400 %

C = +4900 %

Multiplication argument (Nielsen, 2008)

a) “In a multiplication, if you want to increase the outcome by

a certain percentage, you can increase any of the factors

by that percentage. It doesn’t matter which factor is

increased — the result will be the same.

b) Thus, to double a site’s business, you can double the

number of unique visitors. However, this would be very

expensive, requiring that you more than double the

advertising budget (assuming you’re already advertising

under the most-promising keywords, and thus need to buy

traffic from less promising or more expensive sources).

c) Alternatively, you can double the conversion rate and

achieve the same business improvement. (…) In most

cases, it’s far cheaper to use 15% of your development

budget than to more than double your advertising budget.”

67

Example: halving the advertising costs

• When you want to double sales, either double the

advertising budget (if possible) or the conversion rate.

• As the number of paid visitors increases, the more

feasible conversion optimization becomes as an

investment

68

Case A: Low

conversion

Case B: High

conversion

Ad spend 100,000 50,000

Visitors (CPC = 0.25) 400,000 200,000

Conversion 1 % 2 %

Sales quantity 4,000 4,000

Return to eCommerce metrics

(Fellman, 2015)

• Visitors

• Conversion rate

• Average basket

• Margin

• Example (monthly sales):

• 100,000 x 0.02 x 100 € x 0.40 = 80,000 €

69

Visitors Conversion rate Average

basket

Margin

Dissect each, and

consider how you can

optimize.

(Remember: this was the

breakdown of metrics that

were deemed important

for ecommerce.)

Metrics for newsletters: tying the chosen

metrics to the optimization process

70

Subscribers x delivery rate

x open rate

x click rate

x conversion rate

x margin

= profit

Ways to improve

Lead generation tactics

Bundling, high-markup

items

Conversion optimization

Quality (mitigation of

spam complaints)

Headline optimization

Why is conversion always low? (Regardless

of optimization)

• P(cv) = P(sees) x P(understands) x P(has need) x

P(has money)

• Let’s advertise to the whole of Finland!

• Assume an optimistic 10% at each step

• 5,000,000 x 0.1 x 0.1 x 0.1 x 0.1 = 500

• 500 customers, yey!

• …but wait, the campaign costed us 50,000 €. Let’s

see: 50,000/500 = 100 €. (Will I keep my job or not?)

71

Another way to look at it…

• At time point t, a sub-set m of total market M has the

need for product x. In a world like this, most products

are not finding a match in the market at the time of

advertising. However, the long-term effect should be

positive to justify the cost.

• (Which takes us back to the question of choosing the

lookback window.)

72

So, what can we do?

All else being equal, a marketer can increase

conversion by improving targeting (driving more

qualified traffic, e.g., focusing only on the end of

the purchasing path) or by driving more overall

traffic in the beginning of the purchasing path.

73

But will the quality

of the traffic

remain constant?

Potential solution: micro conversions

(Google, 2015)

74

Conclusions (1/2)

• Conversion optimization is feasible when

C < avB x (S1 – S0), i.e.

• the fixed cost of optimization (C) is smaller than the

average basket (avB) times the change in sales

quantity (S1 – S0), or when additional sales cover the

costs.

• When there’s a small number of visitors, the cost of

optimization, whether in-house or agency, can easily

exceed the benefits. However, what would be the

exception?

75

”every product has an amazing dropoff of

usage from when people first encounter it”

(Chen, 2015)

76

Conclusions (2/2)

a. Generally, the largest gains from conversion

optimization emerge from removing bottle necks

(which can be found by analysing users’ behavior)

b. The more traffic and sales, the more lucrative it is for

a company to invest in conversion optimization

(because of the scaling effect)

c. When chained, a large number of small

improvements may generate a positive ROI

(because of compounding benefits).

77

The end.

78