View
355
Download
0
Category
Preview:
Citation preview
Information Technology Program
Aalto University, 2015
Dr. Joni Salminen
joolsa@utu.fi, tel. +358 44 06 36 468
DIGITAL ANALYTICS
1
Approaching R
• Download R: (google ’download r’)
• Download R-Studio: (google ’download r studio’)
• Install both.
1
Just one question from my friend…
2
What’s the most
important
application of
analytics?
OPTIMIZATION
3
You could have
also answered
”decision-making”,
but this is how
Joni wants it today.
What is optimization?
• A constant process of improving an object or set of
object in order to better the values of key metrics tied
to business performance.
• i.e., making changes to make more money.
4
It’s always about
the green$, baby.
The many faces of optimization
• Google → search-engine optimization
• Facebook → EdgeRank optimization (newsfeed
optimization)
• Website → landing page optimization, conversion
optimization
• etc.
• In each, content is a common denominator.
Systematic testing for finding compatibility between
content and audience.
• (Content = product, message, blog article, social
media post)
5
Whenever there is something to measure, it
can be optimized.
• rank on a website
• popularity of a social media post
• people’s purchase behavior
6
If you can’t
measure it…
In many situations, you seemingly optimize
for an algorithm. However, almost always
the algorithm follows human decision-
making.
• e.g., most popular search results rank higher in
Google, most popular posts in Facebook
• therefore, eventually you always want to optimize for
user experience (no tricks, remember UFO = user-
focused optimization)
7
UFO… That’s
clever!
Conversion optimization
• Measures taken to improve the likelihood of users
taking a desired action after the click.
• In practice, we modify websites: test different value
propositions, visual elements, placements, layouts,
features, or offerings.
• Note:
– Users are active agents in making concious decisions
when browsing a website.
– Conversion optimization is seen to take place mostly
after the click (however, when does this not apply?)
8
Conversion differs from channel to channel
Display Search engine Facebook
Visitors 1000 1000 1000
Cost (€) 1000 1000 1000
CVR (%) 2 5 2
9
Yo! Check this out:
• let’s buy 3,000 visitors for an ecommerce site from three
sources, 1,000 from each
• CPC = 1 €
• CVR varies
• calculate CPA – which channel has the lowest?
Conversion differs from channel to channel
Display Search engine Facebook
Visitors 1000 1000 1000
Cost (€) 1000 1000 1000
CVR (%) 2 5 2
CPA (€) 50 20 50
10
Yo! Check this out:
• let’s buy 3,000 visitors for an ecommerce site from three
sources, 1,000 from each
• CPC = 1 €
• CVR varies
• calculate CPA – which channel has the lowest?
Why do channels convert differently?
A/B testing
11
You think landing page A is better, Joni
says B is better. What do we do?
12
control (A) test (B) +3,6 %
Let’s test and see who was
right.
The structure of the test:
Only change call to action,
what do we learn?
A/B testing in one picture (Lillevälja, 2013)
13
One test version is made, and it is
run in parallel with the original
version (control). The website traffic
will be divided randomly between
the variations.
Some considerations in A/B testing
• keep clear separation between manipulated and non-
manipulated variables: the more variables
manipulated, the more difficult it is to show a cause-
effect relationship (preferably, test one variable (or
design) at a time)
• make sure you have enough data for statistical
significance
• have a hypothesis & theory (what do you think will be
the result? why?)
14
Hypothesis formulation (Lillevälja, 2013)
15
“Problem: Less than one percent of
visitors sign up for our newsletter.
Hypothesis: “Visitors don’t see the value
in signing up for our newsletter. Adding
three bullet points about the benefits will
increase signup rates.”
In this case, we would try placing a good
summary of benefits the newsletter
member would get from joining the
newsletter. Even if the original version
works better in your A/B test, you
learned something about your visitors.
You clearly defined why you did the test
and can draw conclusions based on the
outcome.”
A very common error in optimization
• Toni: ”Oh, look Oliver, the ad version 1 has 50 clicks
and the version 2 only 20 clicks! Let’s stop version 2,
or what do you reckon?”
• Oliver: ”You’re right buddy, let’s stop that sucker
from wasting our impression. Joni would be so proud
of us…”
• Toni: ”Hooray!”
16
The reality…
Version 1 Version 2
Clicks 50 20
Impressions 300 100
17
What are the guys missing?
The reality…
Version 1 Version 2
Clicks 50 20
Impressions 300 100
CTR 17% 20%
18
What are the guys still missing?
A/B TESTING: ONLINE ADVERTISING
• Open the Excel file, and calculate if there is statistical
significance.
19
The reality…
Version 1 Version 2
Clicks 50 20
Impressions 300 100
CTR 17% 20%
Significant NO
20
The method applies to ANY
performance-based marketing:
• Google AdWords
• Facebook Ads
• display advertising
• etc.
A/B testing and confidence interval
(Nanigans, 2014)
21
few clicks, large
chance of error
WEBSITE CONVERSION OPTIMIZATION:
OPTIMIZELY [JONI SHOWS]
22
Stockmann.com is not performing.
Let’s go to:
https://www.optimizely.com/ab-
testing/
Let’s do these changes:
1. remove second menu
2. move ”back to school” up
3. change logo to black and white
How it works in the background
1. JavaScript replaces content dynamically
2. matches the variations with selected audience
3. checks if they perform the desired action
4. calculates statistical significance and reports the
winning design
23
Got it!
Where to apply A/B testing do-it-yourself?
• long sales pages
• home page (hide/remove elements, rearrange,
change fonts, pictures, CTAs…)
• as you can see, many are minor changes. Yet, e.g.
lack of wordings and guiding can have a
proportionally big impact.
24
For anything more complex than that, you
need IT people, a management process,
and a healthy culture of experimentation.
Which is why 99% of companies don’t do
split testing.
25
It’s just too much
effort – data is for
nerds, while golf is
for managers.
Sequential vs. split test
• Sequential: first test A, then B (easier to execute)
• Split: simultaneously A and B (usually harder to
execute)
• In both, traffic is split randomly between A and B.
• Why is split usually considered better?
26
But, sequential testing can be okay as well,
if it’s not mission-critical stuff
• pick a point of change
• implement change (CMS)
• keep change for a period (e.g., 2 weeks)
• measure change before-after (equal periods)
• if improvement, leave it
• if worse, go back
27
Mission-critical
means somebody’s
life depends on it.
Some managerial considerations.
28
Optimization strategy alternatives
1. Large disruption
2. Chained micro-gains
…there’s also a mid-way of first getting large disruption
(close to global maximum) and then decreasing the
magnitude of tests (fine-tuning to global maximum)
29
Optimization strategy
1. Capture low-hanging fruits
2. Go detailed
”as campaigns mature and settle into a steady state, the
amount of “low-hanging fruit” gradually decreases. This
necessitates a shift of focus from speed to accuracy,
which means that it becomes increasingly important to
make solid, data-driven decisions.” (Yamaguchi, 2013)
30
Consider two things!
a. SIGNIFICANCE of change
b. MAGNITUDE of change
• not significant = not reliable
• no big difference = not that important
31
Like any other investment decision, the
potential gains to run a test need to exceed
its costs (time, money, effort)
If small expected gain and cumbersome
implementation, don’t do.
If vice versa, do. Something in the middle,
consider case by case.
32
Optimization process
1. Get a corporate buy-in: secure resources, access,
set the right expectations
2. Identify low hanging fruits. (How? By spotting
anomalies, such as bottle necks.)
3. Choose a very simple KPI you want to optimize for
4. Choose other metrics that support that KPI
5. Then operationalize, i.e. make a plan on how to
improve each metric
6. Don’t forget – it’s okay to make errors; you’re
optimizing many many times, so individual fuck-ups
regress to mean in the long run.
33
Optimization via funnel stage analysis
a. If top of the funnel is the bottleck for conversion,
then optimize for awareness metrics (e.g., reach,
frequency, CPM)
b. If bottom of the funnel is the bottleneck for
conversion, then optimize for direct sales metrics
(conversion-%, avg. basket)
• Ideally, you want to take both into account, but practice has
shown it’s best to focus on one thing at a time. Either first build
good environment for conversion and then start to drive traffic, or
then start to drive traffic and iterate quickly as you get live data.
• (remember, in a given point in time, only select people need your
product.)
34
How to identify points of improvement?
• look at industry best practices: what others have
tested
• identify loopholes and bottle necks with your
analytics:
– pages getting the most traffic
– landing pages with most value per visit
– biggest exit pages
– pages with biggest bounce
35
What can you test?
• website layout
• value propositions (e.g. price vs. selection)
• copy text (message formulation, use of punctuation,
CTAs)
• political slogans
• target audiences
• visuals (images: placement, size, content)
• forms (length, number of fields)
• product contents & pricing
36
Oh boy, sky is the
limit.
As you can see, conversion optimization is
not (only) some geeky stuff like changing
button colors. It is driven from the premise
that ”people are not stupid”, and therefore
involves all the magic tricks marketers have
on their sleeves (e.g., pricing psychology,
bundling, scarcity, social effects).
37
Every day, I admire
marketers more
and more.
PIE: potential, importance, ease
(Goward, 2013)
38
Brainstorm. Give value for every item on PIE
dimensions. Calculate the average of each
item. Prioritize based on averages.
Some best practices
39
The anatomy of a landing page
40
social
media
indicators
testimonials /
customer
feedback
certificates /
referrals
call-to-action
pictures
copy text
Adhering to conventions
41
convention = element and
position to which a customer
is accustomed (i.e., a ruling
practice)
shopping basket top-right
logo top-left
categories left side
products in the middle
campaign banner above
products
search in header
contact info & help in
footer
Do not deviate
from conversions
without a good
reason. (Confuses
users.)
Full of social proof (Amazon, 2012)
42
reviews
indicate
product
quality (”fake
it till you
make it”)
purchase
frequency tells of
the choices by
other customers
targeted recommendation
is based on other users’
behavior
Remember, people are social
animals (Aristotle, ~300 BC)
Trust seals & brand solicits
(reference proof)
43
reference
customers,
brand spill over
effects
”If big boys trust it, I guess so can I”
Authentic pictures
“Marketers often use stock images that imply
nothing about the value of the offer, settling for
‘pretty’ images that make no clear connection to
the offer’s core value. Remember, images that
say nothing are worth nothing. The force of an
image increases with its authenticity. Images can
bring a realism that reduces the ‘virtual
distance’ between an offer’s value and the
recipient’s perception of that value. Therefore,
marketers must attempt to find images that help
the visitor see and experience the core value of
the product.” (Marketing Experiments 2010)
44
Always show
real people, not
some stock
photos!
”Marketing bullshit” – avoid this!
45
• Be specific. ”industry
leader”, ”best data” – BS!
• murky qualitative claims
• information value zero
• Watch the
tone. ”Your
hunt is over!”
Do I know you?
• The customer should
be given choice in
being contacted.
Concretizing value propositions
”To offer an example, commonly used PPC terms such
as ‘biggest’ mean nothing out of context. Instead of
wasting ad space with unsubstantiated generalities,
choose to tell the user, ‘106,000+ new users in 2010.’
Don’t tell the user that something offers ‘fastest
downloads’ when it is more effective to say, ‘Download
time is X seconds.’ Such superlatives offer the user no
information that would encourage a clickthrough.”
(Marketing Experiments 2011)
46
It’s like all research. You have some data,
and then you generalize.
But yeah… All this is just ”best practice”,
something that other people tell it’s the best. It
may appeal to your marketer’s intuition, but
there are books written about how intuition can
fail. (So, let the data tell what’s right in your
case.)
47
An example of marketer’s intuition
48
1,15%
49
0,92%
Survivor bias to do and case studies of A/B
testing
• “An A/B test that worked for another company
isn’t always repeatable. Don’t blindly copy tests
from success stories expecting similar results.
Instead try to understand why it worked for them,
and what lessons you can draw from it.” (Kogan,
2014)
• usually, only the most successful tests are published
(cf. lottery)
• these are a small minority
• therefore, the majority will not get reported at all!
• the result: over-optimistism over A/B testing
(survivor bias).
50
What’s the purpose of testing?
To understand why.
“Marketers should not assume that a popular page
design will be effective for every situation. The problem
with ‘best practices’ or ‘best designs’ is that they rarely
work across the board. It is more important to move
beyond understanding the ‘what’ of page layout, and use
testing to attain the deeper understanding of ‘why’.”
(Marketing Experiments 2010)
51
Discovering, through a reliable test, which
alternative yields the best response,
enables you to generalize that alternative
beyond the test. For example, test five
different copy texts which are ”the nominees”
for new slogan in Facebook, and based on the
results you can say which one resonates best
with the audience (before launching it in mass
marketing).
52
Testing with ”focus group” mentality (small
budget, limited reach), and then expanding
to the mass marketing. (Nothing new here, it
was already done by Ogilvy and Hopkins
starting from 1920s, but now it’s hell of a lot
easier, cheaper and faster. And yet most
copywriters don’t do it!)
53
Testing does not automatically lead into
improvements…
54
Why?
This is why:
• most tests fail to show improvement (survivor
bias)
• small changes → small improvements (garbage
in, garbage out)
• postponing or cancelling the proposed changes
(HiPPO)
55
There are
always
complications.
Smaller differences take longer to clear.
Instead, you would like to make many tests
in a given time period, not only one.
CVR (%) Change (%) Required
sample size
per variant
Days to
clear
Test 1 3 5 72,300 72
Test 2 3 10 18,500 18
Test 3 3 30 2,250 2
56
(Johns, 2015)
”Goodbye Google” (aka 41 shades of blue)
• “Yes, it’s true that a team at Google couldn’t decide
between two blues, so they’re testing 41 shades
between each blue to see which one performs better.
I had a recent debate over whether a border should
be 3, 4 or 5 pixels wide, and was asked to prove my
case. I can’t operate in an environment like that. I’ve
grown tired of debating such minuscule design
decisions. There are more exciting design problems
in this world to tackle.” (Bowman, 2009)
• Is this a case of local maximum problem? What do
you think?
57
How to survive the local maximum
problem?
58
Problem with A/B testing: HiPPO
• ”Listen, I know what works. Let’s do like this and
that’ll be the end of it.”
• ”…okay.”
59
I wish I didn’t
always cave in!
Some theoretical considerations
60
THEORY?? Dude,
who cares!
(Just kidding.)
Compounding benefits of optimization
(Marketing Experiments, 2005)
• Base level in the example:
– monthly sales = 100,000 $
– costs = 85,000 $, monthly profit 15,000 $
• The company implements nine improvements in nine
months (one per month).
61
Compounding benefits of optimization
(Marketing Experiments, 2005)
62
The benefit table
Improvements (1 per month) Gain Profit per
month
Change in
profit
(0. Base line) N/A $15,000 0%
1. Improving PPC ad copy 5% (CTR) $19,500 30%
2. Decreasing CPC 5% (CTR) $19,999 3%
3. Optimizing landing page 5% (CVR) $30,249 51%
4. Optimizing order form 5% (CVR) $35,761 18%
5. Streamlining website copy 5% (CVR) $41,549 16%
6. Adding upselling and cross-selling
options 5% (Sales) $47,627 15%
7. Lowering price 5% (Sales) $49,988 5%
8. Changing shopping cart flow 5% (Sales) $56,487 13%
9. Adding trust indicators 5% (Sales) $63,312 12%
Compounding benefits of optimization:
example (Marketing Experiments, 2005)
• Results:
– Extra profit = 48,312 $
– Compounding benefit = 322 %
– (Simple benefit = 163 %)
• In other words, there is an ”interest to interest” effect
in optimization.
63
Implications for a marketer
• Make systematically small improvements
• Focus on both before and after the click
64
I know it’s boring
at first, but seeing
the fruits of your
labor motivates!
Scaling of optimization: example
• Three companies
A. 1000 visitors per day
B. 10,000 visitors per day
C. 100,000 visitors per day
• Other parameters
– Conversion prior to optimization = 1 %
– Conversion after the optimization = 2 %
– Fixed monthly cost of optimization = 2000 €
– Avg. basket = 50 €
• Let’s optimize. What is the ROI for each company?
65
Scaling of optimization efforts: results
66
0
20000
40000
60000
80000
100000
120000
A B C
myynnin muutos
myynti
kiinteä kust per kk
ROI:
A = -50 %
B = +400 %
C = +4900 %
Multiplication argument (Nielsen, 2008)
a) “In a multiplication, if you want to increase the outcome by
a certain percentage, you can increase any of the factors
by that percentage. It doesn’t matter which factor is
increased — the result will be the same.
b) Thus, to double a site’s business, you can double the
number of unique visitors. However, this would be very
expensive, requiring that you more than double the
advertising budget (assuming you’re already advertising
under the most-promising keywords, and thus need to buy
traffic from less promising or more expensive sources).
c) Alternatively, you can double the conversion rate and
achieve the same business improvement. (…) In most
cases, it’s far cheaper to use 15% of your development
budget than to more than double your advertising budget.”
67
Example: halving the advertising costs
• When you want to double sales, either double the
advertising budget (if possible) or the conversion rate.
• As the number of paid visitors increases, the more
feasible conversion optimization becomes as an
investment
68
Case A: Low
conversion
Case B: High
conversion
Ad spend 100,000 50,000
Visitors (CPC = 0.25) 400,000 200,000
Conversion 1 % 2 %
Sales quantity 4,000 4,000
Return to eCommerce metrics
(Fellman, 2015)
• Visitors
• Conversion rate
• Average basket
• Margin
• Example (monthly sales):
• 100,000 x 0.02 x 100 € x 0.40 = 80,000 €
69
Visitors Conversion rate Average
basket
Margin
Dissect each, and
consider how you can
optimize.
(Remember: this was the
breakdown of metrics that
were deemed important
for ecommerce.)
Metrics for newsletters: tying the chosen
metrics to the optimization process
70
Subscribers x delivery rate
x open rate
x click rate
x conversion rate
x margin
= profit
Ways to improve
Lead generation tactics
Bundling, high-markup
items
Conversion optimization
Quality (mitigation of
spam complaints)
Headline optimization
Why is conversion always low? (Regardless
of optimization)
• P(cv) = P(sees) x P(understands) x P(has need) x
P(has money)
• Let’s advertise to the whole of Finland!
• Assume an optimistic 10% at each step
• 5,000,000 x 0.1 x 0.1 x 0.1 x 0.1 = 500
• 500 customers, yey!
• …but wait, the campaign costed us 50,000 €. Let’s
see: 50,000/500 = 100 €. (Will I keep my job or not?)
71
Another way to look at it…
• At time point t, a sub-set m of total market M has the
need for product x. In a world like this, most products
are not finding a match in the market at the time of
advertising. However, the long-term effect should be
positive to justify the cost.
• (Which takes us back to the question of choosing the
lookback window.)
72
So, what can we do?
All else being equal, a marketer can increase
conversion by improving targeting (driving more
qualified traffic, e.g., focusing only on the end of
the purchasing path) or by driving more overall
traffic in the beginning of the purchasing path.
73
But will the quality
of the traffic
remain constant?
Potential solution: micro conversions
(Google, 2015)
74
Conclusions (1/2)
• Conversion optimization is feasible when
C < avB x (S1 – S0), i.e.
• the fixed cost of optimization (C) is smaller than the
average basket (avB) times the change in sales
quantity (S1 – S0), or when additional sales cover the
costs.
• When there’s a small number of visitors, the cost of
optimization, whether in-house or agency, can easily
exceed the benefits. However, what would be the
exception?
75
”every product has an amazing dropoff of
usage from when people first encounter it”
(Chen, 2015)
76
Conclusions (2/2)
a. Generally, the largest gains from conversion
optimization emerge from removing bottle necks
(which can be found by analysing users’ behavior)
b. The more traffic and sales, the more lucrative it is for
a company to invest in conversion optimization
(because of the scaling effect)
c. When chained, a large number of small
improvements may generate a positive ROI
(because of compounding benefits).
77
The end.
78
Recommended