Causal Attribution - Proposing a better industry standard for measuring digital advertising effectiveness

ProPosing A Better industry stAndArd MeAsure of digitAl Advertising effectivenessNew research provides a foundation for a better system of measuring the effect of

digital advertising and provides a wake-up call to marketers using existing, often

badly misleading, attribution solutions.

Billions of Advertising dollArs Are Being wAsted.

That’s because industry standard measures like

click through rate (CTR), post click and post

impression attribution are not only inherently

flawed, but are being widely manipulated by

intermediaries, and often trick marketers into

optimizing away from their best prospects.

In this study, Collective presents an alterna-

tive method for measuring digital advertising,

which uses rigorous experiments to measure the

increase in desired outcomes caused by display

advertising, verses a correlation effect measured

by existing attribution solutions.

The Causal Attribution measurement approach

outlined in this research will allow advertisers, for

the first time, to accurately evaluate how their

online advertising affects the behavior of specific

audiences and measure the value generated from

their advertising spend.

Key findings:

// Existing attribution solutions are either too

subjective, misleading, or complex to provide

a meaningful industry-standard for advertising

measurement.

// Existing attribution solutions often steal credit

from other advertising sources and misdirect mar-

keters into making poor media decisions.

// Causal Attribution measures the real impact of

a campaign by creating an experiment where the

only difference between two otherwise

Identical pools of users is that one pool (the test)

was potentially exposed to advertising, and the

other pool (the control) was not – ensuring that

any difference in performance observed on the

two pools must have been caused by the decision

to advertise to the test group.

// In multiple live campaign tests, the Causal At-

tribution approach provided an unbiased method

of linking true ROI to advertising spend, including

exposing campaigns that aren’t working.

// Because the experiment is cookie based, ROI

can be cascaded down to individual Audience

Segments, providing rich insights into the types of

users who are being influenced by the advertising.

// Beyond measuring online conversions or their

proxies, Causal Attribution can be used to mea-

sure offline conversions and brand lift.

Further uses of Causal Attribution include:

// Directly comparing the performance of

different sources of media.

// Testing multiple, potentially radically different,

creatives to identify which audiences are most

influenced by each.

// Quantifying the effect of frequency (how many

times each cookie is shown an advertisement)

on ROI.

We believe that widespread adoption of Causal

Attribution could have a tremendous positive

impact on the digital advertising industry. Proper

ROI measurement would lead to increased

digital advertising spending, higher returns for

advertisers and an incentive for publishers to

create quality, engaging content. Finally, con-

sumers would benefit from fewer pay-walls, and

more relevant advertisements that could truly

help their purchase decisions.

executivesuMMAry//

causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness

2

AlgoritHMicAttriBution //

weigHtedAttriBution //

clicK tHrougH rAte //

lAst clicKAttriBution //

lAst iMPressionAttriBution //

cAusAlAttriBution //

trA

nsP

Ar

ent

BlA

cK B

ox

MisleAding AccurAte


3

The maturation of the Web as an advertising me-

dium has spawned several attribution techniques.

These techniques attempt to leverage the Web’s

unique ability to record a recipient’s response to

ad exposure in near real time to quantify the ef-

fects of a campaign on brand value and, ultimate-

ly, sales. Advertising practitioners routinely rely on

the most commonly deployed methodologies to

make significant marketing investment decisions.

Existing attribution models, however, have severe

biases, and when optimized to (or gamed by

intermediaries to “compete” on performance),

these methodologies can erode brand value.

The research that follows demonstrates, for the

first time, how a Causal Attribution solution can

provide marketers with a true accounting of what

advertisements and audiences drive ROI.

The research will also show how the current slate

of attribution methodologies fail to provide mar-

keters with a reliable and transparent method for

understanding the true value of the media they

buy, and in some cases, produces effects counter

to the marketer’s objectives.

But first, a brief overview of the most commonly

used attribution methods:

tHe ProBleM witH current online AttriBution Models // MisleAding, eAsy to gAMe, too suBjective or too coMPlex


4

Half the money i spend on advertisingis wasted; the trouble is i don’t know which half.

–John Wanamaker

“

” {As true in digitAl in 2012 As it WAs in print in 1900}

AlgoritHMicAttriBution //(soMetiMes cAlled ‘interActive’ or, ‘MediA Mix’)

In general, attribution algorithms can be divided

into two categories - weighting schemes and

predictive models - both of which have significant

theoretical and practical weaknesses. Weighting

scheme attribution algorithms rely on human

judgment to assign weights to different types

of advertising events that occurred prior to the

desired outcome. The choice of which events to

weight and how much weight to assign to each

is subjective, even when informed by data, and

may not be any more accurate than using a single

flawed measure alone.

In response to these shortcomings, other compa-

nies have developed attribution algorithms that

use computers to assign credit rather than human

derived weights. These solutions work by assem-

bling a data set of all outcomes and all advertising

events that occurred prior to those outcomes.

Then, using a multivariate predictive model where

the dependent variable is binary (desired outcome

or not) and the independent variables are all

advertising events, these algorithms attempt to

identify which types of advertising events were

more likely to precede the desired outcome than

the undesired outcome.

These computer based attribution algorithms

(often referred to as machine learning algorithms

or predictive models) suffer from significant limita-

tions when used for attribution. First and fore-

most, they often confuse correlation with causality.

This occurs when the computer finds that certain

advertisements tend to precede the occurrence

of the desired outcome in historical data (the two

are correlated). The computer concludes, incor-

rectly, that the advertisements caused the desired

outcomes, and assigns them credit.

In fact, it is often some other unmeasured factor

that was responsible for the observed desired

outcomes.

For example, suppose that a campaign is run-

ning on two very similar sites (A and B), and the

computer is trying to determine how much credit

to assign to each site. Suppose that site A was

bought with retargeting, while site B was bought

without retargeting. The computer will see signifi-

cantly higher conversions for users reached on site

A, driven by the retargeted audience being more

likely to convert regardless of whether or not they

are advertised to. This will lead the computer to

give site A far more credit than it deserves.

Compound this simple example with the

plethora of types of audience targeting,

frequency capping, distribution channels

(display, video, social, mobile, television, etc.),

and other compounding effects (convenience,

social and repeat purchases) and it becomes

clear that these computer based attribution

algorithms are just more complex, with no real

hope of discovering the ‘truth’.

While some attribution models may improve over

traditional measures, like last-click or last-view

based attribution, their “black box” nature makes

gauging success across vendors and campaigns

challenging. Only their creators truly know how

the algorithm works, what it measures and how

it evaluates ROI. Hence, it is unlikely that a

single form of Algorithmic Attribution could ever

become an industry standard.


5

Half the money i spend on advertisingis wasted; the trouble is i don’t know which half.

–John Wanamaker”

confuses correlAtion for cAusAtion.

definitionComputer algorithm assigns credit

for outcomes by analyzing data.

ProsCan overcome some limitations

in simpler metrics.

BottoM line“Pay no attention to the man behind

the curtain.”

consOften confuses correlation

for causation.

Requires measuring all user

interactions in all channels.

No industry standard likely to

be developed.

Only programmers really know how

they work.

wHAt you getRIgHT AuDIENCE

RIgHT mEDIA

RIgHT CREATIvE

? ? ? ? ?? ? ? ? ?? ? ? ? ?

The de facto standard of web metrics, CTR

reports the cumulative number of clicks recorded

on a display ad campaign, divided by the

number of impressions served. The oldest and

easiest measure to capture remains one of the

most commonly used techniques to assess

campaign performance (Chief marketer 2011

Interactive marketing Survey, 2011; Digital

Display Advertising 2010, Collective/ Advertiser

Perceptions, 2010), despite ample evidence of

its irrelevance as a meaningful measure (Natural

Born Clickers, comScore with Starcom uSA and

Tacoda, 2009/2010; CTR: Brand marketing’s most

misleading measure,

Collective 2010).

The case against CTR is overwhelming. The

comScore/Starcom study showed that only 16%

of all Internet users in 2010 clicked on a display

ad in a month, down from 32% a year earlier.

Collective’s own examination of one billion

advertising impressions served in the first months

of 2011 revealed that 99% of stable user cookies

examined never clicked on an ad, and that those

who did were more than two times as likely to

click again in the future. The study also estimated

that as many as 20% of the clicks were accidental,

while Click Forensics estimated in October of

2010 as many as 23% of clicks were fraudulent.

Further, a study conducted by Collective of 100

campaigns showed no correlation between CTR

and brand lift nor purchase intent as measured

by independent post-impression surveys. Hence,

optimization of campaigns to achieve higher CTR

may in fact be reducing brand ROI.

CTR is also an easy metric for third parties

to manipulate by running ads in high-click

environments, such as gaming and mobile sites,

where users will be more likely to accidentally

click on the advertisement. These accidental clicks

often frustrate the user by taking them away from

the content they were viewing and placing them

on a landing page for the brand. In these cases,

the frustrating experience may create negative

associations with the brand, causing users to be

less likely to convert than if no advertising had

been done at all.



6


RIgHT mEDIA

RIgHT CREATIvE

clicKers don’t Buy.

definitionClicks per impression.

ProsAlways on.

No conversions required.

Proxy for engagement, as the ad

must be visible to be clicked.

BottoM lineDigital advertising’s oldest

metric is also its most misleading.

consvery few people click.

Clickers rarely buy.

most clicks are accidental

or fraudulent.

Easily gamed by placing ads near

high click content (e.g., games).

Last click is a derivation of the Click through rate

born of “lower funnel” search advertising, which

assigns credit only to the last click that occurred

within a designated period of time prior to the

desired outcome (the look-back window). This

method disregards all clicks where no conversion

event followed, thus removing all accidental clicks.

However, by assigning 100% of the credit to the

last click this method fails to account for all the

previous advertising messages the user consumed

prior to making the last click, and thus tends to

undervalue display and video advertising. Further,

many users click on search advertisements as a

means of navigating to the product home page,

and would have clicked on an organic link had the

search advertising not appeared. In this fashion,

last click attribution often significantly over-values

search advertising.

The 2008 Atlas study referenced earlier cast

further doubt on Last click methodologies, point-

ing out that, “between 93% and 95% of audience

engagements with online advertising receive no

credit at all when advertisers review the ROI on

their campaigns,” because of misplaced emphasis

on Last click attribution. Quotes the study, “The

“last ad” model forces marketers to place greater

importance on the aspects of their advertising that

support the model, rather than the aspects that

support their advertising success.”

tHe study went on to drAw tHe following MetAPHor:

To illustrate the faulty logic of the model,

imagine that you’re standing in the grocery store

knowing precisely what you want to buy. You’ve

seen the product ads on TV, the full-page ad in

a magazine, and a full color mailer that actually

made you hungry just looking at the pictures.

You’ve even clipped a coupon and brought it

with you to the supermarket. When you ask the

grocery clerk where to find the specific item, he

smiles, points and says, “Aisle five.” Off you go

to aisle five, find the item, pay, and leave the

store. If you applied the “last ad” model to this

scenario, the grocery clerk would get 100% of

the credit for your purchase (no wonder he’s

smiling). As a result, marketers would invest

heavily in grocery clerks, and they’d pull their

advertising dollars from the marketing channels

that actually piqued your interest or moved you

through the funnel toward the purchase.



7


RIgHT mEDIA

RIgHT CREATIvE

disPlAy is often undervAlued.

definitionLast ad clicked prior to outcome.

Prosmeasures only productive clicks.

Difficult for intermediaries to game.

Can be linked to ROI.

BottoM lineuseful for ‘infomercial’ products.

consAds that build awareness or intent

are given no credit.

Some users would have

converted anyway.

Sometimes called the view-through conversion

rate (vTR), last impression attribution assigns

100% of the credit to the last advertising

impression served (or viewed) within a given

period of time prior to the desired outcome. In

some cases, conversions where the user clicked

are not attributed to viewed impressions, in

other cases they are. In contrast to the last click

attribution method, the look-back window is often

much longer (7, 14, 30 or even 60 or 90 days). No

clear standard exists within the industry. google,

for example, defines VTR in its Help Center as “a

measure of the number of online conversions that

happened within 30 days after a user saw, but did

not click, a display ad.”

Last impression attribution was introduced to

overcome shortcomings in last click attribution by

ensuring that viewed impressions also received

credit for conversions. In practice, however, it

gives far too much credit to digital advertising.

When the standard is an impression delivered,

the user may not even have been able to see

the advertisement that receives credit. When

the standard is a ‘viewed’ impression then while

the advertisement may be visible, there is no

guarantee that the consumer was influenced

by the advertisement to complete the desired

outcome. This leads to retargeting receiving far

more credit under last impression attribution

schemes than it should, due to the fact that

the users would have completed the desired

outcome regardless of whether or not they saw an

advertisement, yet the advertisement gets 100%

of the credit anyway.

Last impression attribution is also very easy to

manipulate (even more so than click through rate).

Intermediaries are incentivized to target users

who are likely to convert, regardless of whether

they will be influenced by the advertisement, with

the cheapest media buy possible. In the worst

case scenario, when an advertiser works with

only one intermediary, that provider can simply

hit every stable cookie with a single poor quality

advertisement once per N days (where N is the

look-back window) and they will receive credit for

close to 100% of conversions.

lAst iMPression //

1 Determining if an advertisement is visible remains a challenge for many vendors. The vendor must run code in the browser to determine if, and for how long,

an advertisement is viewable to the user. Such code is often confounded by iFrames, a publisher technology that quarantines advertisements.

See www.adexchanger.com/data-driven-thinking/viewable-impression for more details.


8

definitionLast ad viewed prior to outcome.

Prosgives credit to views not clicked on.

BottoM lineEncourages advertising to consumers

who would have converted anyway.

consAds do not have to influence

consumer to receive 100% credit.

Quality of media, placement and

creative have little impact.

Easily gamed through ‘spray & pray’

media buying.

Overvalues retargeting (would have

bought anyway).


RIgHT mEDIA

RIgHT CREATIvE

Research from Collective’s Data Sciences team

explores the viability of a Causal Attribution

measurement scheme that, through a carefully

constructed experiment, directly observes

the change in desired outcomes caused by

incremental digital advertising spend. Because

this method uses an A/B test experimental

design established prior to the incremental

advertising, the results are unbiased by organic

conversions and by other advertising spend,

including spend offline.

Further, because the design is based on random

audience groups, rather than randomly selected

impressions, Causal Attribution can measure

the cumulative effect of multiple advertising

impressions over a period of time on individual

users. This is accomplished without having

to waste any impressions on public service

announcements (PSAs), a common requirement

of other experiment based digital measurement

approaches. When combined with a measure

of desired outcome value (e.g., profit from a

conversion), Causal Attribution can measure

the true return on investment (ROI) driven by

incremental advertising spend.

In the sections that follow, we will:

// Illustrate how Causal Attribution works through

a simple six step process.

// Outline the types of outcomes that can be

measured using Causal Attribution.

// Examine in detail the results of two real

campaigns where Causal Attribution was used to

quantify ROI and identify optimal audiences.

// Introduce an alternate methodology which

uses PSA advertisements.

// Outline additional applications that can

optimize creative, media and frequency.

We will then conclude with a more thorough

discussion of the benefits (and limitations) of

Causal Attribution for advertisers, and a discussion

of how widespread adoption of this best practice

could impact the industry as a whole.

cAusAl AttriBution //


9

siMPle, scientific, unBiAsed, true roi.

definitionExperiment measures outcomes

caused by advertising.

ProsDirectly quantifies ROI.

unbiased by all other advertising.

Transparent and easy to administer.

Provides rich audience analytics.

BottoM linemeasures causality, not correlation,

providing a true measure of ROI.

consRequires large audience database.

Does not always produce results

that are statistically significant.


RIgHT mEDIA

RIgHT CREATIvE

Step 1 // define tHe Audience cloud

We first limit the experiment to cookies that

are likely to persist for the duration of the

experiment. This is done to limit the impact of

cookie deletion. At Collective, we define a stable

cookie to be one seen at least once within the

last 28 days and on at least two separate days

over the life of the cookie. This ensures that the

cookie was recently seen, and was persistent for

at least a period of two days (and so unlikely to

have been automatically deleted by the browser

after the session).

We have found that cookies seen on at least two

separate days of the prior 28 are 22 times more

likely to appear the following day than cookies

seen on only one day. On average at Collective,

this provides a stable universe of approximately

200 million users, which we refer to as our

Audience Cloud.

Step 2 // divide tHe users into test And control grouPs

Before the campaign begins, we randomly divide

the entire Audience Cloud into test and control

groups using a robust random number generator.

This split is repeated with a different initial seed

for every advertising campaign, ensuring that

the randomization is unique for each advertiser

and there is no cross-experiment pollution. In

practice, we typically choose anywhere from 5%

to 50% of the cookies to be in the control group,

depending upon the length of the campaign and

the frequency of the desired outcome absent

advertising.

given an Audience Cloud of 200m users, this

means our control group will consist of anywhere

from 10m to 100m users. These sample sizes

ensure that the test and control groups will be

balanced across all other influencing factors, such

as audience demographics and exposure to other

advertising (either digital or traditional).

Step 3 // deliver AdvertiseMents only to tHe test grouP

We then begin the advertising campaign, and

ensure that cookies in the control group are

never exposed to the advertising. In practice,

this is done by negatively targeting an audience

segment in our ad-serving engine that includes all

of the cookies in the control group. Whenever an

impression arrives for a user in the control group,

the campaign is excluded from the potential set of

campaigns to be shown.

Note that for the purposes of measuring the

impact of the campaign we do not need to

adjust the experiment to control for any audience

targeting, media selection, creative optimization

or frequency capping was used in the campaign.

This is possible because we are comparing the

entire control group with the entire test group,

regardless of who the ad was delivered to.

Being able to ignore these other factors drastically

simplifies the process of analyzing the experiment,

and also ensures that we can run the experiment

on any campaign without disruption, allowing for

ongoing monitoring and optimization.

How cAusAlAttriBution worKs //The great French physicist and mathematician, Henri Poincaré, once said that “experiment is the sole source of truth; it alone can teach us something new; it

alone can give us certainty.” Without experimenting, no amount of conjecture or analysis of data can prove a scientific theory. Similarly, without experimenting, we

cannot prove that advertising has caused a desired outcome, nor can we truly optimize our ROI.

Causal Attribution works by creating an experiment where the only difference between two otherwise identical pools of users is that one pool (the test) was

potentially exposed to advertising, and the other pool (the control) was not. This design ensures that it was our decision to advertise to the test group that caused

any statistically significant difference measured in the desired outcome rates of the test and control pools.

In practice, there are some subtle and important decisions to be made in conducting a Causal Attribution experiment. The following six steps outline the process

and illustrate how Collective conducts these experiments on behalf of advertisers.


10

2 Negatively targeting the control group also allows the campaign to deliver outside of the universe of stable cookies, which is often desirable for meeting

delivery goals and maximizing reach. One complication of this approach is that when computing ROI we must remove spend on impressions outside of the test

and control groups (see step 6).

Step 4 // oBserve desired outcoMes on All users

During the course of the campaign, and for a

period of time after the campaign concludes we

observe all desired outcomes on all cookies in

the Audience Cloud.

It is critically important that we observe all

desired outcomes, as the outcome rate in

the control group serves as our baseline for

evaluating lift caused by the advertisement.

For example, applying a last view or last click

attribution scheme would discard those desired

outcomes that did not have a view or click event

within a selected look-back window. Thus,

the control group conversion rate would be

artificially reduced to 0% and the experiment

would be invalidated.

Similarly, all desired outcomes should be counted

in the experiment, regardless of the path that

the user took to complete the desired outcome.

For example, if we are measuring online product

purchases, then we should track all purchases

online regardless of how the user clicked to arrive

at the checkout page – be it directly through the

uRL, through an organic or sponsored search, or

by clicking on a display or video advertisement.

This ensures that we capture the full impact of the

advertising impressions by including all purchases

that the advertising could have influenced.

Step 5 // MeAsure tHe desired outcoMes rAtes

After the campaign has concluded and the

follow-up period has elapsed we can compute

the rate of desired outcomes, defined as the

total number of outcomes divided by the total

number of cookies, in both the test and the

control groups. Note that, depending upon the

type of outcome being measured, you can either

allow cookies to complete the desired outcome

more than once or only once. For example, if

the outcome is a purchase, then you should

count all purchases from each cookie. If instead,

the outcome is a survey response or a site

registration, you may wish to count the outcome

only once per cookie.

Step 6 //coMPute tHe cAusAl lift

We define the causal lift of the campaign as the

lift observed in the desired outcome rates in the

test group over the control group. For example,

suppose we find that 0.12% of users in the test

group purchased the product, but only 0.10%

of users in the control group purchased the

product. given that these two user pools were

randomly chosen from the same population, and

the only difference was our decision to advertise

to the control group, we can conclude that the

advertising caused users to be 20% more likely to

convert over the time period in question.

sPlit test & control

(Audience cloud 200 Million cooKies)

cAMPAign runs MeAsure outcoMes

ADVERTISE TO TESTOBSERVE OUTCOMES

TEST (REMAINING 90%)

CONTROL (RANDOM 10%)

TEST OUTCOME RATE

# TEST OUTCOMES# TEST COOKIES

# CONTROL OUTCOMES # CONTROL COOKIES

CONTROL OUTCOME RATE

=

=




TEST OUTCOME RATE




=

=




TEST OUTCOME RATE




=

=

online conversions orActions (Proxies):

Consumer online purchasing or other desired online

activities, such as visiting a website, completing a

registration, viewing a video, etc., can be measured

using outcome pixels, such as DART Spotlight

activities or AmP audience pixels. In general, these

pixels should be closely linked to potential revenue

generation, and should be widely accessible to

consumers, regardless of their path to conversion.

This ensures the full impact of the advertising spend

is captured, and also helps to ensure statistical

significance of results.

off-line conversions:

Though a less immediate measure, off-line sales can

be applied to Causal Attribution logic through the

import of anonymized point of sale and/or credit

card transaction data, available through third party

vendors. Similarly, an advertiser’s internal CRm

purchase log data can be anonymized and moved

online to a cookie through a data management

platform. This approach allows the advertiser to

quantify the impact their online advertising is having

on offline sales without having to use any personally

identifiable information (thus protecting consumer

privacy) in the design or analysis of the experiment.

BrAnd MeAsureMent:

An important and often overlooked metric in

digital advertising today is evaluating the impact

of advertising spend online on brand awareness,

message recall and future purchase intent.

Custom surveys executed through rich media ad

units can record the responses directly to a user

cookie so that the (self-reported) results of a brand

measurement survey can be observed across

the test and control groups . At the conclusion

of the experiment, any lift observed in desired

survey response rates in the test group over the

control group can be credited to the incremental

advertising spend.

desired outcoMes tHAt cAusAl AttriBution cAn MeAsure //cAusAl AttriBution cAn MeAsure A cross section of Advertising oBjectives:

3 Note that for brand measurement, it can be significantly more cost effective to conduct the Causal Attribution study using a creative based approach with

public service announcements (PSAs). This limits the audience where surveys should be collected to just those users who received an advertisement in the test

group or a PSA in the control group. See the PSA methodology section for more information.


12

The Collective Data Sciences team analyzed the

behaviors of 200 million stable audience cookies

across 14 live advertising campaigns during a

period from October 2011 through January

2012. In all, 380 million advertising impressions

were examined. The audience profiles examined

contained an average of 70 data attributes, such as

location, demographics and enthusiast behaviors.

Two Selected examples follow.

exAMPles of cAusAl AttriBution tests on live cAMPAigns //

In this test, advertisements for a national hotel

chain were analyzed for their ability to drive online

bookings. The test analyzed a test group of 180

million cookies and a control group of 20 million

cookies. At the end of two months, the exposed

group converted at a 0.0050% rate, and the

control group converted at a 0.0044% rate. This

means that the advertising spend caused users

to be 14% more likely to convert than they would

have otherwise. using a difference in propor-

tions test , we find that the p-value for this lift is

0.00005, indicating that it is highly statistically

significant, with less than a 1 in 20,000 chance of

observing a lift this high randomly.

What is more, we can convert this lift directly

into a return on investment (ROI). The test group

converted at 0.006 percentage points higher than

the control group. multiplying this rate by the

number of cookies in the test group we can con-

clude that there were 1,099 additional conversions

caused by this campaign. The client valued online

conversions at $30 each, therefore, this additional

advertising spend generated $32,970 worth of

incremental value for the hotel chain.

The campaign cost $25,000, and so the ROI for

the advertiser was 32%. meaning that, for every

$100 the advertiser invested in this campaign, they

made $32 of incremental profit.

The experiment becomes even more interesting

as the nature of the converting customers is exam-

ined and we compare Causal Attribution results

for audience segments to performance measured

using other attribution methods.

For example, we can segment the Audience Cloud

by the context of the pages each user most often

frequents. We call these segments enthusiast

behaviors, as they indicate that a cookie is enthu-

siastic about a category of content (e.g., music,

sports, food). Because the test and control groups

were randomly selected from the entire Audience

Cloud, each of these enthusiast groups will be ran-

domly split between the test and control popula-

tions, and we can analyze the causal lift generated

within each segment.

In the chart to the right, we show how each enthu-

siast group compares to the average by plotting

the percentage lift (on the horizontal axis) for a

particular metric for each group.

The metrics shown are the control conversion rate

in gray (how often the control group converts ab-

sent advertising), the click through rate in pink and

the causal lift in blue (how much more likely users

are to convert when exposed to advertisements).

The enthusiast behaviors where we observed at

least 500,000 users in the control groups have

been sorted on the vertical axis by the control

conversion rate lift in descending order.

We find that the users most likely to book hotel

rooms in the control group are those reading

about real estate and, not surprisingly, travel.

These groups are also among the highest in click

through rate. Yet their causal lift is actually nega-

tive, indicating that advertising to these groups

will not make them any more likely to convert than

they already were.

cAse study trAvel BrAnd //


14

teSt ReSULtS: tRAVeL BRAND //While a 32% ROi is a pOWeRful Result (eveRy $100 spent dROve $32 Of incRemental pROfit), audience taRgeting cOuld have have geneRated an ROi as high as 150%.

14%

0.00005

1,099

$30

$32,970

$25,000

32%

criteriA

# cooKies

# conversions

conversion rAte

conversion lift

stAtisticAl significAnce

conversions cAused

vAlue of A conversion

vAlue generAted

sPend

roi

CONVERSIONS / COOKIES

(TEST RATE - CONTROL RATE ) / CONTROL RATE

P - VALUE FOR CONVERSION LIFE EXCEEDING 0 %

(TEST RATE - CONTROL RATE) X TEST COOKIES

CONVERSIONS CAUSED X VALUE OF CONVERSION

(VALUE GENERATED - SPEND) / SPEND

control

20

882

0.0044%

test

180

9,037

0.0050%


15

MEASUREMENT

MEASURE LIFT (%)

CLICKERS

CONTROL CONVERTERS

CASUAL LIFT

GAMES

FASHION

FINANCE

MUSIC

TECHNOL-

HEALTH

SCIENTIFIC

AUTOMO-

POLITICAL

FAMILY

SHOPPING

FOOD

WEATHER

SPORTS

TRAVEL

REAL

-100 -50 0 50 100 150

While real estate and travel enthusiasts book many hotel rooms, they Were not influenced by the advertising. PeoPle reading about fashion and shoPPing Were.

clicKers

control converters

cAusAl lift

-100 -50 0 50 100 150

MeAsure lift (%)

reAl estAte

trAvel

sPorts

weAtHer

food

sHoPPing

fAMily

PoliticAl

AutoMotive

scientific

HeAltH

tecHnology

Music

finAnce

fAsHion

gAMes

In contrast, near the bottom of the chart we find

that in the control group those users reading

about fashion are less likely to book hotel rooms

than those reading about travel, and are less likely

to click on the hotel chains ads. However, they

are more than 50% more likely to convert when

they were exposed to the campaign. Thus, we

can conclude that the advertising is significantly

influencing these users despite their low response

rates using traditional attribution methods.

What then is the optimal audience profile for this

hotel chain? To answer that question, we can

analyze the return on investment for every

audience segment.

While the brand achieved a 37% ROI on the

campaign as a whole, there are large audience

groups where the ROI is significantly higher. Users

who are older and users with higher incomes

generate ROIs ranging from 50% to 125%.

Similarly, there are large pockets of users by

geography, browser and enthusiast behavior who

generate much higher ROIs.

By selectively targeting those segments exhibiting

higher ROIs, an advertiser can drive materially

higher performance. Further, sophisticated look

alike modeling techniques can be employed to

score every user in the audience cloud from lowest

expected ROI to highest expected ROI, using

a multitude of factors including demographics,

geography and behavior online. Then, a custom

audience group can be created of just those users

who we expect to deliver the highest ROI for the

advertiser.

cAse study : trAvel BrAnd (continued)


16

18-2

425

-34

35-4

445

-54

55-6

464

+

00-2

525

-50

50-7

575

-100

100-

150

150-

250

250+

FEM

ALE

MAL

E

ES C

ENTR

ALW

S CE

NTRA

L

EN C

ENTR

ALS

ATLA

NTIC

MID

DLE

ATLA

NTIC

PACI

FIC

MOU

NTAI

NW

N CE

NTRA

LNE

W E

NGLA

ND

CHRO

ME

IEFI

REFO

X

FITN

ESS

TRAV

ELRE

LIGI

ONRE

AL E

STAT

E

FAM

ILYSP

ORTS

AUTO

MOT

IVE

FOOD

WEA

THER

GAM

ESM

USIC

POLI

TICA

LSC

IENT

IFIC

FINA

NCE

HEA

LTH

HOU

SEH

OLD

TECH

NOLO

GYPE

TSSH

OPPI

NGFA

SHIO

NGA

DGET

SJO

BS

AUDIENCE SEGMENT

150

100

50

0

-50

-100

-150

To illustrate the impact on performance that op-

timizing to ROI can have, suppose the advertiser

selected the top 20% of audience segments by

their conversion rate alone. The advertiser would

achieve a conversion rate of 0.093% and a causal

ROI of 33%. If instead they were to select the top

20% of audience segments by ROI, their conver-

sion rate would drop to 0.059% (a 35% decrease),

significantly hurting their last view impression

conversion rate. However, their ROI would rise to

104% (a 216% increase), generating more than 3

times as much value per dollar invested.

This is illustrated in the chart to the right, in which

each audience segment is a circle arrayed on a

plot where the x-axis is that audience’s conversion

rate in the control group, and the y-axis is that

audience’s ROI. Last view conversion attribution

analysis includes the 6 pink audience groups in

the top 20%, whereas an ROI analysis correctly

identifies the 6 blue audience groups instead. (The

4 split color circles in the upper right were selected

by both strategies.)

This finding also holds true for click through rate.

Were the advertiser to select the top 20% of audi-

ence segments by their CTR alone, they would

achieve a CTR of 0.048% and a causal ROI of 45%.

If instead they were to select the top 20% of audi-

ence segments by ROI, their CTR would drop to

0.026% (a 46% decrease), yet their ROI would rise

to 104% (a 129% increase).

cAse study : trAvel BrAnd (continued)


17

KeyoPtiMize to conversion rAteoPtiMize to roioPtiMize to BotH

0.002 0.004 0.006 0.008 0.010 0.012

-50

0

50

100

CONVERSION RATE (%)

RETU

RN O

N IN

VES

TMEN

T (%

)

oPtimizing on causal attributiondrives 216% higher rois than last vieW.

In this test, advertisements for a fashion retail

brand were analyzed for their ability to drive

registrations for an online website. The test

analyzed 180 million exposed cookies and a

control group of 20 million. The exposed group

converted at a 0.0185% rate, and the control

group at a 0.0184% rate, meaning that the

test group that included users exposed to the

advertising was just slightly more likely to register

for the website as those who had not been

exposed to the ads. The p-value for this lift is 0.37,

indicating that it is not statistically significant, with

more than a 1 in 3 chance of observing lift at this

level by chance alone.

The campaign caused 188 conversions, valued at

$10 each, for a total of $1,880 of value generated

for the advertiser. The campaign cost $15,000,

demonstrating a negative 87% return

on investment.

This experiment exposed for the client that

their current advertising campaign was not

driving enough registrations to justify their

media spend. It is critical to observe that in this

case the standard attribution models painted

a very different picture. Whereas the standard

performance measures being used by the client

led them to believe that this campaign was one

of their best performing, our Causal Attribution

study revealed that it was actually having almost

no impact on online registrations. This ability to

definitively identify campaigns and optimization

strategies that are not working is one of the

greatest advantages of using Causal Attribution.

For example, the control converters, those who

converted without being advertised to, were

young and generally mid-to-high income, whereas

the ‘clickers’ were older and lower income. In

contrast, the blue causal lift lines better represent

the actual responsiveness of a group to online

ads. In this case, causal lift was fairly consistent

regardless of age and household income.

This audience analysis highlights how misleading

existing attribution solutions can be. By targeting

high-income young audiences, a partner could

drive a very high last impression conversion rate.

By targeting middle-income older audiences, a

partner could drive a very high click through rate.

But neither strategy would materially impact causal

lift – that is, convince consumers to register online

who would not have registered anyway.

In general, we find that audience groups that

are naturally predisposed to be likely to convert

or click are rarely the groups that will be most

influenced to convert by online advertising. Only

through the analysis of Causal Attribution results

can an advertiser truly discover who the optimal

audience is for a given advertising campaign.

The results of this analysis led the advertiser to

conclude that they should experiment with new

creative strategies, and use Causal ROI to test

which audience groups are most receptive to

each creative.

cAse study retAil BrAnd //


18

teSt ReSULtS: RetAiL BRAND // tRaditiOnal metRics (last click, last vieW) pOsitiOned this campaign as One Of the tOp 2 On this adveRtiseR’s buy, yet causal attRibutiOn pROved that it Wasn’t deliveRing value. the adveRtiseR is using the Rich RepORting fROm the causal attRibutiOn analysis tO Revisit theiR cReative and audience taRgeting stRategies.

1%

0.37213

188

$10

$1880

$15,000

-87%

criteriA

# cooKies

# conversions

conversion rAte

conversion lift

stAtisticAl significAnce

conversions cAused

vAlue of A conversion

vAlue generAted

sPend

roi

CONVERSIONS / COOKIES

(TEST RATE - CONTROL RATE ) / CONTROL RATE

P - VALUE FOR CONVERSION LIFE EXCEEDING 0 %

(TEST RATE - CONTROL RATE) X TEST COOKIES

CONVERSIONS CAUSED X VALUE OF CONVERSION

(VALUE GENERATED - SPEND) / SPEND

control

20

3,689

0.0044%

test

180

33,389

0.0185%


19

MeAsureMent

clicKers

control converters

cAusAl lift

AGE HHI Measurement

Clickers

Control Converters

Causal Lift

150

100

50

0

-50

-100

18-24 25-34 35-44 45-54 55-64 0-25 25-50 50-75 75-100 100-150 150-250 250+65+

inverses : converters Were young but clickers Were older.

converters Were Wealthier;clickers, less so.

An alternative to the methodology outlined in

this paper is to execute a Causal Attribution

experiment through the use of public service

announcements (PSAs) in the creative serving

engine. Steps 1 & 2 are the same, but in step 3

instead of negatively targeting the control group,

we serve a public service announcement (PSA)

to the control group, and serve the campaign

advertisement to the test group. Note that the

test and control groups must still be cookie based,

rather than impression based, or else cookies

will move back and forth between the test and

control groups polluting any cookie based

conversion analyses.

The main advantage of serving a PSA is that it

narrows the scope of the experiment from all

200m cookies in the Audience Cloud to just those

served an advertisement (either PSA or from the

campaign) in the experiment. In some cases, this

can improve the statistical significance of the

results. It can also dramatically reduce the number

of desired outcomes that must be gathered

when measuring brand lift impact. However, the

requirement to serve a PSA is cost prohibitive,

especially when conducting Causal Attribution on

a routine basis with a large control group.

An additional advantage to using a PSA is that

control over the audience targeting and ad

decisioning is no longer needed. Instead, the

experiment is conducted purely in the creative

serving, ensuring that the control group only

receives PSA advertisements. This allows Causal

Attribution to be extended across media buys

spanning multiple partners and channels.

One common misconception is that in conducting

a creative-based Causal Attribution experiment

one can analyze the impact that frequency has on

the desired outcome rate. This can be attempted

by evaluating the lift in test over control

conversions by cookie stratified by the frequency

with which the cookie was reached (by either

the PSA or real advertisement). Thus, one might

observe that the lift in conversions for users with

a frequency of 5 was 20% higher than the lift in

conversions for users with a frequency of 1.

This type of analysis, however, is seriously flawed.

The primary driving factor behind this measured

frequency is not the decision to serve the

advertisement, but is instead the user’s frequency

of use of the Internet, and the percentage of

those impressions purchased. Thus, any difference

in conversion rates by frequency could be caused

by these underlying biases rather than by having

been exposed to the advertisement

additional times //

PsAMetHodology//


20

creAtive AnAlysis

The choice and execution of creative, be it display

or video, is undoubtedly one of the most influen-

tial decisions an advertiser can make in executing

a digital advertising campaign. Rather than relying

upon intuition alone in designing creative, Causal

Attribution allows an advertiser to test multiple,

potentially radically different, creatives to identify

which audiences are most influenced by each.

Thus, an advertiser can increase the effective reach

of their campaign and truly serve the right audi-

ence with the right creative. This is accomplished

by dividing the test audience into random groups,

each of which is only ever exposed to one type of

creative. Combined with dynamic creative optimi-

zation, a given creative can be further optimized to

achieve maximal ROI.

MediA AnAlysis

The choice of where an advertisement is run is

also a critical decision in any digital campaign.

Causal Attribution experiments can be designed

to test the impact that advertising in a premium

environment has over advertising elsewhere on the

Internet, in either longer tail inventory or with user

created content. Again, this is accomplished by

dividing the test group into multiple pools, each of

which is targeted only in certain ad environments.

Similarly, one can compare inventory purchased

directly from publishers with inventory purchased

indirectly through exchanges.

frequency AnAlysis

Causal Attribution can also be used to quantify

the effect of frequency (how many times each

cookie is shown an advertisement) on ROI. This

is accomplished by dividing the Audience Cloud

into several smaller pools (e.g., 2 million cookies

each). One is kept as a control group, and the oth-

ers are targeted in the same manner with varying

frequency caps. For example, we might create four

small pools:

// A control pool that will never be shown

an advertisement.

// A pool with a 1 per 1 day frequency cap.

// A pool with no frequency cap.

Keeping each pool relatively small increases our

chances of hitting cookies in the pools at close to

the desired frequency cap rate.

furtHerAPPlicAtions //In addition to analyzing ROI at the campaign and audience profile level, there are three significant ad-

ditional applications of Causal Attribution. While we will only briefly cover them in this white paper, they

will be the basis of additional research in the future. Note that in the following sub-sections we describe

each experiment in isolation, but they can in fact be conducted simultaneously using multivariate

testing strategies.


21

4 Statistical significance is governed by three factors: i) the size of the control group, ii) the amount of lift generated by the campaign, iii) the proportion of users

completing the desired outcome absent advertising. In practice, we have found that the third factor is most often to blame for inconclusive results, and caution

against running causal attribution studies based on very rare desired outcome events.

tHe Benefits And liMitAtionsfor cAusAl AttriBution for Advertisers //

tHe Benefits

The present study establishes that Causal Attribu-

tion provides a compelling system for evaluating

digital advertising ROI that avoids many of the

pitfalls present in other attribution solutions. The

following are some of the principle benefits of

Causal Attribution to advertisers:

// Directly quantifies return on investment (value

generated from spend)

// unbiased by all other advertising (occurs in

both test and control groups)

// Transparent and easy to administer (no propri-

etary algorithms)

// Provides rich audience analytics (identify con-

sumers who will be influenced)

// Effective across all channels where cookies are

used (display, video, mobile web)

tHe liMitAtions

Simply put, no other attribution system can pro-

vide all of these advantages. However, there are

some limitations of Causal Attribution that should

be noted:

// Depends on browser cookies, which are often

deleted (biases ROI low)

// Limits the reach of a given campaign by the size

of the control group

// Does not always produce results that are statis-

tically significant

tHe conclusion: A ProPosAl for A new industry MeAsureMent stAndArd

At Collective, we firmly believe that the benefits to

advertisers of Causal Attribution far outweigh the

limitations. Combined with our findings on how

current methodologies can produce false success

signals that misguide marketers into making poor

media planning and optimization decisions, we

believe that Causal Attribution should be adopted

as an industry standard.


22

The digital advertising industry has become

enamored with data, technology and algorithms.

Billions of data points are analyzed, inventory is

dynamically selected through real time bidding,

and sophisticated audience, media and creative

optimization strategies are deployed. And yet, all

of this investment is frequently evaluated using

misleading attribution methodologies. When

coupled with highly competitive and sophisticated

media partners using questionable optimization

tactics, these metrics are severely hindering

the industry.

To quantify the economic impact of the problem

today, consider that Forrester estimates in their

“uS Interactive marketing Forecast, 2011 to 2016”

report that the market for display advertising was

close to $12 billion in the uS in 2011. Assume

that 75% of this spend was for direct response

campaigns. Further, conservatively assume that

20% of that spend was wasted due to misleading

measurement systems. Then this implies that $1.8

billion of spend was wasted in 2011 in the

uS alone.

Widespread adoption of Causal Attribution would

have an enormous impact on the entire advertis-

ing ecosystem. Digital advertisers would rapidly

cut the fraction of their spend that is not effective,

and be willing to pay significantly more for the

right audiences and ad environments that deliver

high ROIs. CmOs would be willing to spend more

of their advertising budgets in digital where they

can concretely measure the value the advertis-

ing is generating. Publishers would respond by

limiting the supply of lower quality and less visible

advertising inventory, and instead focus on provid-

ing advertisers with impressions that have a higher

probability of influencing consumers.

Ultimately, more advertising dollars would flow

into digital channels, Advertisers would get higher

returns on their investment and publishers would

be rewarded for generating higher quality engag-

ing content. Finally, consumers would win by en-

joying higher quality content with fewer pay-walls,

and by receiving more relevant advertisements

that could truly help their purchase decisions.

How cAusAl AttriBution could cHAnge tHe industry //


23

causal attribution measurement will allow advertisers, for the first time, to accurately evaluate how their online advertising affects the behavior of specific audiences and measure the roi generated from their advertising spend.

“”

– Jeremy stAnley, sVp product And dAtA sciences, collectiVe

MeAsureMentstrAtegies //

definitionLast ad clicked prior to outcome.

Prosmeasures only productive clicks.

Difficult for intermediaries to game.

Can be linked to ROI.

BottoM lineuseful for ‘infomercial’ products.

consAds that build awareness or intent

are given no credit.

Some users would have converted anyway.



RIgHT mEDIA

RIgHT CREATIvE

definitionClicks per impression.

ProsAlways on.

No conversions required.

Proxy for engagement, as the ad

must be visible to be clicked.

BottoM lineDigital advertising’s oldest metric is

also its most misleading.

consvery few people click.

Clickers rarely buy.

most clicks are accidental or fraudulent.

Easily gamed by placing ads near high

click content (e.g., games).



RIgHT mEDIA

RIgHT CREATIvE

definitionLast ad viewed prior to outcome.

Prosgives credit to views not clicked on.

BottoM lineEncourages advertising to consumers

who would have converted anyway.

consAds do not have to influence consumer

to receive 100% credit.

Quality of media, placement and creative

have little impact.

Easily gamed through ‘spray & pray’

media buying.

Overvalues retargeting (would have

bought anyway).

lAst iMPressionAttriBution //


RIgHT mEDIA

RIgHT CREATIvE


24

definitionComputer algorithm assigns credit

for outcomes by analyzing data.

ProsCan overcome some limitations

in simpler metrics.

BottoM line“Pay no attention to the man behind

the curtain.”

consOften confuses correlation for causation.

Requires measuring all user interactions

in all channels.

No industry standard likely to be developed.

Only programmers really know how

they work.


RIgHT mEDIA

RIgHT CREATIvE

? ? ? ? ?? ? ? ? ?? ? ? ? ?

AlgoritHMicAttriBution //

definitionuser determines a weighting scheme

for mixing CTR, last click and last view.

ProsAverages out some of the misleading

facets of simpler metrics.

gives the advertiser or agency “knobs

and dials“to control.

BottoM linegarbage in, garbage out.

consPoor metrics cannot be mixed together

into a good metric.

Choice of weights, even when informed by

data, is highly subjective.

Still provides no direct link to ROI.

weigHtedAttriBution //


RIgHT mEDIA

RIgHT CREATIvE

definitionExperiment measures outcomes

caused by advertising.

ProsDirectly quantifies ROI.

unbiased by all other advertising.

Transparent and easy to administer.

Provides rich audience analytics.

BottoM linemeasures causality, not correlation,

providing a true measure of ROI.

consRequires large audience database.

Does not always produce results

that are statistically significant.


RIgHT mEDIA

RIgHT CREATIvE

cAusAlAttriBution //


25

ABout collective //Collective intelligently connects brands to audiences with high-impact experiences across

display, video and mobile. Collective’s AmP(R) Data and media management platform

powers the ad businesses of over 50 leading media brands, including our flagship media

products, Collective Display and Collective video.(R) Collective’s complete buy-side solu-

tion, Ensemble,(Tm) provides brand advertisers with audience buying combined with rich

media and DCO.

Collective is headquartered in New York with offices in Atlanta, Boston, Chicago, Dallas,

Detroit, Los Angeles, San Francisco, London and Bangalore. Collective’s investors include

Accel Partners(R), greycroft Partners and iNovia Capital. For more information, please visit

www.collective.com.

Press contAct //Laura Colona

Director of Communications

[email protected]

sAles //[email protected]

99 Park Avenue, 5th Floor New York, NY 10017 888-460-9513

Marketing

Causal Attribution - Proposing a better industry standard for measuring digital advertising effectiveness