Upload
vivastream
View
2.228
Download
2
Embed Size (px)
Citation preview
Presented by Disney & SAS
October 2012
Disney Marketing ROI Case Study
DMA Conference
Defining A Marketing ROI Solution
Reach the right audience
Through the right channel
At the right time
With the right frequency
At the right price
Maximize Return
on investment for
marketing spend
Stand-alone studies often fail to achieve long-term
success—trying to implement a project instead of a process!
Presentation Agenda
• Introduction
– Disney Management Science & Integration
– SAS
– The Science Behind Marketing ROI
• Case Study Overview
– Project Goals & Organization
– Data Management
– Science Integration
– Tool Development
• Lessons Learned
• Questions & Answers
Disney Management Science and Integration
• Consulting support for analytics, data and reporting needs
• Technology integration for reporting and data tools
• Development and management of decision science tools
4 employees - 2008 30 employees - 2012
SAS® Company Overview
• 2011 & 2010 Fortune
Magazine: #1 Place to Work
• 2011 Revenue: $2.73 billion
• SAS® reinvests ~25% of annual
revenue into R&D
• 90 of top 100 companies on
FORTUNE Global 500® use
SAS®
SAS® is the largest
independent software vendor
in the world
SAS Annual Revenue 1976-2011
Science Behind Marketing ROI – Modeling
Marketing Spend
Sale
s
TV
Radio
Marketing Effort
For Each Channel (spend, impressions, etc.)
vs.Response
Variable (sales, leads, etc.)
Measurement Model
More effective
Lesseffective
Science Behind Marketing ROI – The “Right” Model
Regression / Time Series Model
R2 = 97%
Econometric / Panel Model
R2 = 67%
Sales (t) = …
+0.7 * Sales (t-1)
-0.2 * Price
+0.06 * TV
-0.005 * Online
+…
Sales (t) = …
+0.2 * Sales (t-1)
-1.0 * Price
+0.1 * TV
+0.02 * Online
+…
Heavy weight on lagged sales; sales not
responsive to price & media changes
Less weight on lagged sales; price &
media elasticities more reasonable
Selecting the right modeling approach is critical for success!
Better for FORECASTING Better for MEASUREMENT
Science Behind Marketing ROI – Measurement
Analysts pay careful attention to data considerations and
choice of models to robustly fit the data for measurement
Saturation Curves
GoodwillImpressions by
Media Type
Model
Cable
Radio
Spend
Rati
ng
s
Rati
ng
s
Time
Imp
res
sio
ns
Time
Model Input Model Output
Saturation Curves
GoodwillImpressions by
Media Type
Model
Cable
Radio
Spend
Rati
ng
s
Rati
ng
s
Time
Imp
res
sio
ns
Time
Radio
Spend
Cable
Radio
Spend
Ra
tin
gs
Science Behind Marketing ROI – Optimization
Imp
res
sio
ns
Time
Optimal Media Mix
Optimal Flighting
Planners leverage model output and their insights to adjust
and optimize marketing plans per business constraints
Case Study Overview
• How effective is our current marketing spend?
• Which shows should get more marketing dollars?
• Which channels are the most effective? Most efficient?
• Based on current practices, where are we over-saturated?
A television network is seeking decision science
support to improve return on investment for the
marketing of primetime television shows
Case Study Challenges
• Data is warehoused in multiple systems, with few connection points
• Impression-level data is extremely difficult to capture, with actualized
data existing in combinations of spreadsheets, e-mails, and faxes
• Given the state of the data, common reports can take days to generate
Limited data availability prevents the network from getting
accurate measures of performance for marketing efforts
Previous attempts to answer these questions have yielded
valuable insights, but have not created sustained changes
• Avoid the temptation to answer all questions with a single model
• Ensure inputs into the solution are readily available and cost effective
• Avoid bundling decisions that are controlled by separate teams
Disney and SAS® Partnership
Project Management 15% 15%
Data Management 30% 15%
Science Integration 30% 30%
Tool Development 25% 40%
Project Timeline
Established a separate timeline for each work stream, inclusive
of milestone and reports out to key stakeholders
Data Collection Overview
• Identified over 30 potential data sources and almost 250 variables
• Data sources ranged from databases, spreadsheets, e-mails, and faxes
• Established weekly meetings with key stakeholders and implemented
dashboards to review data collection progress
• Placed an analyst in the media agency office for four weeks to speed
data collection and improve understanding of the data
Data collection ultimately took four times longer than
originally planned, due in large part to data quality issues
Data collection is never really over—continue to find errors
or missed opportunities even months later!
Data Collection Challenges
Model database changed 17 times during a 1-year span,
most often due to missing data or data collection errors
Bad circulation
estimate for
Entertainment
Weekly
Misclassified OOH
support as Events
“Week 53”
Issue
Nielsen P3
vs. C3
Duplication from
SQL Errors
Magazine Cume based
on all publications
instead of purchased
Data Visualization
Showing clients the relationship between impressions and costs helped
to identify likely errors in the data (e.g., misclassification of spending)
Data Visualization (cont.)
Exploring flights enabled us to recognize the need to model
certain media types differently than others
15% 70% 15%
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
S
S M
S M T
S M T W
S M T W R
S M T W R F
S M T W R F S
Episode
Air Date
Promos in
Calendar Week
Promos in
Past 7 Days
Transform to a full week
Data Transformation
Often necessary to transform the data for measurement
variables in our models to avoid creating misleading
insights or recommendations
M T W R F S
T W R F S
W R F S
R F S
F S
S
S
S M
S M T
S M T W
S M T W R
S M T W R F
S M T W R F S
Program Name
Air Date
Start Time
Duration
Program Type
Program Rating
Lead-in Rating
Competition
Survey Respondents
Aware Respondents
% AwareUnaided & Aided
Intent to ViewTop Box, Top 2 Box, Non Committed, Bottom Box
On-Air PromosTRPs, Seconds, # of Spots
Digital Impressions & Clicks
Cinema Impressions, Seconds Per Spot
National Cable TRPs
Newspaper Impressions & Circulation
Magazine Total & Weekly Impressions
Spot CableTRPs & Impressions
Spot Radio TRPs
OOH Impressions
NIELSEN AWARENESSPROMOS & MARKETING
Data Handoff to Science
Key milestone was the go/no-go decision on beginning
the development of the measurement model
Network Radio
Synergy Cable
Synergy Online
Emails & Newsletters
Public Relations
Affiliate Promotions
On-Air PromosDay-of-Week, Promo Length
NielsenReach, Share, HUT, PUT
PrintSize, Placement, Inserts
National CableChannel, # of Spots, Promo Length
Spot Cable & Radio# of Spots, Seconds of Promo
OOH# of Units, Size, Media Form
DigitalSize, Placement, Pillar
Social MediaFacebook, Twitter, Blog Mentions
Geo-Panel DataLocal Market Ratings and Marketing
On-Air Promo PrecisionMinute-by-Minute Ratings
EfficiencyCosts for Marketing & Promotions
On-Air Promos
Digital Impressions
MISSING DATA MISSING COMPLEXITY
MODEL EXPANSION
DATA RECONCILIATION
Data Handoff to Science (cont.)
Future iterations of the model will incorporate new data that is either
unavailable right now or represents a higher level of complexity
Science Integration
Critical to integrate science team with tool
developers to ensure alignment with the
expected input and outputs of the models
Integration between the team managing
data collection and model development
is critical to the success of the project
When it doesn’t work well—each revision of
the data model would delay the science
timeline by 3 weeks!
Science Data
Science Tool
Overview of Planning & Optimization Tool
The tool is designed to become self-sustaining to support updates
to the measurement model and to allow media plan comparisons
Data Model
Agency Media Plans
Recommended Media Plans
Measurement Model
Optimization Model Goals &
Constraints
Actualized Media Plans
HistoricalData
(one time)
ApprovedMedia Plans
Model Adjustments
Optimization Goals
Objective is to maximize total ratings for the premiere episodes
of all shows within a marketing campaign portfolio
• Provide recommended spending by channel for each show/week
combination
• Allow users to input constraints on total spending by show/channel/week
• Define spend thresholds that reflect minimum purchase amounts for
each channel
• Compare optimal recommendations against manually created plans
Critical to understand relationship between spend and
impressions; some channels have a significant delay between
purchase and delivery!
Evaluating Media Plans
Ability to compare different plans by measuring the number of
new households generated for each incremental unit of spend
Week Cable Radio Print Outdoor Cinema
t = -5 20 N/A 20 20 20t = -4 20 N/A 20 20 20t = -3 20 N/A 20 20 20t = -2 20 20 20 20 20t = -1 20 20 20 20 20t = 0 20 20 20 20 20
Recommended Plan:
Media Agency Plan: Week Cable Radio Print Outdoor Cinema
t = -5 70 N/A 80 110 10
t = -4 105 N/A 5 170 15
t = -3 160 N/A 5 5 30
t = -2 240 5 5 10 80
t = -1 355 75 25 30 125
t = 0 25 10 50 50 150
(balanced by optimization)
1 2
3
4
5
(incremental opportunities)
Key Lessons Learned
Creating Clear Requirements
Having a Test Environment
Designing a Structured QA
Process & Team
“Shadow” Implementation
Questions and Answers