Measure What Matters: Monitoring and Measurement

in the New Media World

PRSA NCC Katie Delahaye Paine CEO Paine Publishing March 19, 2015

www.painepublishing.com

@queenofmetrics

measurementqueen@gmail.com

Paine Publishing: Providing communications’

professionals the knowledge and information they need to navigate their path to perfect measurement

Newsletters

Training Courses

Consulting painepublishing.com

Katie Delahaye Paine: Helping communications professionals define and measure success for 25 years.

Founder of:

The Delahaye Group

KDPaine & Partners

Paine Publishing

Author of:

Measuring the Networked Non-Profit

Measure What Matters

Measuring Public Relationships measurementqueen@gmail.com

Don’t Measure What’s Easy

3

Cartoon by Rob Cottingham

Measure What Matters

20/03/2015 News Group International 4

Cartoon by Rob Cottingham

Measuring Attribution in an era of all media everywhere

Web Analytics

eCommerce

iCloud &

Google+

Map My Run/Social Sharing

CRM/Convio

Conversations

$$$$$$$$$$$$$$$$$$$$$$$$$$

Savings, shorter cycles, more

renewals, better ideas,

Product

Support HR IR Sales CI R&D Mktg

Research

Prod.

Mktg

PR//Mkt

7

Social Media isn’t media, it’s your business

3/20/2015

We need a new Attribution Model

ROI

Other Paid

Marketing

Digital/On-line

Media Buy

Print/TV Media

Buy

ROI

Media Relations

/Social Media/PR

Digital/ Online Media

Buy

Print/TV

Media Buy

Media

Relations/

Social

Media/PR

What’s Changed?

Collapse of mass media

Growth of media

everywhere

Intolerance for messaging

The Barcelona Principles

9

“Viewers are more likely to stop watching commercials at the moment in which brand logos appear on the screen” - ARF Study

The Barcelona Principles, The Conclave &

Industry Standards

1. Importance of Goal Setting and Measurement

2. Measuring the Effect on Outcomes is Preferred to Measuring Outputs

3. The Effect on Business Results Can and Should Be Measured Where Possible

4. Media Measurement Requires Quantity and Quality

5. Earned Media Value/AVEs are not the value of Public Relations

6. Social Media Can and Should be Measured

7. Transparency and Replicability are Paramount to Sound Measurement

All standards are available

on

http://www.painepublishing.c

om/standards-central

The Conclave

Social Media Measurement Standards have been published for:

Content Sourcing & Methods 1

Reach and Impressions 2

Engagement 3

Influence & Relevance 4

Opinion & Advocacy 5

Impact & Value 6

3/20/2015

The Transparency Table:

Eyeball counting

HITS Outcomes

MSM Online Social Media

Impressions are not awareness. Where’s the “So What”?

13

Likes Are Not Engagement

14

Advocacy Commitment Trial/Consideration

Followers Likes Impressions

3/20/2015

• Influence happens when you are persuaded to

change behavior or opinion that would

otherwise not have changed.

• Must include:

• Reach

• Engagement around individual

• Relevance to topic

• Frequency of posts around the topic

• Audience impact as measured by the

ability to get the target audience to change

behavior or opinion.

• If an individual scores a 0 on one element, they

have no influence

• Influence cannot be expressed in a single score

or algorithm

#4: Influence & Relevance

#5 Standards for Measuring Tone/Sentiment

Sentiment is overrated

Messaging and Positioning &

Recommendations are more useful

Whatever process is defined and

applied, it must be used

consistently throughout any

analysis.

Validity testing is critical

Comparison of Tonality Between Vendors

33.12%

58.00%

7.84%

11.95%

26.53%

13.00%

0.32%

9.46%

40.35%

30.00%

92.11%

78.00%

SDL

Beyond

NetBase

UberVu

% positive %negative %neutral

Discussions of nature, specifically the celebrity’s Twitter fundraising efforts were

most successful in conveying positive relationships concepts

• Impact and value will always be

dependent on client objectives

• Need to define outcomes in advance

– will likely span multiple business

goals, especially for social (crosses

disciplines)

• “ROI” should be strictly limited to

measurable financial impact;

#6: Impact & Value

Type I love Zappos into Google,

and you find 1.19 million

references

Type Citibank and you get 21,000

references. Citibank spends 100

times more a year on advertising

than Zappos.

Cost per delegate acquired:

Obama: $6,024

Clinton: $147,058

Romney: $2,389,464

ROI

ROI is:

Cost savings

Greater efficiency

Lower legal costs

Faster time to implementation

Lower cost per impact vs. other

marketing channels

Lower cost per message

exposure vs. other channels

ROI is not:

Impressions

Earned media value or AVE

Number of shares, views or

likes

Custom Standards Standards for Media Reporting

Web Analytics

5

ACA Measurement Program Parameters

Goal: Move to Industry Standards Metrics and away from media value and ad value equivalency

Measure quality not just quantity

Began in January 2013

Established a benchmark: January-March 2013

Customized “Quality Score” that indexes message content, tone, and media importance on a scale of +10 to -10

Correlated with web traffic to measure outcomes

Correlated with longitudinal research to determine impact on preference and intent to visit

Data used continuously in strategic decision making

Definitions of “Success” Workshop defined the criteria

All criteria linked back to the goals:

High Quality Media Coverage

Intent to visit

Improved reputation

Visits/Non-gaming revenue

Reality: Negative Coverage of AC is Constant and Growing

0

100

200

300

400

500

600

700

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

Tone of Coverage about Atlantic City Over Time

Negative Neutral Positive

Miss America

Solution: ACA Programs generate predominantly positive

coverage

0

20

40

60

80

100

120

140

160

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

Sentiment of ACA Coverage Over Time

Negative Neutral Positive

Without ACA events, OCS Scores for Atlantic City would

have been significantly lower

4

3.25

2.75 2.99

3.65

2.96 3.36 3.24

2.34 2.37 2.43

1.30

-1.24

0.37 -0.05

0.28 0.28

-1.56

4.91

3.92

2.99

3.58

4.14 4.1 4.27 4.12 4.29

2.78 2.56

1.53

-0.29

0.61 0.20

1.44 1.77

0.63

-2

-1

0

1

2

3

4

5

6

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June

Ave

rage

OC

S Sc

ore

Average OCS Score Over Time

Atlantic City OCS without ACA Atlantic City OCS with ACA

The red line represents coverage of Atlantic City minus all mentions of ACA and its programs

ACA has made a demonstrable difference in the quality of

media coverage

If an ACA or one of its

programs was mentioned

in a media story, it was:

More likely to contain

an endorsement

More likely to be

positive

More likely to contain

key messages

Less likely to be

negative

0.22

0.00

0.62

1.42

4.61

0.08

0.08

0.01

-0.61

-0.66

Endorsements

Positive Visual

Negative Visual

Tone

MessageCommunication

Average of OCS

Differences in Quality of Coverage with and without

ACA

Atlantic City Atlantic City Alliance

OCS scores for Atlantic City significantly improved thanks

to ACA events

36

278

458 334

537 432

342 225

995

329 324 247

139

419

260

476 538

925

4.89

3.92

2.99

3.58

4.14 4.10 4.27 4.12

4.29

2.78 2.56

1.55

-0.29

0.61

0.20

1.44 1.77

0.63

-1.00

0.00

1.00

2.00

3.00

4.00

5.00

6.00

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June

0

200

400

600

800

1000

1200

OC

S Sc

ore

Nu

mb

er o

f it

ems

AC Mentions vs. OCS Over Time

AC Mentions AC OCS

Dead whale, non-gaming promotions, Caesar’s CEO quote,

Lady Antibellum

Revel Casino value, Electric

Jitney test, Fake Chips, Phil Ivey suit, Blake

Shelton

Showboat closing, Revel facing 2nd

bankruptcy

Communication of the "Making a Comeback" & "Something for Everyone" messages increased as ACA announced new programs

14 39

23 36

133

235

8 7 4 2 10 15 8 11 1 2 9 2 24

70

21

73 88

110

13

104 74

138

87

48

Jan Feb Mar Apr May Jun

Qtr1 Qtr2

2014

Atlantic City has something for everyone Atlantic City is a clean and safe place to visit

Atlantic City is a year-round destination spot Atlantic City is making a comeback

Atlantic City offers more than just gaming

Free entertainment generated the highest OCS Scores in

Q2

5.26 5.73

6.36 6.54

6.71 7.15

7.31 7.36

7.50 7.67

8.00 8.10

8.68 9.30

Miss AmericaMeet AC

DO ACJuly 4th Fireworks

Air ShowBlake Shelton

Miss'd America PageantSand Blast

Hello SummerBoardwalk Hall Light Show

Challenge TriathlonSand Sculpting World CupLady Antebellum Concert

Free Entertainment

Top Programs By OCS

When ACA programs received media coverage, traffic

followed

0

50

100

150

200

250

300

0

20000

40000

60000

80000

100000

120000

140000

Relationship between ACA Program Mentions and Site Visits

Site Visits Program Mentions

ACA programs drive higher OCS scores, which correlate

highly with web visits

0.41

0.44

0.47

AC items

ACA Items

ACA OCS Scores

Correlations between Web Visits and PR Metrics

Pearson r. value

PR is having a positive impact on preference and

perceptions

People who say they have recently heard news about Atlantic

City are very likely to perceive AC as fun – a key driver of

preference

Respondents who remember seeing news reports about

Atlantic City are also very likely to associate key messages

and positioning statements such as “place I am excited to go

to” and “fun place to hang out with friends.”

The same respondents are also more likely to recommend

Atlantic City to friends.

6 steps to standards-compliant

measurement

Step 1: Define your goal(s).

What outcomes is this strategy or tactic going to achieve?

What are your measurable objectives?

Step 2: Define the parameters

Who are you are trying to reach? How do your efforts

connect with those audiences to achieve the goal.

Step 3: Define your benchmarks.

Who or what are you going to compare your results to?

Step 4: Define your metrics.

What are the indicators to judge your progress?

Step 5: Select your data collection tool(s).

Step 6: Analyze your data.

Turn it into action, measure again

33

Six Steps

to Success

1

2

3

4

5

6

Definitions

Monitoring – process by which data are systematically and regularly collected about a program over time.

Measurement – a way of giving an activity a precise dimension, generally by comparison to some standard; usually done in a quantifiable or numerical manner; see also: data, scale

Metrics – an agreed upon number or percentage that indicates progress towards a goal

Evaluation –a form of research that determines the relative effectiveness of a public relations campaign or program by measuring program*

34

*IPR Dictionary of Measurement Terms, Dr. Donald Stacks www..instituteforpr.org

Why do we communicate?

3/20/2015 35

Outtakes

(Intermediary Effects)

• Awareness

• Knowledge/Education

• Understanding

Outcomes

(Target Audience Action)

• Revenue

• Leads

• Engagement

• Advocacy

Activities How does what you do

contribute to the bottom line?

Step 1: Define the goals

What return is expected? – Define in terms of the

mission.

Define your champagne moment. If you are

celebrating complete 100% success a year from

now, what is different about the organization?

36

Goals & Suggested Metrics

Goals Goal 1: Meet Sales Targets

Goal 2: Reduce Risk/Threats

Goal 3: Increase market share in

new market

Communications Contribution

• Expand the marketable universe

• Reduce sales cycle

• Increase Trust • Increase

advocacy

• Expand the marketable

universe

Metrics

• % Increase in desirable share of voice

• % in awareness

• % decrease in undesirable voice

• % increase in trust scores

• % increase in desirable voice in new market

• % increase in awareness of brand in new market

Step 2: Understand the parameters.

What management’s priorities?

Who are you are trying to reach?

How do your efforts connect with

those audiences to achieve the

goal?

What influences their decisions?

What’s important to them?

What makes them act?

38

Goal: Get the cat to stop howling

Options: Local? Cheap? Convenient?

Strategy: Buy cat food

Step 3: Establish benchmarks

Past Performance Over Time

Think 3

Whatever keeps your

C-suite up at night

39

Step 4: Pick your Kick-Butt Index

The Perfect KBI

Is actionable

Is there when you need it

Continuously improves your processes

& gets you where you want to go

You become what you measure, so

pick your KBI carefully

40

Criteria Result Score Result Score

Tonality Favorable +3 Unfavorable -3

Messaging Yes +3 No -1

Quotes Yes +1 No -2

Dominance Dominant +1

Not Dominant

-or-

Unfavorable &

Dominant -1

Visibility High Visibility +2

Low Visibility

-or-

Unfavorable High

Visibility -3

Final Score +10 -10

How to calculate your Kick Butt Index

41

Step 5: Pick the right measurement tools

If you want to measure messaging, positioning,

themes, sentiment:

Content analysis

If you want to measure awareness, perception,

relationships, preference:

Survey research

If you want to measure engagement, action,

purchase:

Web analytics

If you want predictions and correlations

you need two out of three

Step 5: Selecting a measurement tool

43

Objective KBI Tool

Increase message

communications

Increase percent of items containing one or

more messages

CyberAlert, Prime Research

Increase

awareness/preference

% of audience preferring your brand

to the competition

Survey Monkey, Survata

Engage

marketplace

% increase in engagement on website and/or

social sites

Simply Measured, Unmetric,

Google Analytics, Site Catalyst,

Network Analysis

Become a thought

leader on a specific

issue/market/topic

% increase in desirable share of voice on

issue/market/top

Little Bird, Trackr, CyberAlert

Testing the accuracy of coding

48.94%

68.31%

88.64%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

SDL

UberVu

NetBase

% Agreement with human coding

SDL UberVu NetBase100.00%

62.32%

14.82%

72.36%

0.00%

37.68%

85.18%

27.64%

Mixed

Negative

Neutral

Positive

Degree to which agreement occurred

No Yes

3

Testing validity of criteria

Biggest Areas of Discrepancies

7

Geographies vs Languages

Lack of consistent competitors list

Different definitions of Products/Brands Covered

Time frames

Media Types

Search strings and filters

Step 6: Be Data Informed, not Data Driven

Rank order results from worst to best

Ask “So What?” at least three times

Put your data into an overall framework consistent with C-Suite expectations

Find your “Data Geek”

Compare to last month, last quarter, 13-month average

48 Page 48

Photo Event

High Message Content

Resource Use

Low

Hig

h

Med

ium

Ver

y h

igh

Level of Engagement

Ver

y H

igh

Med

ium

Hig

h

Low

High Resources

No Message

Low Resources

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

Advocacy vs. resource use

Resource Use

Low

Hig

h

Med

ium

Ver

y h

igh

Ver

y H

igh

Med

ium

Hig

h

Low

High Resources Low Resources

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

Charting KBI over time between divisions

-100%

0%

100%

200%

300%

400%

500%

Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul

2007 2008

% D

iffer

ence

Kick Butt Score Relative to other LOBs The Percent Difference Between Each Business Unit's Average KBI and the

KBI for each Business Unit

SAS

IDS

IIS

MSDIB

FBX-T

ALR-67(V)3

DDG-1000 MSE

APG-79, APG63

AESA

for F-15E,

VIIRS delays

Glory APS and VIIRS vs.

competitors' EPX

ASAT, Patriot

ERGM cancellation

Patriot (Korea)

RISS, GBS

MALD, AMRAAM

APG-63 (v3)

NPOESS; BOE B-52 jammer

RIS JPL

50

Connecting data yield greater insights

106 38 24

73 101 49 25

1087

1331

699 637

848

1331

699

10.25

35.03

29.13

8.73 8.40

27.16 27.96

0

5

10

15

20

25

30

35

40

0

200

400

600

800

1000

1200

1400

Jan Feb Mar April May June July

Relationship between interviews and placements

Number of Interviews Number of placements Interview per placement

968 1027

914

208

297 261 269

215 222 145

328

204

510

188 222 234

41 71 50 36 25 56 30 71 45

107

Share & Likes Over Time

Average of PostLikes Average of PShares

Not all engagement is equal

Mia Farrow

David Beckham & Dierks Bentley

Video

White House Volunteerism Office (CNCS) is able to connect

specific social outreach to registrations on serve.gov

53

ReTweets compared to Visitors to Serve.gov

MLK saw the highest correlation of coverage to

visits

0.133

0.156

0.355

0.379

0.427

0.848

Learn & Serve

SeniorCorps

CNCS

AmeriCorps

Get Involved

MLK

10 Ways to find the Money for Measurement

1. Don’t call it measurement – It’s Research

2. Consolidate - Social? IR? Agencies? Research?

3. Crowdsource – Who else has tools or surveys?

4. Spread the cost over 2 fiscal years

5. Reach out to universities

6. Take advantage of free stuff

7. Monitor only what matters

8. Narrow your Top Tier media list to what you CAN afford

9. Google analytics

10. Facebook Insights

11. Twitter Analytics

Remember These Points

It’s not about the media, it’s about the business and the customer 1

It’s not about Big Data, but about how you use it. 2

You need to be data informed, not data-driven 3

It’s not how loud you’re shouting it’s about relationships. 4

Standards are a reality not an excuse to hide behind 5

Thank You!

For more information on measurement, read my blog:

http://www.painepublishing.com/blog

For a copy of this presentation or to subscribe to our newsletter, give me

your card or email me at measurementqueen@gmail.com

Follow me on Twitter: @queenofmetrics

Friend me on Facebook: Katie Paine

Or call me at 1-603-682-0735

57