Social Media Measurement 101 + Day of aTONEment

SOCIAL MEDIASOCIAL MEDIA MEASUREMENT 101A Day of aTONEmentA Day of aTONEment

MRA ANNUAL CONFERENCE

JUNE 6, 2011

Alan Chumley

Senior Vice President

@ l h l / @CARMA T t@alanchumley / @CARMA_Tweets

1615 M Street, NW; Suite 750 Washington, DC 20036 USA Tel 1.202.842.1818carma.com

ALAN CHUMLEY, SVP

• 14 years in the industry

R i PR titi ING B ll C d• Recovering PR practitioner: ING, Bell Canada

• Former Director of Measurement @ Hill & Knowlton

• Masters in Communications & Culture:

• Research foci:

• media effects & uses

• reception analysisreception analysis

• audience studies

• University‐level adjunct instructor of PR measurement

Li k dI l 2006• LinkedIn early 2006

• Blogging / Tweeting since early 2007

1615 M Street, NW; Suite 750 Washington, DC 20036 USA Tel 1.202.842.1818carma.com 2

@alanchumley / #mra_ac

ABOUT CARMA

• Traditional & social

M di it i & l i• Media monitoring & analysis

• Largely human‐based….on an automation journey

• In biz since 1984

• Head office here in DC

• 14 offices in 11 countries14 offices in 11 countries

• Clients:

PR f lk i bi i ti• PR folk in big organizations

• Market research folks in big organizations are starting to emerge



PART 1 (1.5 hrs): SOCIAL MEDIA MEASUREMENT

IN-ROOM STRETCH (5 mins)

PART 2 (30 i ) TONE ALONEPART 2 (30 mins): TONE ALONE

PART 3: (35 mins): REAL DATA WITH MALCOM DE LEO NETBASE




ISN’T

• Not a Case Study

• Not a mention of Old Spice in sight

• Not a specific ‘how to’ / no one-size-fits all

• Doesn’t focus on a singular or few social media platforms—looks at all holistically



ISN’T IS

• An emerging / experimental philosophy / approach / filter / lens through which to look at measuring social media

• Not a Case Study

• Not a mention of Old Spice in sight

• A high level connected / systems way of approaching the problem

b d

• Not a specific ‘how to’ / no one-size-fits all

• About communities not an audience

• Beyond simple counts and sentiment

• Doesn’t focus on a singular or few social media platforms—looks at all holistically

• More sociologist than publicist

Influence potential


• Influence potential


Every hour more than 500,000 new blog + micro blog posts status updatesblog + micro-blog posts, status updates and comments are posted


MEASUREMENT ACCOUNTS FOR 5 / 7 TOP SOCIAL MEDIA CHALLENGES

Why is this the case?Gap between leap ofGap between leap of faith true believers and C-suite’s need for metrics to validate the

$

Source: Harvard Business Review: the New Conversation: Taking Social Media from Talk to Action

Surveyed 2100 Harvard Biz Review subscribers

time/$?



SOCIAL MEDIA IS A GAME CHANGER… …& A SOURCE OF ANXIETY

Top Down Command & Control

of the Message is Gone

• Transparency demanded

• Dialogue vs. Monologue

• Many-to-many not one-to-many

• Conversation & community NOT (strictly) coverageNOT (strictly) coverage

• About relationships

• Relationship trust is co-created by the organization and its communities of interest via conversation






There is no Passive Ready-to-Consume Audience’



• Prosumers not just consumers

• Actively engaged in creating and re-creating conversations and


• Conversation & community NOT (strictly) coverage

gcommunities

• Challenge in tracking & measuring something as nebulous as a communityNOT (strictly) coverage


• Relationship trust is co-

nebulous as a community

created by the organization and its communities of interest via conversation






There is no Passive Ready-to-Consumer ‘Audience’

Disciplinary Lines are Eroding





• PA, PR, Marketing, CRM, Customer Service etc.

• Customer touch points



gcommunities

• Challenge in tracking & measuring something as nebulous as a community

• Integrated cross-discipline re: strategy

• Integrate cross-discipline NOT (strictly) coverage



nebulous as a community re: measurement

created by the organization and its communities of interest via conversation






There is no Passive Ready-to-Consumer ‘Audience’

Disciplinary Lines are Eroding





• PA, PR, Marketing, CRM, Customer Service etc.

• Customer touch points



gcommunities

• Challenge in tracking & measuring something as nebulous as a community

• Integrated cross-discipline re: strategy

• Integrate cross-discipline NOT (strictly) coverage



nebulous as a community re: measurement

C Suite is Intrigued but Anxious / Want Data to:created by the organization and its communities of interest via conversation

C-Suite is Intrigued but Anxious / Want Data to:• Show why and how they should…• Demonstrate the value of having done so…



SIMPLE STATUS QUO

Counts:

Quick Quantity counts:•Fans Likes Follows Etc•Fans, Likes, Follows, Etc.•Tantamount to reach in the offline or 1.0 world•Reach does not equal Influence

Quality:•Quick automated read on sentiment: but concerns re: accuracy

Rankings:

•Visibility, popularity, traffic, link loveVisibility, popularity, traffic, link love•Technorati for blogs•NOT RECOMMENDED: Twitrratr, Klout, •BETTER: Twendz, Traackr, Twittalyzer, Twinfluence

The rarel q alif people or content (the message)•They rarely qualify people or content (the message)

•The message AND the medium are the measure



LOT’S OF TOOLS OUT THERE

BEWARE OF THE SILOs AVOID USING (ONLY) THE NEW SHINEY FREEBIES

•Isolated / silo’d approach•No one tool on it’s own is sufficient not even one paid tool•No one tool on it s own is sufficient—not even one paid tool•Have to connect the dots with a more relational, systems approach

Would you like your job performance (read: raise/bonus) to be based only on a


(read: raise/bonus) to be based only on a Klout Score, for example?


REACH SHOULD BE A RELATIVE NOT AN ABSOLUTE& NOT ALL CONVERSATIONS ARE RELEVANT

Off topic / Irrelevant

R l t “ ”R l tRelevant “+” Relevant “-”Relevant Neutral



REMEMBER: IT’S SOCIAL MEDIA

•It’s not coverage

•It’s not staticIt s not static

•There isn’t an ‘audience’ of consumers

•Prosumers not just consumers:•Prosumers not just consumers: •simultaneously receiving, interpreting, co-opting, re-appropriating and creating content

It’ t di l th t l•It’s more many-to-many dialogue than one-to-many monologue

•So, traditional off line ways of measuring ‘coverage’ don’t cut it

•They don’t adequately account for the multi-dimensionality and dynamic nature of a conversation and a community of interest in a network

•We need to think bigger


gg


THINK BIGGER / INTEGRATE / CORRELATE

•Think Bigger / Broader Beyond PR’s Historical Silo –social Media’s Pushing us all that Way •Think about all customer‐facing touch points•Understand PR / Marketing / Business Metrics •Integrate Cross‐Disciplineg p•Report on the Critical Few•Focus on /correlate with Outcomes•Close the loop / connect the dots

S W b Sh d i k M t & St t P ti S Oli Bl h d S i l M di ROI


Source: Weber-Shandwick Measurement & Strategy Practice Source: Oliver Blanchard, Social Media ROI

18


THE MEASUREMENT MASHUP



THINK BIGGER & BROADER MOVE BEYOND ‘MOMENT-IN-TIME’ or ‘VANITY’ METRICS

S t N t k dSystems or Networked Thinking:

Interdependent relationships between pthose in a network

Social Capital:

Connections within and between those in a social network as a source of civic engagement among


a community leading to cohesion; that has ‘value’


FOCUSING ON CONTENT ANALYSIS AND NETWORK ANALYSISAS A WAY OF MEASURING INFLUENCE POTENTIAL

S t N t k dSystems or Networked Thinking:

Interdependent relationships between pthose in a network

Social Capital:

Connections within and between those in a social network as a source of civic engagement among


a community leading to cohesion; that has ‘value’


The 7Cs THINKING MORE LIKE A SOCIOLOGIST THAN PUBLICIST

Counting What some call search, syndication and site; stuff we can count like traffic, popularity, fans, followers, retweets, etc.

Content Analysis that is. Using content analysis, ideally a hybrid of some automation for volume and speed and Content y g y , y y phuman analysts for context, nuance and depth to look for how your brand, messaging, topics, issues, stakeholder voices are portrayed; the inclusiveness, accuracy and tone of the conversations.

Conversations And conversationships recognizing that social media is not strictly another communications channel through which to get word out or reach and influence. It is more about genuinely engaged dialogue than a simple corporate monologue; not only a vertical, one‐to‐many conversation between anthan a simple corporate monologue; not only a vertical, one to many conversation between an organization and stakeholders but a lateral, many‐to‐many conversation among stakeholders.

Cohesion The extent to which folks are agreeing with you, your position, your brand essences. The extent to which they are agreeing with each other and coalescing around and advocating a core theme, idea, or some call to action either in support or opposition to your organization’s positionsome call to action either in support or opposition to your organization s position.

Community The extent to which a core group of people with common interests are gathering and growing. Measuring the change in community size, and volume and nature of chatter over time

Connectedness A look at how inter‐connected the highly engaged/key influencers/advocates are and how centrally Connectednesslocated they are in both the network and dialogue. (See Appendix for the 5Ps of influence)

Conversion Stay tuned. More to follow.



THE 5Ps OF INFLUENCE

Popular Visible, vocal, has a substantial following, reach. In‐bound links, trackbacks, subscribers, bookmarks, followers, friends, views, listens, saves, downloads, etc.

Polarized in tone Neutrality does little to drive influence way or the other. A clearly positive or negative view will polarize readers/followers and is more likely to drive cohesion and mobilize advocates and have those advocates coalesce around a core theme, idea, or call to action.

Prolific / Relevant / Frequent

Raw author contribution and # of on‐topic, related posts

Prominent / Are they an idea starter or spreader; source or spider? They may be prolific but are they prominent? Are they highly inter related inter connected and centrally located in the

Authoritativeprominent? Are they highly inter‐related, inter‐connected, and centrally located in the network? How engaged is this person’s following in a dialogue? How much dialogue is there and what is its nature? Here we need to recognize, though, that authority is contextual and topical. One might be an authority on PR measurement but not on 18th century Russian literature.

/ l H f th f ll / t t ti t ib t d ti d iPromoter / Loyal Advocate

How many of the followers/commentators active contributors advocating, endorsing, advancing (or the opposite) your position? Are they adding links, tags. Is the nature of the language they are using inter‐connective, expanded, clarifying, reinterpreting? RTs, digs, fans, votes, buzz‐ups, up/downloads, shares, likes, invites, favorites, embeds. (More active than the metrics in popularity)



VARIETY OF OBJECTIVES & CAMPAIGN TYPES (NON-EXHAUSTIVE)

Objectives

• Visibility

Campaign Types

• Visibility• Visits• Volume• Velocity

• Push (get bloggers to review / write)• Pull (drive visitors on‐line)• CSR• Product Launch

• Awareness• Message delivery• Link love• Dialogue vs Monologue

• Product Launch• Product Engagement• Marketing• Blogger Outreach

• Dialogue vs. Monologue• Generate dialogue / comments• Feedback• Using, sharing, pass‐a‐long

• Advocacy• Issues Management• Community building• Customer relations

• Embedding of interactive content• Diggs, Delicious tags, stumbles, etc.• Votes• Etc., etc., etc.

Customer relations• Thought leadership • Executive profiling• Etc., etc., etc.


Etc., etc., etc.

24


SOME THINKING ON AN APPROACHAN INDEX IS ONLY PART OF THE ANSWER



COMPONENTS OF AN INDEX ?

Popularity + Relevance + Authority + Presence + Engagement + Advocacy/Loyalty + Cascade

x (+/‐) Sentiment

=

Influence Potential

• Based on an assumption that the components above are drivers of on-line influence, potential action

• They can be measured individually, but it’s important to measure them collectively as they are inter-related

• The index is intended as a measure of potential to influence not actual real influence that is another• The index is intended as a measure of potential to influence not actual real influence—that is another research project all together.

• All components are important (but not equal) drivers of potential influence on-line


p p ( q ) p

26


INDICATORS DEFINED

• Cascade• Citation analysis / ripple index• Network analysis

V l & l i f d h

•Authority•Of both poster and commentators•Idea starters or spreaders?•Degree of inter connectedness / inter relatedness

High authority

low relevance

High authority AND

relevance• Volume & velocity of message spread across the network

• Inter‐connectedness / inter‐relationships among:• all• Only the influentials

• Advocacy (active)

•Degree of inter‐connectedness / inter‐relatedness•Connected but respected?•How central is the influencer to the network?•Following: follower•RTs/1000 followers

• Rele ance

relevance relevance

Low authority AND

relevance

High relevance

low authority

y ( )• Are social media users advancing our cause?• % containing links• % active contributors• Tags added

• Engagement (more coming)

• Relevance• On topic?• Author contribution: how often / prolific?• Topical / textual / temporary

S ti t ( i )Engagement (more coming)• Dialogue? 2‐way? How much? How good? • How many commentators? Repeats? Nature of

the language• Network analysis: inter‐connected/related

• Entity Presence

• Sentiment (more coming)• Multi‐level, multi‐author tone• Post‐level, theme level, sentence level• Gaps in tone in originating content and resulting

discussion

Entity Presence• Present? Prominent? Frequent?• # of + mentions• Competitive share of original post• Competitive share of resulting discussion• Attribution recognition

• Popularity (passive) • ‘Reach’ ?• Voice of the consumer stuff we can count• Links, votes, shares, recommends, subscribers,

registrations, tweets, RTs, followers, fans, friends, diggs shares likes fav’s upvotes buzzups


diggs, shares, likes, fav s, upvotes, buzzups, embeds, tags, wish list saves, forwards, invites, etc.

27

@alanchumley / @CARMA_Tweets

NETWORK ANALYSIS

How are these changing over time?

Source: K. Ognyanova, UCLA, 2010

Might be part of many systems / networks?

Linking networks to networks


@alanchumley / @CARMA_Tweets

(A QUICK ASIDE ON) SENTIMENT AS MARKET RESEARCH PROXY / PREDICTOR

Think here about social media monitoring and analysis as:

•Real time situational awareness•Real time situational awareness•Listening at the point of need•Listening close to the moment of influence

A real-time digital form of or proxy for market research and CRM:

Mining conversations to tease out how people ‘feel’ about products/services/issues/causes and perhaps even:and perhaps even:

1) their intended (reported) behavior and what motivates them to buy, (as cited in the conversations), and

2) customer satisfaction

3)their actual behavior tracked via CRM 2.0 and Web 2.0 analytics


y

29

Source: Crimson Hexagon


BUT REMEMBER THAT OBJECTIVES DRIVE METRICS

Objectives

• Visibility

Campaign Types

• Visibility• Visits• Volume• Velocity

• Push (get bloggers to review / write)• Pull (drive visitors on‐line)• CSR• Product Launch

• Awareness• Message delivery• Link love• Dialogue vs Monologue

• Product Launch• Product Engagement• Marketing• Blogger Outreach

• Dialogue vs. Monologue• Generate dialogue / comments• Feedback• Using, sharing, pass‐a‐long

• Advocacy• Issues Management• Community building• Customer relations

• Embedding of interactive content• Diggs, Delicious tags, stumbles, etc.• Votes• Etc., etc., etc.

Customer relations• Thought leadership • Executive profiling• Etc., etc., etc.


Etc., etc., etc.

30


IMAGINE A DIFF’T. PYRAMID FOR DIFF’T. CAMPAIGN TYPES / OBJECTIVES



AN INDEX WITH MOVABLE INDICATORSFLEXIBLE ENOUGH TO ACCOUNT FOR DIFFERENT OBJECTIVES

Indicator Popularity Relevance Authority Brand Engagement Advocacy Cascade SentimentPresence

Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7

(@ d f lt)(@ default)

Popularity + Relevance + Authority + Presence + Engagement + Advocacy + Cascade X Sentiment = Influence Potential

• Each of the above indicators can be adjusted to account for different types of campaigns & contexts

• The weights would change. The scores don’t.

• For an organization, new to social media, perhaps they might only want to establish an on‐line presence

• So we’d dial or slide the weights of all but the popularity indicator down to zero

Has a significantly scaled back / rudimentary version of this. This is inspired by but significantly expands on R6 thinking.


p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g




Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g


WHAT DOES THE INDEX INDICATE?

• It’s a relative measure of the quality of social media coverage:– One post relative to another and over time – Many posts relative to a number of other similar clusters of postings

• It is a measure of both output and outtake, but not outcome– Output: how much and how good?– Outtake: how engaged, inter‐connected, inter‐related, volume & velocity of network /

community spread– Outtake: advocacy

d l b f– it does not claim to be a measure of outcome or impact

• It’s a proxy for on‐line influence; it’s about potential to influence– to measure actual influence, we’ve have to survey

• In a diagnostic benchmarking sense the index can help inform an organization’s social media• In a diagnostic, benchmarking sense the index can help inform an organization s social media engagement strategy if tracked over time

• It’s tactical on the fly for mid‐campaign course correction and looking back over the long term

• It’s strategic looking forward: helps shape communications recommendations

• This data can be correlated with tangible outcomes—CARMA Connect.

• This data can be plugged into a market mix model to show causality…isolate for PR’s unique contribution to the marketing mix.



SO WHAT?WHAT ABOUTWHAT ABOUT CONVERSION & ROI?


WE’RE BACK TO THE PYRAMID THE TOP THIS TIME



OUTCOMES: KINDS OF CONVERSIONSETTING THE STAGE FOR CORRELATION

• Conversion to a tangible like web traffic, sales, donations, awareness, opinion, usage, brand preference, likelihood to try/buy/recommend.

Micro (steps toward destination) Macro (Destination)

Usually an intermediate means to an end The end

• Could also mean conversion toward any measurable marketing communication or public affairs, issues, advocacy-based objective

Non‐transactional (Engagement) Transactional

Non‐financial Financial

Get to x clicks in a site, spend x time on Sale, donation, objective.

• Here we could / should also be looking at the extent to which the quantity and quality

, psite, download, lead generated, watch a video, created account, contacted us etc.

, ,(paid) subscription etc.

q y q yof social media activity correlates with these outputs.



2 WEEKS IN THE SOCIAL MEDIA ECOSYSTEM…THEN…YOUR WEBSITE

Where were they? What drove them to your site? What did they do once on your site? What are they ‘worth’ to you?



SOCIAL MEDIA PERFORMANCE DATA + WEB ANALYTICS

Export Data to Your Web Analytics•#of posts by date•# of posts with key message included•# of posts with key message included

•# of post with brand mentioned X or more times

•# posts X average favorability by date•# posts X average favorability X reach by datep g y y

Or

Integrate web analytics into your social media monitoring platform



LEVELS OF LINKING OUTPUT TO OUTCOMES

Level of Linkage Notes

Just graph ‘er up Plot # of articles vs. web traffic on a graph

•No indication of relationship other than lines that might trend together

•Could be a fluke

Source: Olivier Blanchard, Social Media ROI







•Could be a fluke

Correlation •“R score”S t t

•Showing there is l ti hi•Some stats

•You can do this in excel•Pearson’s Product Moment

some relationship beyond coincidence

•Not ‘proving’ media exclusively responsible for sales








•Could be a fluke







Causality •Heavy duty stats•You so can’t do this in

•Isolates (statistically

excel•Sophisticated modeling•Market Mix Modeling•Multivariate Regression Analysis

( y‘proves’) for PR’s unique contributionto the communications mix


Regression Analysis•Ton ‘o data, time, $

50


TECH INDUSTRY EXAMPLE

Correlating opinion with CARMA IQ (favorability + reach)



AUTO INDUSTRY EXAMPLE

Monthly Sales Results vs. Four Month CARMA Favorability: Monthly Sales to Four Previous Months’ Favorability



Toyota’s Heat Map: CARMA Media IQ Metrics by Various Time Frames Correlated to Monthly Sales Resultsy



UPPERCASE ‘ROI’ vs. lowercase ‘roi’ & Other Return-Offs

ROI roi

MBA PR

Financial Non financialFinancial Non‐financial

Macro Micro (not all conversion events are ROI)

A mathematical equation that works

Usually conceptual only; an equation in words only; can’t actually be calculated; something the industry tosses around too liberallythat works be calculated; something the industry tosses around too liberally

A business metric All‐too‐often a media or audience metric

Direct, clear A loose proxy or an intermediate metric (return on expectation, return on objective return on engagement)return on objective, return on engagement)




THOSE USING TXT ANALYTICS ARE MEASURING ‘ROI’ NOT ‘roi’

S T t A l ti 2009 U P ti S l ti d P idSource: Text Analytics 2009: User Perspectives on Solutions and Providers



SOCIAL MEDIA ROI CALCULATOR(S) THE GOOD & THE BAD



SOCIAL MEDIA BOOK BAG

• Social Media Metrics: • Jim Sterne, (@jimsterne)

• Social Media ROI: • Olivier Blanchard (@thebrandbuilder)

• Social Media Analytics: Effective Tools for Building, Interpreting & Using Metrics: • Marshal Sponder (@webmetricsguru)

M Wh M• Measure What Matters: • Katie Paine (@kdpaine)

• Using Web Analytics to Measure Impact of Earned Online Media…:g y p• Seth Duncan (www.instituteforpr.org)





PART 2 (30 mins): TONE ALONE

PART 3: (35 mins): REAL DATA WITH MALCOM DE LEO NETBASE


AGENDA FOR PART 2 DEEP DIVE ON SENTIMENT

•Sentiment Analysis

•Why is it important/ why should we care?

•Sentiment as a Proxy / Predictor

•Applications

•Humans or Machines or Both?







•Applications

•Humans or Machines or Both?•Why Automation?

S S•Situating Automated Sentiment Analysis

•Defining Automated Sentiment Analysis

•Output

•Taxonomies etc.

•Automated Vendors

•Complicationsp







•Applications

•Humans or Machines or Both?•Why Automation?

S S

•Human vs Automated

•Situating Automated Sentiment Analysis

•Defining Automated Sentiment Analysis

•Output•Human vs. Automated

•Human-based vendors

•Human-Based Sentiment Analysis

•Taxonomies etc.

•Automated Vendors

•Complications•Favorability not (just) Sentiment

•Humans supplemented by automation

p



Linking outputs to outcomes revisited

I AM NOT IAM

A COMPUTATIONAL LINGUISTA SEMANTICIANA SEMANTICIANCOMPUTER ENGINEERCOMPUTER SCIENTISTLOGICIANPSYCHOLINGUIST

BUT CHECK OUT SETH GRIMES’:BUT…CHECK OUT SETH GRIMES :Text Analytics 2009: User Perspectives on Solutions and ProvidersA number of excellent presentations on LinkedIn / Slide Share

I’ve borrowed liberally from both for this Part 2



I AM NOT IAM

A COMPUTATIONAL LINGUISTA SEMANTICIAN A RECOVERING PR FLACKA SEMANTICIANCOMPUTER ENGINEERCOMPUTER SCIENTISTLOGICIAN

CO G CPR MEASUREMENT MOUTHPIECEWITH AN INTEREST IN MONITORING + MINING ONLINE CONVERSATIONS

PSYCHOLINGUIST

BUT CHECK OUT SETH GRIMES’:BUT…CHECK OUT SETH GRIMES : Sentiment through the eyes of a simpletonText Analytics 2009: User Perspectives on Solutions and Providers

A number of excellent presentations on LinkedIn / Slide Share

I’ve borrowed liberally from both for this Part 2



IMAGINE A DIFF’T. PYRAMID FOR DIFF’T. CAMPAIGN TYPES / OBJECTIVES



WHY IS IT IMPORTANT? WHY SHOULD WE CARE?

Questions for biz & gov’t.:

Wh t l i ?•What are people saying?•What’s trending?•How has opinion about x changed?•Does it differ channel to channeloes d e c a e o c a e•How has sentiment correlated with news / marketing / events etc.?•Can we link opinions to transactions?

Computers are trading stock based on automated sentiment analysis of online content



SENTIMENT ANALYSIS AS MARKET RESEARCH PROXY / PREDICTOR

Think here about social media monitoring and analysis as:

•Real time situational awareness•Listening at the point of need•Listening close to the moment of influence

A real-time digital form of or proxy for market research and CRM:

Mining conversations to tease out how people ‘feel’ about products/services/issues/causesfeel about products/services/issues/causes and perhaps even:

1) their intended (reported) behavior and what motivates them to buy, (as cited in the

ti ) dconversations), and

2) their actual behavior tracked via CRM 2.0 and Web 2.0 analytics


Source: Crimson Hexagon


APPLICATIONS


Call Centre Notes?


SITUATING SENTIMENT ANALYSIS AUTOMATED

Text analytics:

•Applies linguistic +/or statistical

Artificial Intelligence

T t A l ti

techniques to extract concepts and

patterns

•Transforms unstructured information intoText Analytics •Transforms unstructured information into

data for analysis

•Unlocks meaning and relationships in

Sentiment Analysis large volumes of information

•Adds semantic understanding of:•SentimentOpinions•Opinions

•Attitudes •Entities•ConceptsFacts



•Facts•Abstract attributes: “expensive,” “comfortable”

AUTOMATED SENTIMENT ANALYSIS DEFINED

Sentiment analysis or opinion mining refers to the application of:•Natural language processing•Computation linguistics Comp ScienceLinguisticsp g•Text analytics…

…to identify and extract subjective information in content. Statistics

Natural Language Processing• Grounded in statistical machine learning• Much overlap with…

Computational LinguisticsComputational Linguistics• Statistical or rule‐based modeling of language

Text Analytics• Turning text into data• Lexical analysis: word frequencies• Entity recognition• Pattern recognition• Tagging/ annotation


Tagging/ annotation


EXTRACTING ATTITUDES TURNING ATTITUDES INTO DATA

Generally speaking, sentiment analysis aims to determine the attitude of a speaker or a writer with respect to some topic or the overall tonality of a document.

The attitude may be his or her:

• judgment or evaluation

• affective state (emotional state of the author when writing), or

• the intended emotional communication (emotional effect the author wishes to have on the reader).



HUMANS OR MACHINES OR BOTH?ALGORITHM AL GORITHM

Can it be over ridden?Can it be over ridden?Does it learn?



THE ALGORITHMICS

WHY AUTOMATION?

Time Spent Analyzing Contenty g

w/out automation

Ti STime Spent on Insights &

Implications

w/ automation



WHY AUTOMATION?

Time Spent Analyzing Contenty g

w/out automation

Ti STime Spent on Insights &

Implications

w/ automation



THE OUTPUT

E tit E t tiEntity Extraction

Issues / Themes / MessagesMessages



TAXONOMIES…LEXICONS…DICTIONARIES…BAGS-O-WORDS…CARTRIDGES

PRE‐SET • Words/phrases that are generally good or

GENERALare generally good or bad (Hate. Love)

• Caution!




PRE‐SET•Words/phrases that are generally good or badPRE SET

GENERAL

PRE‐SET

generally good or bad (Hate. Love)

•Caution!

•Financial / market newsPRE SET

SPECIFIC•Mergers & Acquisitions•Competition•Better




•Words/phrases that are generally good or bad

PRE‐SET GENERAL

PRE SET


•Caution!

•Financial / market newsPRE‐SET

SPECIFIC

CUSTOM

•Mergers & Acquisitions•Competition•Better

CUSTOM

CLIENT/PROJECT

SPECIFIC

•Client‐specific context•Best




•Words/phrases that are generally good or bad

PRE‐SET GENERAL

PRE SET


•Caution!

•Financial / market newsPRE‐SET

SPECIFIC

CUSTOM

•Mergers & Acquisitions•Competition•Better

CUSTOM

CLIENT/PROJECT

SPECIFIC

•Client‐specific context•Best•Multiple cartridges

But! Still needs some human intervention / validation /


feedback so it can ‘learn’ and improve


AUTOMATED TEXT ANALYTICS VENDORS



1st GENERATION vs. 2nd GENERATION AUTOMATED SENTIMENT

Key Word (only) Conceptual Search-Based

Deeper linguistic level Based

Explicit References

p g

Operates on a semantic level

Understands sentence grammar:References

Counting up the “+” & “-”

•Subject•Predicate•Object•Who?•Action•Idea

•…and the relationships between these•…must contain these terms but in a particular relationship to one another

Most studies show that key word matching is wrong relationship to one another

Opinions, claims, arguments, facts, actions, concepts, messages

N i li it th / i it / i t t

word matching is wrong more than 50% of the time


Non-implicit: theme / spirit / intent


COMPLICATIONS & PROBLEMS w/ 1ST GEN AUTOMATED APPROACHES

•Sentiment may be of interest at multiple levels:•Document•Several DocumentsStatement / sentence•Statement / sentence

•Entity•Topic or Theme•Concept

•Language is chaotic:•Language is chaotic:•Jargon, slang, (slanguage) •Colloquialism•Forum-specific acronyms•ambiguity, •Anaphora (it, them)Anaphora (it, them)•Irony, sarcasm•Misspellings•Poor grammar•Variations in expression•Synonyms•Polysemy

•Context is key:•Meaning can change depending on context. •Need to look not only at the word but the connective tissues surrounding the word.



CONTEXT IS KEY



ANALYSING THE “CONNECTIVE TISSUE” OF A SENTENCE



HUMAN VS. AUTOMATION BOTH IS BEST

•Most sentiment analysis algorithms rely on

humans using simple terms to express our sentiment AUTOMATION

HUMANS

FOR

about a product or service. We don’t.

C lt l f t li i ti d diff i

FOR VOLUME & SPEED

RELEVANCE, DEPTH, CONTEXT, NUANCE

•Cultural factors, linguistic nuances and differing

contexts make it extremely difficult to automate turning

a string of written text into a simple pro or con Bridge the Gap with Human analysisg p p

sentiment. Human analysis of a sample

•The fact that humans often disagree on the sentiment

of text illustrates how big a task it is for computers to

get this right1615 M Street, NW; Suite 750 Washington, DC 20036 USA Tel 1.202.842.1818carma.com 84

get this right.


HUMAN (or hybrid)-BASED VENDORS



HUMAN-BASED ANALYSIS

Variety of scales out there both in terms of function and form:

Some sound, some silly:

•Just + or -

•Positive = 5; Negative = 0 (should a negative be negative?)

•-5 to +5

•Likert scale-ish: completely / somewhat / neither…

•Green red•Green red

•Sun, clouds, rain

BUT!: TONE ALONE AIN’T ALL THAT


BUT!: TONE ALONE AIN T ALL THAT


DOES SENTIMENT = FAVORABILITY = INFLUENCE POTENTIAL?

Do you think the sentiment and

how favorably your brand is portrayedhow favorably your brand is portrayed

or how

potentially influential a blog / blogger is

are the same thing?



FAVORABILITY; NOT (just) SENTIMENT OR TONE

Examining & Scoring factors that speak to the quality of the story + how favorable

the story was in its entirety & context.

NOT:

•Just tone / sentiment

•Accuracy, slant, fairness

•How it might or would likely affect the reader

•Researcher’s personal opinionResearcher s personal opinion



FAVORABILITY; NOT (just) SENTIMENT OR TONEWHO (else) IS TALKING ABOUT YOU? WHAT ARE THEY SAYING? IMPACT?

We quantify and compare the quality of how favorably eBay and its competitors are portrayed

Entities to be tracked•eBay•PayPalBillM L t•BillMeLater

•Gitti gidiyo•eBay foundation•Skype•Stubhub•CraigslistCraigslist•Etc.

Article Focus•Corporate•Auction•Customer•Product / service•Management•Financial general•Stock price (exclude given

l ?)


volume?) •Industry



What stakeholder commentary is appearing in eBay’s coverage? What impact is it having on eBay’s and their competitors’


Entities to be tracked•eBay•PayPalBillM L t

Sources•Company official•Customer•Customer group

and their competitors portrayals? Is it the usual list of stakeholder suspects or are others emerging over time?

•BillMeLater•Gitti gidiyo•eBay foundation•Skype•Stubhub•Craigslist

Customer group•Analyst•Partners•Government•Competitor•Etc.Craigslist

•Etc.

Article Focus•Corporate•Auction

Key Topics•Privacy security•Company future•Industry future•Partnerships / alliances•Customer

•Product / service•Management•Financial general•Stock price (exclude given

l ?)

•Partnerships / alliances•Mergers / acquisitions•Corporate strategy / vision•Management•Financial performance general•Stock performance



Stock performance•Market share



What stakeholder commentary is appearing in eBay’s coverage? What impact is it having on eBay’s and their competitors’


Entities to be tracked•eBay•PayPalBillM L t

Sources•Company official•Customer•Customer group

What messages are appearing? Are there other themes/ideas not on this list that are appearing

and their competitors portrayals? Is it the usual list of stakeholder suspects or are others emerging over time?

•BillMeLater•Gitti gidiyo•eBay foundation•Skype•Stubhub•Craigslist

Customer group•Analyst•Partners•Government•Competitor•Etc.

+/- Messages•Industry leader•Financially healthy•A good investment•Has good strategy / strong vision for the future

in the coverage? What impact are they having on ADP’s and your competitor’s portrayals?

Craigslist•Etc.

Article Focus•Corporate•Auction

Key Topics•Privacy security•Company future•Industry future•Partnerships / alliances

•Has good strategy / strong vision for the future•Well-managed•Growing•Well positioned for competition / outperforming competitors•Excellent customer service (customer endorsement)

•Customer•Product / service•Management•Financial general•Stock price (exclude given

l ?)

•Partnerships / alliances•Mergers / acquisitions•Corporate strategy / vision•Management•Financial performance general•Stock performance

Excellent customer service (customer endorsement)•Customer-focused•Delivers solutions that solve customer needs•Delivers high quality products, services•Delivers high value products, services•Delivers technically sound products, services



Stock performance•Market share •Stands behind its products, services

•Products/services are priced well or fairly•Etc.


BUT EVEN FAVORABILITY IS ONLY PART OF THE INFLUENCE PICTURE

Examining & Scoring factors that speak to the

quality of the story + how favorable the story was in its entirety & context.

NOT:

•Just tone / sentiment

•Accuracy, slant, fairness

•How it might or would likely affect the reader

•Researcher’s personal opinion





Score 10 10 10 10 10 10 10 10Weight 1 2 3 4 5 6 7









p y g y p g






•Could be a fluke







Causality •Heavy duty stats•You so can’t do this in

•Isolates (statistically

excel•Sophisticated modeling•Market Mix Modeling•Multivariate Regression Analysis

( y‘proves’) for PR’s unique contributionto the communications mix


Regression Analysis•Ton ‘o data, time, $

94


UPPERCASE ‘ROI’ vs. lowercase ‘roi’ & Other Return-Offs

ROI roi

MBA PR

Financial Non financialFinancial Non‐financial

Macro Micro (not all conversion events are ROI)

A mathematical equation that works

Usually conceptual only; an equation in words only; can’t actually be calculated; something the industry tosses around too liberallythat works be calculated; something the industry tosses around too liberally

A business metric All‐too‐often a media or audience metric

Direct, clear A loose proxy or an intermediate metric (return on expectation, return on objective return on engagement)return on objective, return on engagement)






PART 2 (30 mins): TONE ALONE

PART 3: (35 mins): REAL DATA WITH MALCOM DE LEO


Malcolm De Leo Chief EvangelistChief EvangelistNetBase

June 6, 2011

| Confidential | © 2011 NetBase Solutions. All Rights Reserved Worldwide.979

Consumer Insight Methods and Instruments

High In‐store observations

NetnographyIn‐depth interviews

Ethnography

of In

sight

g p y

Test Marketing

Dep

th o Test Marketing

Surveys (online)

Low

Focus Groups

Surveys (intercept)Surveys (phone)


Broad SamplingLow High

ow


A Love Story…

++ =Randy LoveProduct Marketing ManagerComcast Cable Communications

| Confidential | © 2011 NetBase Solutions. All Rights Reserved Worldwide.

100

Communications

The NetBase Difference – Language Understanding

Natural Language Processing

PositiveAttribute

BrandReason

I know Listerine kills germs because it hurts.


Text Analytics

Keyword Search

Brand NegativeWord

NegativeWord

NegativeWord


y

Keyword Keyword Keyword Keyword Keyword Keyword


101

A Short Netnography Case StudyCase Study


1021

Burn Burn


103

Verbatim

h i h I love Listerine coz it burns.

I hate it when Listerine burns your freaking mouth off!g

I love the burn of Listerine at 530 in

I really hate listerine*it [listerine*] makes Listerine at 530 in

the morning.it [listerine*] makes my tounge burn!

Man I love listerinebut boy does it [listerine] burn!!!

i hate when the listerine starts to burn my cheeks.


104

[ ]y


105

Overview

This study is an analysis of consumers’ perception of Listerine.

• Source data: Social Media ‐ content on the Internet generated by consumers, e.g. blogs.

• Quantitative analysis: Amount of discussion about Listerine relative to other topics.

Consumer sentiment & passion about Listerine relative to other topics.

Drivers of consumer behavior in the mouthwash category.

• Qualitative analysis: Overall reasons consumers like or dislike Listerine.

K i i ht b t Li t i


106

Key insights about Listerine consumers.

Summary of Findings

• Consumers most like Listerine’s germ‐killing action.

• The most prevalent complaint from consumers is that Listerine is too harsh.The most prevalent complaint from consumers is that Listerine is too harsh.

• Overall reasons given for liking and disliking Listerine are as follows (in rank order):

Likes Dislikes

Kills germs Too harsh

Treats toenail fungus Toxicity

Good mosquito repellent Efficacy

Treats bad breath Taste

Treats acne Link to alcoholism

• Listerine has a tremendous number of off‐label uses for personal and home care based on its efficacy.


107

Research Approach

• Use freely available consumer generated discussions (“Social Media”) to understand:( Social Media ) to understand:

Overall positive/negative preference for Listerine.

Opportunities for leveraging likes and compensating for orOpportunities for leveraging likes and compensating for or overcoming dislikes.

I ti ti t li d Li t i b t l i l d d• Investigations centralized on Listerine but also included some related terms:

Specific flavors of ListerineSpecific flavors of Listerine

Related Listerine products, e.g. Listerine Strip


108

Social Media Sources for Consumer Insights

Internet Forums Online communities where consumers post questions answers

Blogs Commentary or news on a particular subject; other blogs function as

Microblogs Communication platforms where users post short status messagesconsumers post questions, answers,

comments, and general discussions.

Forums are often associated with particular topics such as hobbies, geographies, sports, lifestyles, illnesses and more

subject; other blogs function as personal online diaries.

Some blogs allow readers to leave comments, thus fostering discussions like those found in online forums.

users post short status messages.

The most popular microblog currently is Twitter.

illnesses, and more.


109

How many “hits” are there for Listerine?

The term Listerine is mentioned infrequently on social media.

• Netflix is a good comparison since it is mentioned so much on the web.

• Listerine is mentioned about as much as the major airlines.


110

Mouthwashes Buzz


111

What is the overall sentiment for Listerine?

• As compared to other brands, consumer sentiment for Listerine is moderate.

• Photoshop and Comcast are reliable control variables for positive and p pnegative extremes.


112

Sentiment by Brand


113

Passion Index


114

Positive Themes


115

Negative Themes


116

Recommend Behavior


117

Attributes – Net Scores


118

Main Opportunities & Threats

The main positives and negatives were analyzed for opportunities and threatsopportunities and threats.

Potential actions are proposed to address these opportunities and threatsthreats.


119

Opportunity: Toenail Fungus Treatment

Potential InnovationInsight

Determine effectiveness of Listerine for toe nail fungus treatment.

Introduce a toe nail fungus treatment “from the makers of Listerine”

The second most mentioned benefit of Listerine is that it can be used to treat toenail fungus.

This theme was 9% of positive comments the makers of Listerine”. This theme was 9% of positive comments.


120

Opportunity: Mosquito Repellent


Determine effectiveness of Listerine as a mosquito repellent.

Publish a survival kit with Listerine listed as not only a mouth wash but also a

A commonly mentioned benefit of Listerine is that it can be used to repel mosquitoes.

This theme was 6% of positive comments.as not only a mouth wash but also a mosquito repellent.


121

Opportunity: Acne Treatment


Determine effectiveness of Listerine for acne treatment.

Introduce an acne treatment “from the makers of Listerine”

Many consumers talk about using Listerine to treat their acne.

A few disputed the efficacy of this practice.makers of Listerine”.

This theme was 4% of positive comments.


122

Opportunity: Listerine Reminds Me of Grandpa


Include a grandfather figure in advertising creative.

Test whether emphasizing the masculinity of Listerine (e g its strong flavor) makes it

A few consumers talked about their emotional connection to Listerine.

Listerine tends to remind some people of their grandfathers of Listerine (e.g. its strong flavor) makes it

more appealing to some consumers.grandfathers.


123

Opportunity: Effectiveness of Twice a Day campaign

Insight Listerine and Reach created the “October Oral Care Challenge” for kids 6 and older To complete thekids 6 and older. To complete the challenge, our family had to brush, floss and rinse every day, twice a dayfor two weeks. We currently use, and have used both Listerine and Reach products for years, so this was an easy challenge for our family.

Listerine’s recent ad campaign encourages people to use Listerine twice per day.

Many consumers have not only

Potential Actions

begun to do so, but they also have begun to talk about it.

Lately, some others have been added: gargle twice a day with warm salt water, or Listerine; drink hot liquids,

h t d hi k t d

I brush my teeth 2+ times a day every day, and I use Listerine.

Begin to measure effectiveness of campaigns by looking for the messaging among consumers talking to each other.

such as tea and chicken soup, to ward off or diminish virus proliferation in throat.

I would start using Listerine 2 to 3 times a day.

g

Chuck Norris doesn't brush his teeth; he simply pours Listerine into his vodka twice a day. Also brushing at least twice a day and

using Antiseptic mouthwash like Listerine twice a day AM & PM.


124

y

What are the key takeaways/actions

•Why is sentiment important? It’s the elephant in the room

The suggestion box is public The suggestion box is public

Your market research doesn’t wear clothes anymoreanymore…


125