Download pdf - Negative Sentiment (or "Sentiment Analysis is Sh*te")

Mat Morrison@mediaczar

Tuesday, 5 July 2011



Elephant in the roomTuesday, 5 July 2011

or Emperor’s ClothesTuesday, 5 July 2011


Volume


VolumeRelevanceAUTHORITYSentiment

qualified by


quick Question:

How Accurate

is Sentiment analysis?


The

stinks


Robot Human

supervi

sed

Machin

eLearnin

g


Robots first...


I’m unqualified to

make judgements

about the science...


I’m unqualified to

make judgements

about the science...

...and So are you.


Because the Science Is hidden in a box


Because the Science Is hidden in a box

But scientific Results are Reproducable!!!


What’s the buzz around “Green Lantern”?


Positive Negative Net

Sysomos (Blogs) 40 17 23

TweetSentiment 29 11 18

WDTIT? 56.8 42 14.8

SocialMention 18 5 13

SM2 20 16 4

Sysomos (Tweets) 6.4 7.6 -1.2

Rotten Tomatoes 27 73 -46

A quick poll of a few sentiment engines’ analysis of the buzz surrounding the film “Green Lantern” demonstrates no correspondence.




RT @BangsandaBun: According to

some, I wasn't deserving/grateful/

cool enough to be there, but I

blogged about my Glastonbury

experience http://t.co/eEInrnu

Turk THis!


http://t.co/eEInrnu

http://t.co/eEInrnu

RT @BangsandaBun: According to

some, I wasn't deserving/grateful/

cool enough to be there, but I

blogged about my Glastonbury

experience http://t.co/eEInrnu

Turk THis!

Robot says Negative


http://t.co/eEInrnu

http://t.co/eEInrnu

http://t.co/eEInrnu

I know the bands for Glastonbury

were great this year, but I still

don't think you could have paid me to

be part of that mental crowd! You?

Turk THis!


I know the bands for Glastonbury

were great this year, but I still

don't think you could have paid me to

be part of that mental crowd! You?

Turk THis!

Robot says positive


SENTIMENT scoring using Amazon Mechanical Turk disagreed with 16/40 (40%) robot judgments.

Only 13/40 (32.5%) of robotic judgments were passed by all three human Judges.

Raw Data available at http://mczar.me/mkue1p


http://mczar.me/mkue1p

http://mczar.me/mkue1p

But Are Humans Reliable?

Emotional State/Mood

Priming

Cultural bias

Cognitive Bias

Famously not.

???


It Gets Worse...

The Same Human Scorer,

when presented with the

same statement on

multiple occasions, may

score it differently each

time.cf. A Guerilla Experiment in Sentiment Analysis http://mczar.me/lrp7Yi


http://mczar.me/lrp7Yi

http://mczar.me/lrp7Yi


The

stinks


Academic studiesgenerally focus on

Movies & the STOCK Market

Why is this?


Social Media data are full of irrelevant...


Social Media data are full of irrelevant...



VolumeRelevanceAuthoritySentiment


VolumeRelevanceAuthoritySentiment

All four are needed to make sense of data

But I’m only talking sentiment


The

stinks


To qualify social media buzz

Why might marketers want to perform Sentiment analysis?

To guide customer service interactions

As a Performance indicator

To identify fans


Is sentiment actionable?

Product Query

Product Complaint

Product endorsement

Product review

Product COmparison

...I need to know more

Is it a

...

???


!"#$

#$

"#$

%##$

%"#$

&##$

&"#$

'##$

()*+,-+.$ /+0+,-+.$ 12342.5$ 6+-.42.5$ 72.08$ 9:.;<$ 725$

!"#"$%&'#

=)*;>$?)@2<$

A2.-4.@)3>$?)@2<$

B;3C>,;<<$?)@2<$

Massive (and unrepresentative) spike in Net Positive

Sentiment caused by specific campaign activity

THE KPI Problem

Should we be allowed to mark our own homework?



