Sentiment, News and the Polarity Problem

Leslie Barrett

www.lbtechconsulting.com

April 13, 2010

Sentiment and Opinion

• Are sentiment and opinion the same?• Are feelings the same as beliefs?• Sentiment can be applied to opinion but

not the other way around (Kim And Hovy 2004)

• The question is – should it apply to anything else? Does it make sense in narrative, exposition, news data?

• How much text should we apply it to?

Sources

• Sentiment analysis has been applied where opinion is the norm – blogs and Tweets

• It has also been applied where opinion is designed to be subtle, if expressed at all – news data

• So maybe news data is never really objective, or else maybe sentiment is really used as simple polarity – separating the world into human ideas of positive and negative “buckets” blind to objectivity

Polarity

• Polarity is the stuff through which sentiment is measured

• Sentiment is usually considered to have the “poles” positive and negative

• These are most often “translated” into “good” and “bad”

• Sentiment analysis is really considered useful for telling us what is “good” and “bad” in our information stream

The “Machine”

• So the sentiment analysis machine takes in some text and tells us whether that text says something “good” or “bad”.

• OK…..but before we unveil our machine, we need to ask some important but often overlooked questions:

• - what text is going in?• - where does “good” stop and “bad” begin?• - what is the text “about”?

Why do we needSentiment AnalysisBeavis?

So we’ll know what we’re thinking!

Let’s Try Feeding the Machine News Data!

• News Headlines sound like a pretty straightforward text type to apply sentiment to, given what we’ve just said.

• Even though news is supposed to be “objective”, headlines sell papers and often can be dramatic

• Keywords like “crash”, “downturn” and “disaster” are abundant and strong sentiment indictors.

• - but are headlines enough? • - we may want document-level sentiment for

news• - does it matter what the news is “about”?

Some “real” headlines

• Short-lived • Coup• Disappoints• Bears

Beware of Headlines in Financial News

• financial news especially is really a genre unto itself

• Its polarity perspective is skewed constantly by pundit “benchmarking”

• Beating bad expectations is better than a good quarter that falls short – in pundit opinion

Can Sentiment Analysis “beat expectations”?

• All kinds of negatives here but the document-level sentiment should be positive – that’s how an analyst would see it

• So if you skew to this, what about other news?

Objectively “bad” Events Happen

• Some events don’t require an opinion holder

• They simply have a generally agreed upon negative or positive polarity

• And we need to get them right because they affect other events (e.g. crop yields, etc)

When Bad Things Happen to Positive Sentiment

• But objectively bad events have their own problems, even in the absence of “expectations”.

• The problem with polarity measures outside of the presence of an opinion holder is topic drift

• An editorial or blog is likely to stick to one sentiment, but bad events can have the dreaded “silver lining”

Disaster+Relief Can Spell Trouble

• Despite some strong negative polarity indicators like “traumatized”, “disaster” and “tsunami” this article has an overall positive theme

Don’t Quote Me!

• Another problem in news data is “opinion blend”

• Often you have an author’s opinion but other opinions that may differ – directly or indirectly cited

• Or an author using quotes to showcase two different opinions

• Coverage of a “debate” for example can get very difficult for even a human to judge

Attribution vs. Quoting

• The author clearly does not believe the positive topic of the article

• But Clinton believes it• So is this positive

sentiment about Clinton?

Pundits vs. Authors vs. Topics

• How can I be sure that “bad news” about my client is about my client?– Make sure the named entity in question is a topic of

the document– So-called “document mates” don’t matter

• Do author names matter? Should I extract them?– Yes! Over time if you classify by author name against

other entities you might detect bias– Do the same for known “pundits” on a topic…..same

result may emerge

What’s it all About?

• Some data just tends to be multi-thematic or non-thematic

• In particular, market and financial reports, which often make their way into news feeds, tend to be this way.

• It is very hard to get a reasonable sentiment reading on either type of document.

SEC Reports: too big, too many sections

• There is the Management Discussion, which can have appropriate sentiment scores

• But there are so many other sections, no single theme

• Many sections have boilerplate, such as the accounting review

Scraping

• Your data is only as good as your news feed.

• Sometimes a site will deliver excess content that creeps into the text field of a feed

• That content could be an ad or even another article, skewing the sentiment reading for the expected article and hurting topic detection too.

Field Overlap from a Typical News Page

What to Do?

• Stop doing Sentiment Analysis on news data?

• NO!• News data is very valuable for reputation

management• Also can be valuable for investment firms

*if* you can tease out the jargon and pundit-speak

• Document-level is still OK!

Best Practices

• Good topic detection

- see what’s closely aligned with a theme and eliminate non-thematic or weak-thematic documents

• Good feed maintenance

- you or your feed provider need to spot check for scraping problems

Tricks & Tips

• Data extraction for problem documents– If document sections are identified with tags, use

them (this is true for SEC reports) and extract the “good” data (see Pang and Lee 2004 on extracting document portions)

– Write regular expression libraries to find quoted and cited material. Remove or use separately

• Topic drift is harder but….– you can extract the first n paragraphs. Main topical

material in news generally in top 25% of document– Secondary topics don’t carry same weight

What’s Next for Polarity?

• Future directions for news-based sentiments analysis are based on looking outside of Positive and Negative poles

• Think about all the “opposites” in the world– Sweet/sour– Cold/hot– Inside/outside– Wet/dry– Hard/soft

Leverage the Semantics of Opposition

• There are many types of opposition to study and they can be used in different ways– Complementary opposites (male,female)– Reversatives (backwards, forwards)– Scalar opposites (tall, short)

• A good deal of semantic research that has yet to be leveraged for opinion analysis and classification (Mettinger, Pustejovsky, Kennedy, Miller, inter alia…)

Opposites and Opinions

• Let’s think of some opinions that fit into poles not definable in terms of “positive” and “negative”– Conserative vs. Liberal– Government Expansion vs. Privatization

• Can these positions be detected automatically?

• ………..

Appendix/Bibliography• Kim, Soo-Min and Eduard Hovy. 2004. Determining the

Sentiment of Opinions. Proceedings of COlING-04. pp. 1367--1373. Geneva, Switzerland.

• James Pustejovsky, "Events and the Semantics of Opposition" in Events as Grammatical Objects , C. Tenny and J. Pustejovsky (eds.), 2000, CSLI Publications.

• Arthur Mettinger, Aspects of Semantic Opposition in English, Clarendon Press, Oxford, 1994

• Bo Pang and Lillian Lee, “A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts”, In Proceedings of the Association for Computational Linguistics, 2004