NATURAL LANGUAGE PROCESSING

Sentiment Analysis on Twitter

Mentor: Prof. Amitabha Mukherjee

ByRohit Kumar JhaSakaar Khurana

Department of Computer Science and Engineering, IIT Kanpur

11/18/2013 1NLP|CS671

Introduction

In the past decade, new forms of communication, such as

microblogging and text messaging have emerged and become

ubiquitous. While there is no limit to the range of information

conveyed by tweets and texts, often these short messages are used to

share opinions and sentiments that people have about what is going

on in the world around them. We plan to work on the following task.

This task was part of SEMEVAL 2013 challenge. The task:

Given a message, classify whether the message is of positive, negative, or neutral sentiment. For messages conveying both a

positive and negative sentiment, whichever is the stronger sentiment should be chosen.

11/18/2013NLP|CS671 2

Motivation

• Tweets and texts are short: a sentence or a headline rather than a document. The

language used is very informal, with creative spelling and punctuation, misspellings,

slang, new words, URLs, and genre-specific terminology and abbreviations, such as,

RT for "re-tweet" and #hashtags, which are a type of tagging for Twitter messages.

• Another aspect of social media data such as Twitter messages is that it includes rich

structured information about the individuals involved in the communication. For

example, Twitter maintains information of who follows whom and re-tweets and tags

inside of tweets provide discourse information.

• Working with these informal text genres presents challenges for natural language

processing beyond those typically encountered when working with more traditional

text genres, such as newswire data.

• Modelling such structured information is important because: (i) it can lead to more

accurate tools for extracting semantic information, and (ii) because it provides means

for empirically studying properties of social interactions (e.g., we can study properties

of persuasive language or what properties are associated with influential users).

11/18/2013NLP|CS671 3

Previous work

11/18/2013NLP|CS671 4

Bag of Words Model• Use a word list where each word has been scored positivity/negativity or sentiment

strength

• Overall polarity determined by the aggregate of polarity of all the words in the text

• In SEMEVAL 2013, Team IITB used Bag of Words model with Discourse Information and able to achieve an accuracy of 39.80%

Naive Bayesian Classifier• Straightforward and frequently used method for supervised learning

• Maximum entropy classifiers are commonly used as alternatives to Naïve

• Bayesian classifier because they do not require statistical independence of the features that serve as predictors

• In SEMEVAM 2013, Team uottawa used Naïve Bayesian Classifier were able to achieve an accuracy of 42.51%

Support Vector Machine• In SEMEVAL 2013, Team NRC-Canada used SVM model with unigram, bigram, POS tags,

negation etc as features and were able to achieve an accuracy of 69.02%

Our Work

Simple and Naive implementation of Bag of

Words Model

Incorporate the use of emoticons

Bag of Words model with Discourse Information

Sentence Weightage assignment in Bag of

Words Model

Incorporate the effect of #hashtags

Incorporate the effect of modifiers like "very",

"too", etc

SVM implementation with unigrams+bigrams, negation, POS tagging,

etc as features

Use Bag of Words Model with SVM

Take care of misspellings, informal use of words

like happpppyyy, abbreviations, etc

11/18/2013NLP|CS671 5

Naive Implementation of Bag of Words

11/18/2013NLP|CS671 6

Important Points

• Use a word list where each word has been scored positivity/negativity or sentiment strength

• Overall polarity determined by the aggregate of polarity of all the words in the text

• Major polarity words have been given polarity between -4 to 4 depending on their polarity, like excellent is +4, but happy is +2, and so on

Results

• Accuracy of ~42% when using Bag of Words Model without any special care

• Team IITB also used Bag of Words model but could achieve only ~40% accuracy. The major difference being probably that they considered only 0, +1, -1 values of polarity of words.

Bag of Words Model with features

11/18/2013 7

Important Features

• Incorporate the use of emoticons. If there are emoticons, clearly showing a particular

emotion, don't proceed further

• Incorporate the use of Discourse Information along with the Bag of Words model

• Assigning double weightage to polarity words in sentences occurring later on in

tweets. But it didn't make an overall difference

• Incorporate the effect of #hashtags. Apply Bag of Words model on the tweets after

doing word boundary segmentation and if some clear sentiment appears, don't

proceed ahead

• Incorporate the effect of modifiers like "very", "too", etc

Results• Achieved accuracy of ~56% after considering these features

NLP|CS671

SVM Implementation

11/18/2013 8

Important Features

• POS: the number of occurrences for each part-of-speech

tags. 'NN','VG','CD','JJ','CC','RB' pos tags are the only ones considered

• emoticons:

- presence/absence of positive and negative emoticons at any position

in the tweet;

- whether the last token is a positive or negative emoticon;

• negation: the number of negated contexts. A negated context also affects

the ngram and lexicon features: each word and associated with it polarity in

a negated context become negated

• frequency: select top 1000 words and their presence

Results

• Achieved accuracy of ~61% after considering these features

NLP|CS671

SVM + Bag of Words Model

11/18/2013NLP|CS671 9

Important Points

• Using a Hybrid Classifier using emoticons, Bag of Words Model and SVM

Classifier

• If there is a clear presence of emoticons denoting the emotion of a tweets,

simply use it

• Otherwise use the Bag of Words model and if there is a certain confidence

level in the polarity reported, simply go with it

• If no clear and confident outcome obtained, use SVM classifier and the

results from the first two steps to decide

Results• Achieved an accuracy of 68.32% with training on only around 8000 tweets,

which is a really small training data

Results

SerialNo.

Methods Used Data Size Accuracy Obtained on SEMEVAL 2013 data

1. Bag of Words Model N.A. 39.80%

2. Naïve Bayes Classifier 10,000 tweets 42.51%

3. Support Vector Machine 1.6 Million tweets

69.02%

4. SVM + Bag of Words Model(Our result)

8,000 tweets 68.32%

11/18/2013NLP|CS671 10

References

Subhabrata Mukherjee, Pushpak Bhattacharyya. Sentiment Analysis in Twitter with

Lightweight Discourse Analysis

http://aclweb.org/anthology//C/C12/C12-1113.pdf

Preslav Nakov, Sara Rosenthal ,Zornitsa Kozareva ,Veselin Stoyanov ,Theresa Wilson. SemEval-

2013 Task 2: Sentiment Analysis in Twitter

http://www.aclweb.org/anthology/S/S13/S13-2052.pdf

Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SentiWordNet 3.0: An

enhanced lexical resource for sentiment analysis and opinion mining. In Nicoletta Calzolari (chair),

Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and

Daniel Tapias, editors, Proceedings of the Seventh International Conference on Language

Resources and Evaluation, LREC ’10, pages 2200–2204, Valletta, Malta.

11/18/2013NLP|CS671 11

Summary and Conclusion

Objective

• Given a message, classify whether the message is of positive, negative, or neutral

sentiment. For messages conveying both a positive and negative sentiment,

whichever is the stronger sentiment should be chosen.

Result Obtained

• Achieved an accuracy of 68.32% with training on only around 8000 tweets, which is a

really small training data, compared to what was used by the team that came first

Significance

• Our method achieves good accuracy with relatively small amount of data compared

to that used by others

• Our method is targeted towards applications that cannot afford to use heavy

processing

Dataset Used

• We used the dataset provided by SEMEVAL 2013 for both training and testing

purposes

11/18/2013NLP|CS671 12