View
222
Download
3
Category
Preview:
Citation preview
#like or #fail
How Can Computers Tell the Difference?
Mark Cieliebak 27.3.2014
Institute of Applied Information Systems (InIT)
ZHAW
Zurich University of Applied Sciences
- 10'000 students, 230 professors (FTE)
InIT
Institute of Applied Information Systems
- more than 30 professors
- Labs on Cloud Computing, Data Science etc.
2
[11]
27.3.2014 Mark Cieliebak
About Me
27.3.2014 Mark Cieliebak 3
Mark Cieliebak Institute of Applied Information Technology (InIT)
ZHAW, Winterthur
Email: ciel@zhaw.ch, Website: www.zhaw.ch/~ciel
Text
Analytics
Open
Data
Automated
Test
Generation
Research
Interests
Social
Network
Analysis
What is "Text Analytics"?
Synonyms: Text Mining, Natural Language Processing (NLP)
4 Picture Sources: see slide "References"
[6]
[5]
[3]
[2]
[1]
[4]
27.3.2014 Mark Cieliebak
Sample Application: Social Media
Monitoring
Text Analytics Components:
• Find relevant documents
• Hot topic Analysis
• sentiment analysis
27.3.2014 Mark Cieliebak 5
[7]
Sentiment Analysis for Product Reviews
"… WiFi Analytics is a free Android app that I find very
handy when it comes to troubleshooting and monitoring
a home network. " [8]
27.3.2014 Mark Cieliebak 6
Flavours of Sentiment Analysis
• Document Based
• Sentence Based
• Target-Specific
• Rating Prediction
27.3.2014 Mark Cieliebak 7
Simple Sentiment Analysis
Idea: Count number of positive and negative words
"This analysis is good[+1]." +1 (pos)
"I find it beautiful[+1] and good[+1]." +2 (pos)
"It looks terrible[-1]." -1 (neg)
"This car has a blue color." 0 (neu)
POSITIVE:
good
love
nice
...
NEUTRAL:
hello
see
I
…
NEGATIVE:
bad
hate
ugly
...
Use Sentiment-Dictionary:
27.3.2014 Mark Cieliebak 8
Improvements
Sample Text
• "This analysis is good."
"This analysis is excellent."
• "The car is really very
expensive."
• "This car has an appealing
design and comfortable
seats, but it is expensive."
Solution
• Fine-Grained Dictionaries:
"This analysis is good [+2]. "
"This analysis is excellent [+3].
• Detect Booster Words:
"The car is really very
expensive[-1 -1 -2] ."
• New Category "Mixed":
"This car has an appealing[+1]
design and comfortable[+1]
seats, but it is expensive[-1]. "
mix(+2/-1)
27.3.2014 Mark Cieliebak 9
Improvements 2
Sample Text
• This analysis is not good.
• The car is appealing and I do
not find it expensive.
• I do not find the car expensive
and it is appealing.
Solution
• Invert all Scores:
"This analysis is not[*-1]
good[+3]."
• Invert only score of words
occuring after the negation:
"The car is appealing[+3] and I
do not[*-1] find it expensive[-2]"
Need to “understand” the
sentence
Need linguistic analysis!
27.3.2014 Mark Cieliebak 10
Linguistic Analysis
-> RULE: Invert scores of words being in the same phrases as negation.
“I do not find the car expensive[+2]
and it is appealing[+3].” → +5 (pos)
Sentence
Sentence Conj. Sentence
Noun
Phrase
Verb Phrase
Verb Adverb Verb Noun
Phrase
Adj. Noun
Phrase
Verb Phrase
Det. Det Noun Det. Verb Participle
I do not find the car expensive and it is appealing
27.3.2014 Mark Cieliebak 11
Technology Stack for Sentiment Analysis on Web Documents using Linguistic Analysis
Linguistic Framework • Gate: text analysis framework with experimentation GUI; includes sentence splitter, tokenizer, chunker, pos-tagger, etc; plugin-
mechanism for new libraries; very widely used. http://gate.ac.uk/
• OpenNLP: Java-based, easy to use; includes tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, etc;
by Apache Foundation. http://opennlp.apache.org/
Web Boilerplate Extraction • Boilerpipe: removes ads, menus, HTML-tags etc. http://boilerpipe-web.appspot.com/
Language Analysis • Nutch language identifier plugin: 14 languages preimplemented, „easy“ to add more. http://wiki.apache.org/nutch/LanguageIdentifierPlugin
Sentence Splitting • OpenNLP: very easy to use, in Java. http://opennlp.apache.org/
• Punkt. In Python http://nltk.org/api/nltk.tokenize.html
Part-of-Speech Tagger • TreeTagger: language-independent, several languages implemented;. http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/
• TweetNLP: POS-tagger just for Twitter. http://www.ark.cs.cmu.edu/TweetNLP/
Chunker • TreeTagger: supports English, German, French. http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/
Dictionaries • WordNet: large lexical database of English. http://wordnet.princeton.edu/
• SentiWordNet: English dictionalry with sentiment annotations. http://sentiwordnet.isti.cnr.it/
27.3.2014 Mark Cieliebak 12
Main Approaches to Sentiment Analysis
Rule-Based Corpus-Based
27.3.2014 Mark Cieliebak 13
Predicted
Label
[9] [10]
Corpus-Based Sentiment Analysis
27.3.2014 Mark Cieliebak 14
Predicted
Label
[10]
Corpus-Based Sentiment Analysis
Annotated Corpus
Sentence Polarity
This analysis is good. Pos
It looks awful. Neg
This car has a blue color. Neu
This car has an appealing design, comfortable seats, but it is expensive. Mix
This car has a very appealing design, comfortable seats, but it is really expensive. Mix
This analysis is not good. Neg
This car has an appealing design, comfortable seats and it is not expensive. Mix
This movie was like a horror event. Neg
This car is appealing and is not expensive. Mix
... ...
27.3.2014 Mark Cieliebak 15
Sample Features for Tweets
• Word ngrams: presence or absence of contiguous sequences of 1, 2, 3,
and 4 tokens; noncontiguous ngrams
• POS: the number of occurrences of each part-of-speech tag
• Sentiment Lexica: each word annotated with tonality score (-1..0..+1)
• Negation: the number of negated contexts
• Punctuation: the number of contiguous sequences of exclamation marks,
question marks, and both exclamation and question marks
• Emoticons: presence or absence, last token is a positive or negative
emoticon;
• Hashtags: the number of hashtags;
• Elongated words: the number of words with one character repeated (e.g.
‘soooo’) from: Mohammad et al., SemEval 2013
27.3.2014 Mark Cieliebak 16
How good are Sentiment Analysis Tools?
27.3.2014 Mark Cieliebak 17
Evaluation joint work with Oliver Dürr and Fatih Uzdilli [15]
• 7 Public Text Corpora
– Single Statements
– Different Media Types
• Tweet, News, Review,
Speech Transcript
– Total: 28653 Texts
• 9 Commercial APIs
– Stand-alone
– Free for this evaluation
– Arbitrary Text
NEGATIVE POSITIVE
OTHER ( neutral / mixed )
18 27.3.2014 Mark Cieliebak
Sentiment Analysis Tools
AlchemyAPI www.alchemyapi.com
Lymbix www.lymbix.com
ML Analyzer www.mashape.com/mlanalyzer/ml-analyzer
Repustate www.repustate.com
Semantria www.semantria.com
Sentigem www.sentigem.com
Skyttle www.skyttle.com
Textalytics core.textalytics.com
Text-processing www.text-processing.com
27.3.2014 Mark Cieliebak 19
Tool Accuracy
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accu
racy
Best Tool per Corpus
Worst Tool per Corpus
20
61%
40%
Avg.
27.3.2014 Mark Cieliebak
Tool Accuracy
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accu
racy
Best Tool per Corpus
Worst Tool per Corpus
Overall Best Tool
21
61% 40% 59%
Avg.
27.3.2014 Mark Cieliebak
Summary
"They all suck…and we suck, too."
CEO of a Sentiment
Analysis Company
27.3.2014 Mark Cieliebak 22
State of the Art in Academia
SemEval 2013: Competition on Semantic Analysis
• Task 2A: Contextual Polarity Disambiguation
• Task 2B: Message Polarity Classification
Corpus: 15'000 tweets from 2012 on various topics
annotated via Mechanical Turk
27.3.2014 Mark Cieliebak 23
SemEval 2013 Task 2B
Results
• 36 Submissions
• Average F1 = 54%
• Only 9 teams above 50%, all others below!
• Best Team: F1 = 69%
• For comparison: best commercial tool has F1 = 62%
27.3.2014 Mark Cieliebak 24
Can a Meta-Classifier do better?
1st Approach: Majority Classifier
Sentiment with most votes chosen
Illustration:
25 27.3.2014 Mark Cieliebak
api1 api2 api3 api4 api5 api6 api7 Majority
Text 1 + + - o - + o +
Text 2 - + + - - - - -
Text 3 - o + + + + - +
Text n o o + o - o o o
Tool Accuracy
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accu
rac
y
Best Tool per Corpus
Worst Tool per Corpus
26 27.3.2014 Mark Cieliebak
Majority Classifier Close to Best Tool
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accu
rac
y
Best Tool per Corpus
Worst Tool per Corpus
Majority Classifier
27 27.3.2014 Mark Cieliebak
2nd Approach: Random-Forest
28 27.3.2014 Mark Cieliebak
api1 api2 api3 api9 annotation
Text 1 + - + o +
Text 2 - + o + -
Text 3 - o - + -
Text 4 + o + - +
Text 5 + o + o o
Text 6 + o o - o
Text 7 + - + o unknown
Text 8 + + o - unknown
Text 9 o - + o unknown
Random
Forest
Classifier
+
+
o
Train
Train
Train
Train
Predict
Predict
Predict
Train
Train
Before Random Forest
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accu
rac
y
Best Tool per Corpus
Worst Tool per Corpus
Majority Classifier
29 27.3.2014 Mark Cieliebak
Random Forest beats Best Single Tool
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accu
rac
y
Best Tool per Corpus
Worst Tool per Corpus
Majority Classifier
Random Forest Classifier
30 27.3.2014 Mark Cieliebak
Challenges in Sentiment Analysis
• "Salaries for software engineers are extremely high."
Context Dependencies
• “What a great car, it stopped working the second day.”
Sarcastic Statements
• "#YouCantDateMe if u still sag ur pants super hard...dat shit is
played the fuck out!!! "
Informal Language
• Huge Effort for New Languages
27.3.2014 Mark Cieliebak
Talk in Short!
1. Sentiment analysis is an
important challenge
2. Commercial tools classify 6
out of 10 docs wrong
3. Academic systems are
not really better
4. Random forest outperform
even the best tools
27.3.2014 Mark Cieliebak 32
[14]
[12]
[13]
References
[1] http://www.teleportmyjob.com/blog/wp-content/uploads/2013/05/find_job.jpg
[2] http://screenshots.de.sftcdn.net/de/scrn/3340000/3340669/swiftkey-23-630x535.png
[3] https://lh4.ggpht.com/Fs4HLaMCgmY6O7BWGKZB8LdOM8GmiRAbKUcdJysNp0k74xmPMTSGVHigwvaLy_Tw=w300
[4] http://researcher.watson.ibm.com/researcher/files/us-bansal/watson2.jpg
[5] http://www.betadaily.com/wp-content/uploads/2011/02/BusinessPlanExecutiveSummary.jpg
[6] http://everycook.org/cms/en/hardware-en/pictures-en#
[7] http://cdn.fulltraffic.net/images/blog/social-media-monitoring-process.png
[8] http://www.pcmag.com/article2/0,2817,2426416,00.asp
[9] http://i1059.photobucket.com/albums/t424/shmi11y/UoS%20LingSite/tompushedthecar.png
[10] http://bigsnarf.files.wordpress.com/2013/04/supervised.png
[11] http://www.alumni.sml.zhaw.ch/portals/0/Volkartgeb%C3%A4ude%20gek%C3%BCrzt%202.jpg
[12] http://www.bitspeed.com/wp-content/uploads/2011/09/Sign-multi-cast.jpg
[13] http://wwwdelivery.superstock.com/WI/223/1829/PreviewComp/SuperStock_1829-47184.jpg
[14] http://www.odditycentral.com/art/arborsculpture-the-art-of-turning-young-trees-into-living-works-of-art.html
[15] Cieliebak, Dürr, Uzdilli, ESSEM 2013
27.3.2014 Mark Cieliebak 33
Recommended