Embed Size (px)
Citation preview
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Deep learning for Tweeter sentiment analysis
Benoit Favre1, Mickael Rouvier
Aix-Marseille Université
April 20, 2016
[email protected]. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 1 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Outline
Context▶ The SENSEI project▶ Sentiment analysis task
Approach▶ Deep learning architecture▶ Word embeddings
SemEval participation▶ Results
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 2 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
The SENSEI project
Making sense of human-human conversations▶ Spoken conversation transcripts▶ Social media texts
Approach▶ Semantic and discourse parsing, para-semantics analysis▶ Feed summarization systems
Scientific problems addressed▶ Adaptation to non canonical language▶ Integration of analysis layers▶ Summarization on novel genres▶ Evaluation by end-users
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 3 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
The sentiment analysis task
Various tasks of sentiment analysis▶ Predict the general polarity of sentences / documents (negative, neutral,
positive)▶ Predict the polarity given a target phrase▶ Predict phrases which are targets (aspect classification)▶ Emotion detection
Classic approaches▶ Word lists (SentiWordNet, NRC Emotion Lexicon...)▶ Classifiers based on textual features (bag of word n-grams)
Resources▶ http://neuro.imm.dtu.dk/wiki/Sentiment_analysis
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 4 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Convolutional Neural Networks for text
Source: Zhang, Y., Wallace, B. (2015)
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 5 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Word embeddings
Objective▶ From one-of-n (or one-hot) representation to low dimensional vectors▶ Similar words should be similarly placed▶ Train from large quantities of text (billions of words)
Hypothesis▶ Meaning is defined by word neighborhood▶ Related to distributional semantics
Approaches▶ LSA (Deerwester et al, 1990)▶ Random indexing (Kanerva et al, 2000)▶ Corrupted n-gram (Colobert et al, 2008)▶ Hidden state from RNNLM or NNLM▶ Word2vec (Mikovol et al, 2013)▶ GloVe (Pennington et al, 2014)
w1 w2 w3 w4 w5w1w2
w4
w3
w5
1 3
22
1
12
3
1
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 6 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Word2vec: detailsProposed by Mikolov et al, 2013, code available athttp://code.google.com/p/word2vec.Task
1 Given bag-of-word from window, predict central word (CBOW)2 Given central word, predict another word from the window (Skip-gram)
sum
Wi-n
WiWi-1
Wi+n
...
...
embedding
CBOW
Wi-nWi
Wi-1
Wi+n
...
...
embedding
Skip-gram
input output input output
Training (simplified)▶ For each word-context (x, y) :
⋆ y = softmax(Wx+ b)⋆ Update W and b via error back-propagation
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 7 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Linguistic regularities
Inflection▶ Plural, feminine▶ Comparatives, superlatives▶ Verbs
Semantic relations▶ Captial / country▶ Leader / group▶ analogies
Linear relations▶ king + (woman - man) = queen▶ paris + (italy - france) = rome
Example2 trained on Wikipedia.
2http://pageperso.lif.univ-mrs.fr/~benoit.favre/tsne-frwiki/B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 8 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation for sentiment analysis
Most similar word embeddings to sentiment-bearing words▶ Antonyms are at the same place
Lexicalgood badgreat goodbad terriblegoid baaadgpod horriblegud lousy
decent shittyagood crappygoood suckyterrible horible
gr8 horrid
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 9 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Motivation (2)
Multiview approach: bias the embeddings to capture different properties
Lexical Part-of-speech Sentimentgood bad good bad good badgreat good great good great terriblebad terrible bad terrible goid horriblegoid baaad nice horrible nice shittygpod horrible gd shitty goood crappygud lousy goid crappy gpod sucky
decent shitty decent baaaad gd lousyagood crappy goos lousy fantastic horridgoood sucky grest sucky wonderful stupidterrible horible guid fickle-minded gud :/
gr8 horrid goo baaaaad bad sucks
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 10 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Types of word emebddings
Lexical embeddings▶ Basic skip-grams from word2vec▶ wi−2, wi−1, wi+1, wi+2 → wi
Part-of-speech embeddings▶ Join model of words and part-of-speech tags▶ (wi−2, pi−2), (wi−1, pi−1), (wi+1 : pi+1), (wi+2, pi+2) → (wi, pi)
Sentiment embeddings▶ Model that predicts lexical context and the polarity of the smileys in the tweet
[Tang et al, 2014]▶ (wi−2, s), (wi−1, s), (wi+1, s), (wi+2, s) → wi
Pre-trained on 90 million tweets collected by searching for n-grams from thetraining corpus
▶ 20 million tweets have smileys
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 11 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Neural architecture
Lexical%embeddings%
Part8of8speech%embeddings%
Sen-ment%embeddings%
w1# w2# w3# wi'1# wi# w1# w2# w3# wi'1# wi# w1# w2# w3# wi'1# wi#
CNN#
Output'
Output'
CNN#
Output'
CNN#
Output'
28level%
18level%
Input'
How to fuse representations▶ Late fusion: train separate systems and fuse their decisions▶ Early fusion: concatenate feature vectors and train a single system
Embedding fusion [Rouvier et al, 2015]▶ Train separate systems, and concatenate activations in their hidden layers as
input of another system
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 12 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Accounting for classic features
Word%embeddings%Feature%Lexicon%@%Word%level%
Feature%Lexicon%@%Sentence%level%w1# w2# w3# wi'1# wi#
CNN#
Output'
Word%Representa-on%
Sentence%Composi-on%
Sentence%Representa-on%
Word and sentence-level features▶ Word presence in lexicons (MPQA, Opinion lexicon, NRC Emotion lexicons)▶ Emoticons by category (+/-)▶ ALL-CAPS words▶ Elongated words (goooooood)▶ Fractionation relations
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 13 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Scientific competition: SemEval 2016
10th International Workshop on Semantic Evaluation / NAACL▶ http://alt.qcri.org/semeval2016/ (San Diego, California, June 16-17,
2016)▶ 14 tasks (textual similarity, sentiment analysis, semantic parsing...)
Participated to Task 4.A: “Polarity prediction in Twitter"▶ 34 teams from 25 countries participated▶ A lot used DNNs, CNNs, word embeddings
Corpus Positive Negative Neutral TotalTrain 7,727 2,916 7,837 18,480Dev 884 279 616 1,779
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 14 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
ExamplesPositive
Good Morning Becky ! Thursday is going to be Fantastic ! @SwedenG @DJ4JG @Grdina@Paverlayer @FSBull @RevkahJC @DicksTrash @borderfox116
Welcome Nick J! :"") “"@nickjonas"" In Manila! So excited for the show on Friday.
It’s midnight on the east coast which means its @nickjonas birthday! HAPPY 20thBIRTHDAY NICK J!!!!!!!!!! <333
Neutral
Gov. Mitch Daniels will announce his Indiana Supreme Court pick at 10 a.m. Will the 3rdtime be the charm for a female member of the panel?
The pain is far deeper than a Billy cundiff missed field goal. Gotta wake up and forgetabout it tomorrow. #Orioles #stayhungry
Smooches from Fido: Yea or Nay? - July 6th was International Kissing Day (of the humansort). Later in the month, Ca...
Negative
Theo Walcott is still shit, watch Rafa and Johnny deal with him on Saturday.
OBVIOUSLY, i am not a fan of Dream High 2 and i haven’t even watched it! well, iwatched the 1st ep,that’s where i heard that song!M
@MeganWitmer that’s crazy. I’m more jealous of u going to Joe Pa’s grave! I was sadSaturday when I walked past where the statue should be:(
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 15 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Evaluation results
Metric: Fnp =Fn+Fp
2
Rank System Score1 SwissCheese 0.6332 SENSEI-LIF 0.6303 UNIMELB 0.6174 INESC-ID 0.6105 aueb.twitter.sentiment 0.6056 SentiSys 0.5987 I2RNTU 0.5968 INSIGHT-1 0.5939 TwiSE 0.586
10 ECNU 0.585...
35 Baseline 0.255
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 16 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Analysis (1)
What is the impact of word embeddings and other features▶ Fusion is better than individual embeddings▶ Best embedding is “sentiment embedding" with distant supervision▶ Word embeddings are the most important features▶ Word-level lexicon features hurt
Feature set Lexical Part-of-speech Sentiment SENSEI-LIFall features 61.3 62.0 62.3 63.0w/o word level lexicon 61.7 (+0.4) 62.4 (+0.4) 61.6 (-0.7) 63.2 (+0.2)w/o sentence level lexicon 60.7 (-0.6) 61.1 (-0.9) 62.0 (-0.3) 62.6 (-0.4)w/o both lexicon 61.0 (-0.3) 61.4 (-0.6) 61.8 (-0.5) 62.8 (-0.2)w/o word embeddings 58.4 (-2.9) 59.1 (-2.9) 59.6 (-2.7) 59.6 (-3.4)
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 17 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Analysis (2)
Is the approach robust to domain / corpus changes?▶ The system is very good on SMS and Blog corpora▶ Sarcasm is hard to detect
Corpus SENSEI-LIF RankTwt2013 70.6 3SMS2013 63.4 3Twt2014 74.4 1TwtSarc2014 46.7 8LvJn2014 74.1 1Twt2015 66.2 2Twt2016 63.0 2
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 18 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Analysis (3)
What is the impact of the “embedding fusion" approach?▶ Was best choice on competition data▶ Can be outperformed by late fusion on some corpora
Corpus Early Late EmbeddingTwt2013 69.4 70.4 70.6SMS2013 62.6 63.7 63.4Twt2014 73.4 74.3 74.4TwtSarc2014 46.2 44.7 46.7LvJn2014 74.3 74.4 74.1Twt2015 64.1 66.8 66.2Twt2016 61.6 62.8 63.0
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 19 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Conclusion and outlook
Sentiment analysis▶ Determine polarity of tweets▶ Has applications in politics, marketing, etc.
Approach▶ Train convolutional neural networks to predict polarity labels▶ Multi-view learning with different word embeddings▶ Embedding fusion
Scientific competition▶ Ranked 2nd at Semeval 2016
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 20 / 21
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Word embedding extensionsDependency embeddings (Levy et al, 2014)
▶ Use dependency tree instead of context window▶ Represent word with dependents and governor▶ Makes much more syntactic embeddings
Multilingual embeddings (Hermann et al, 2014)▶ Train bag-of-word autoencoder on bitexts
⋆ Force sentence-level representations (bag-of-words) to be similar▶ A topic classifier performs without degradation in the other language
Multi-prototype embeddings (Huang et al, 2012; Liu et al, 2015)▶ Each word shall have one embedding for each of its senses▶ Hidden variables: a word has n embeddings▶ Can pre-process with topic tagging (LDA)
Task-adapted embeddings (Socher et al)▶ Combine word-level embeddings▶ Follow parse tree, learn constituent-specific combiners▶ Sentence representation is supervised by task (Sentiment analysis)
B. Favre, M. Rouvier, F. Bechet (AMU) Sentiment Analysis April 20, 2016 21 / 21
![Numismatique des villes de la Phénicie. [4]: Dora ; Enhydra ; Marathos ; Orthosia / [J. Rouvier]](https://img.pdfslide.net/doc/110x75/55345a78550346457f8b4ab2/numismatique-des-villes-de-la-phenicie-4-dora-enhydra-marathos-orthosia-j-rouvier.jpg)
![Numismatique des villes de la Phénicie. [5]: Ptolemaïs-Acé / [J. Rouvier]](https://img.pdfslide.net/doc/110x75/544cc283b1af9fcd3b8b47be/numismatique-des-villes-de-la-phenicie-5-ptolemais-ace-j-rouvier.jpg)
