Harnessing Ratings and Aspect-Sentiment to Estimate ...Introduction Overview Detection and intensity of contradiction Experimental evaluation Conclusion Overview Contradiction and

IntroductionOverview

Detection and intensity of contradictionExperimental evaluation

Conclusion

Harnessing Ratings and Aspect-Sentiment toEstimate Contradiction Intensity in

Temporal-Related Reviews

Ismail Badache - Sébastien Fournier - Adrian-Gabriel Chifu

Pré[email protected]

Laboratoire des Sciences de l’Information et des SystèmesAix-Marseille Université

Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 1 / 24



Conclusion

Plan

Introduction

Overview

Detection and intensity of contradiction

Experimental evaluation

Conclusion




Conclusion

Introduction• The diversity of opinions on a given topic ) Contradiction




Conclusion

Hypotheses

Hypothesis 1 : Reviews are related in timeResource can be updated (e.g. corrected), and these updates willbe made after each session for the case of MOOCs (Massive OpenOnline Courses) that are particularly the subject of our experiment.After each session, users stop reviewing (silence) until the nextsession. Therefore, temporal-related reviews mean the reviewsgenerated during a specific period (called in this paper : session).

Hypothesis 2 : ContradictionA contradiction in reviews related to a web resource meanscontradictory opinions expressed about a specific aspect, which is aform of diversity of sentiments around the aspect for the sameresource.




Conclusion

Hypotheses

Hypothesis 3 : Contradiction intensityAn aspect with a negative sentiment in a review with a positiverating (and vice-versa) has a more important impact on thecontradiction intensity than an aspect with a positive sentiment ina review with a positive rating.




Conclusion

Research Questions

• How to identify a contradiction in reviews ?• How to estimate contradiction intensity between reviews ?• What is the impact of the joint consideration of polarity and

rating of the reviews on the measurement of the intensity ofcontradiction ?




Conclusion

Overview

Contradiction and Controversy Detection• Wikipedia (Wang et al., 2014), News (Tsytsarau et al., 2014),

Debates analysis (Qiu et al., 2013) or generically on the Web(Jang et Allan, 2016).

Aspects Detection• Using HMM (Hidden Markov Models) or CRF (Conditional

Random Fields) as in (Hamdan et al., 2015).• Unsupervised (Kim, 2013), Statistics rules (Poria, 2014).

Sentiment Analysis• Lexicon (Turney, 2002) or Corpus (Mohammad et al., 2013).• Naive Bayes (Pang et al., 2002), RNN (Socher et al., 2013).




Conclusion

How to detect contradiction ?




Conclusion

Clustering reviews (Session)




Conclusion

Identification of aspects

1 Terms frequency calculation of the reviews corpus,2 Terms categorization (part-of-speech tagging) of reviews using

Stanford Parser

1,3 Selection of terms having nominal category without

considering stopwords,4 Selection of nouns with emotional terms in their

five-neighborhoods (using SentiWordNet

2 dictionary),5 Extraction of the most frequent (used) terms in the corpus

among those selected in the previous step. These terms will beconsidered as aspects.

1. http://nlp.stanford.edu:8080/parser/

2. http://sentiwordnet.isti.cnr.it/


http://nlp.stanford.edu:8080/parser/

http://sentiwordnet.isti.cnr.it/



Conclusion

Identification of aspects : example

Step Description

(1) course : 44219, material : 3286, assignments : 3118, content : 2947,lecturer : 2705, ....... terme

i

(2)

The/DT lecturer/NN was/VBD an/DT annoying/VBG speaker/NNand/CC very/RB repetitive/JJ ./. I/PRP found/VBD the/DT format-

ting/NN so/RB different/JJ from/IN other/JJ courses/NNS I/PRP’ve/VBP taken/VBN ,/, that/IN it/PRP was/VBD hard/JJ to/TOget/VB started/VBN and/CC figure/VB things/NNS out/RP ./.

(3) lecturer, speaker, formatting, things(4) lecturer, speaker(5) lecturer

The useful aspect is "lecturer"




Conclusion

Detection of sentiment

Definition 1 : SentimentThe sentiments are a real number in the range [�1, 1] whichindicates the polarity of the opinion expressed in the review segmentwith respect to an aspect (called review-aspect ra). Negative andpositive values respectively represent negative and positive opinions.

Sentiment analysis model :

• Supervised model based on Naive Bayes.• Negation handling (word preceded by "no", "not", "n’t").• Intensifier and adverb processing (very, absolutely).




Conclusion

Measure of contradiction

Definition 2 : ContradictionThere is a contradiction on an aspect between two segments ofreviews ra

i

containing this aspect (ra1, ra2 2 D), where thepolarities around the aspect are opposite (pol(ra1) \ pol(ra2) = �).

•pol(ra

i

) represents the function that returns the polarity(positive, negative) of review-aspect ra

i

.

Intensity of contradiction :

• Dimensions (poli

, rati

) for each review-aspect rai

.• Dispersion of ra

i

presented by a cloud of points.• The greater the distance between the points ra

i

with respectto a centroid ra

centroid

, the greater the degree of contradictionis important.




Conclusion


•Function of dispersion :

Disp(rapolirat

i

,D) =1n

nX

i=1

Distance(poli

,drati

) (1)

with :

Distance(poli

,drati

) =q(pol

i

� pol)2 + (drati

� rat)2 (2)

• Normalisation of ratings drat

i

= rat

i

�32 (drat

i

2 [�1, 1]).•Distance(pol

i

,drati

) is the distance between the point rai

ofthe cloud and the centroid ra

centroid

, and n is the number ofpoints ra

i

of the cloud.Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 14 / 24



Conclusion


Figure – Dispersion of reviews-aspect ra

i

in the cloud (plane)




Conclusion

Measure of contradictionThe coordinates (pol , rat) of the centroid ra

centroid

can becalculated in two different ways :

1 Centroid based on average of pol

i

and rat

i

pol=pol1+pol2+...+pol

n

n

; rat=drat1+d

rat2+...+drat

n

n

(3)

2 Centroid based on the weighted average of pol

i

and rat

i

pol =c1 · pol1 + c2 · pol2 + ...+ c

n

· poln

n

(4)

rat =c1 · drat1 + c2 · drat2 + ...+ c

n

· dratn

n

(5)with :

c

i

=|pol

i

�drat

i

|2n

(6)Ismail Badache KES 2017 Contradiction Intensity Estimation in Reviews 16 / 24



Conclusion

Objectives and dataset

• Evaluating the impact of sentiment analysis and rating oncontradiction detection in reviews around certain aspects.

• Evaluating the impact of the averaged and weighted centroidon the contradiction intensity.

•Dataset : extracted from "coursera.org"

Field Total Number

Courses 2244

Courses Rated 1115

Reviews 73873

Ratings 298326

Reviews 1705

Reviews 1443

Reviews 3302

Reviews 12202

Reviews 55221




Conclusion

Extracted aspects

• 22 aspects identified and extracted automatically from thereviews of "coursera.org"




Conclusion

User Study

•Manual evaluation (contradictions and sentiments) :

•3 assessors.

•10 courses for each aspect.

•1320 reviews/aspect of 220 courses.

•Kappa Cohen (contradictions) : k = 0.68.

•Kappa Cohen (sentiments) : k = 0.76.




Conclusion

Evaluation protocol

• Sentiments analyzer (Naive Bayes) :•

Training set : 50.000 reviews of IMDb

3.

•Test set : the reviews-aspect of coursera.

•Accuracy : 79% (error rate 21%).

• Assessors’ judgments on sentiments are considered as perfect(reference) results and represent an accuracy of 100%.

•Evaluation of the performance of our approach :

•Correlation study (official measure on SemEval tasks

4).

•Using the correlation coefficients of Pearson and Spearman

between the contradiction judgments given by the assessors

and our obtained results.

3. http://ai.stanford.edu/~amaas/data/sentiment/

4. http://alt.qcri.org/semeval2016/task7/


http://ai.stanford.edu/~amaas/data/sentiment/

http://alt.qcri.org/semeval2016/task7/



Conclusion

Results

Figure – Correlation between contradiction judgments and the results of

our approach (with sentiment analysis accuracy of 79%)




Conclusion

Results

Figure – Correlation between contradiction judgments and the results of

our approach (with sentiment analysis accuracy of 100%)




Conclusion

Conclusion

•Contribution : Estimating the intensity of contradiction

•Joint exploitation of polarities and ratings.

•Centroid calculation in 2 ways (averaged and weighted).

•limits :

•Dependence on the quality of sentiment analysis and aspect

detection models.

•Simplicity of pre-processing models.

•The sentences are not processed, only predefined window of 5

words before and after the aspect is considered.

•Perspectives :

•Improving the analysis of sentiments, aspects and sentences.

•Taking into account the profile of the user.

•Further scale-up experiments on other types of datasets are

also envisaged.




Conclusion


Documents

Harnessing Ratings and Aspect-Sentiment to Estimate ...Introduction Overview Detection and intensity of contradiction Experimental evaluation Conclusion Overview Contradiction and