View
36
Download
0
Category
Preview:
DESCRIPTION
Exploiting Subjectivity Classification to Improve Information Extraction. Ellen Riloff University of Utah Janyce Wiebe University of Pittsburgh William Phillips University of Utah. Subjectivity ?. - PowerPoint PPT Presentation
Citation preview
Exploiting Subjectivity Classificationto Improve Information Extraction
Ellen Riloff University of Utah
Janyce Wiebe University of Pittsburgh
William Phillips University of Utah
Subjectivity ?
• Definition: Subjective language expresses or refers to opinions, emotions, sentiments and other private states.
• Related Work: – Sentiments (Turney & Littman 2003; Dave, Lawrence, &
Pennock 2003; Pang & Lee 2004)
– Product Reputation Tracking (Morinaga et al. 2002; Yi et al. 2003)
– Opinion Oriented Summarization and QA (Hu & Liu 2004; Yu & Hatzivassiloglou 2003)
• Opinion - personal beliefs
• Emotion - state of mind
• Sentiments - positive/negative judgements
Motivation
• Our observation: many false hits produced by Information Extraction (IE) systems come from subjective sentences.
• Hypothesis: we can improve IE performance by avoiding extractions from subjective sentences.
Examples
“D’Aubruisson unleashed harsh attacks on Duarte…”
“The Parliament exploded into fury against the government when word leaked out…”
“The subversives must suspend the aggression against the people and the destruction of the economy…”
The Big Picture
SubjectiveSentenceClassifier
subjectivesentences
objectivesentences
FullInformationExtraction
SelectiveInformationExtraction
The Subjectivity Classifier
• Most documents contain a mix of subjective and objective sentences– 44% of sentences in newspaper articles subjective!
(Wiebe et al. 2004)
• We used the Naïve Bayes subjective sentence classifier developed by Wiebe & Riloff [2005].
– Classifies at sentence level– unsupervised
– rivals best supervised methods
Initial Training Data Creation
rule-based subjectivesentenceclassifier
rule-basedobjectivesentenceclassifier
subjective & objective sentences
unlabeled texts
subjective clues
Naïve Bayestraining
POSfeatures
subjectiveclues
Naïve Bayes Training
extractionpattern learner
training set
objectivepatterns
subjectivepatterns
Naïve BayesClassifier
NB Confidence Measure
CM =
MUC-4 IE Task
• To extract information about terrorist events in Latin America.
• Evaluated performance on 4 types of information:– perpetrators (individuals), victims, targets, weapons
• Corpus: 1700 texts– 1400 used for training, 100 for tuning, 200 for testing
• Used Autoslog-TS to generate extraction patterns– system used 397 patterns
Base IE System Performance
System Rec Prec F #Correct #Wrong
IE .52 .42 .47 266 367
Filtering Subjective Sentences
System Rec Prec F #Correct #Wrong
IE .52 .42 .47 266 367IE+SubjFilter .44 .44 .44 218 (-48) 273 (-94)
Source Attribution Sentences
• In news articles, factual information is often prefaced with a source attribution. Examples:
“The Associated Press reported…” “The President stated…”
• Source attribution sentences often contain important facts even if subjective language is also present.
Source Attribution Modification
• Keep the subjective sentences if they contain a source attribution.
1) the sentence contains a communication verb:
{affirm, announce, cite, confirm, convey, disclose, report,
tell, say, state }
2) the subjectivity classifier considers the sentence to be only
weakly subjective (CM 25)
Results with Source Attribution Modification
System Rec. Prec. F #Correct #Wrong
IE .52 .42 .47 266 367
IE+SubjFilter .44 .44 .44 218(-48) 273(-94)
IE+SubjFilter2 .46 .44 .45 231(-35) 289(-78)
Selective Filtering
• We observed that subjective sentence can contain important facts. For example:
“He was
outraged by the terrorist attack on the World Trade Center.”
• Modification: selectively extract information from subjective sentences
• Done using Indicator Patterns.
Indicator Patterns
• We defined an indicator pattern as a pattern that has the following Autoslog-TS statistics :
P(relevant | pattern) 0.65 and Frequency 10
• Indicator Patterns clearly represent a fact of interest– “murder of X” – “X was assassinated”
.
Results for Selective Subjectivity Filtering
System Rec Prec F #Correct #Wrong
IE .52 .42 .47 266 367
IE+SubjFilter .44 .44 .44 218 (-48) 273 (-94)
IE+SubjFilter2 .46 .44 .45 231 (-35) 289 (-78)
IE+SF2+Slct .51 .45 .48 258 (-8) 311 (-56)
Removing Subjective Extraction Patterns
• Example:
“….to destroy the building.”
“…to destroy the process of reconciliation.”
• Use subjectivity analysis to remove subjective patterns.
• We classified a pattern as subjective if:1)
P(subjective | pattern) > .50 and
2) frequency 10
Final Results
System Rec Prec F #Correct #Wrong
IE .52 .42 .47 266 367
IE+SubjFilter .44 .44 .44 218 (-48) 273 (-94)
IE+SubjFilter2 .46 .44 .45 231 (-35) 289 (-78)
IE+SF2+Slct .51 .45 .48 258 (-8) 311 (-56)
IE+SF2+Slct
-SubjEPs .51 .46 .48 258(-8) 305(-62)
Subjectivity Filtering Combined with Topic Classification
System Rec Prec
IE .52 .42IE w/Perfect TC .52 .53IE w/Perfect TC + SubjFilter .51 .56
Conclusions
• Subjectivity filtering strategies improved IE precision with minimal recall loss.
• The benefits of subjectivity classification are synergistic with those of topic classification.
• As subjectivity classification improves, we expect corresponding improvements to IE.
IE Evaluation
Performed at extraction level, before template generation
Standard IE System
texts extracts
Slot Extraction
Component
Template Generation Component
• We defined an indicator pattern as a pattern that has the following Autoslog-TS statistics :
P(relevant | pattern) 0.65 and Frequency 10
• Using only the indicator patterns for IE not sufficient.
Rec Prec FIE .52 .42 .47IE (Indicators Only) .40 .54 .46
IE System
• We used Autoslog-TS to generate extraction patterns.– 40,553 distinct patterns were learned
• We manually reviewed top patterns (2808 patterns)
• The final system used 397 patterns.
Examples of Filtered Extractions
• The demonstrators, convoked by the solidarity with Latin America Committee, verbally attacked Salvadoran President Alfredo Cristiani and have asked the Spanish government to offer itself as a mediator to promote and end to the armed conflict.
PATTERN: attacked <dobj>VICTIM: “Salvadoran President
Alfredo Cristiani”
Examples of Filtered Extractions
• The crime was directed at hindering the development of the electoral process and destroying the reconciliation process…
PATTERN: destroying <dobj>
TARGET: “the reconciliation process”
• Presidents, political and social figures of the continent have said that the solution is not based on the destruction of a native plant but in active fight against drug consumption.
PATTERN: destruction of <np>
TARGET: “a native plant”
Breakdown by Extraction Type
Category Baseline SubjFilter Rec Prec Rec Prec
Perp .47 .33 .45 .38
Victim .51 .50 .50 .52
Target .63 .42 .62 .47
Weapon .45 .39 .43 .42
Total .52 .42 .51 .46
Subjective Patterns
attacks on <np> to attack <dobj>
communique by <np> to destroy <dobj>
<subj> was linked leaders of <np>
<subj> unleashed was aimed at <np>
offensive against <np> dialogue with <np>
The following extraction patterns were classified as subjective:
Metaphor
• False hits can come from subjective sentences that contain metaphorical language.
The Parliament exploded into fury against the government when word leaked out…
Recommended