AUTOMATIC TEXT SUMMARIZATION
By
Chetana GavankarSubhabrata MukherjeeKedharnath NarahariSarbartha Sengupta
under guidance of:Prof Pushpak Bhattacharya
PRESENTATION CONTENT
• Motivation• Types of summaries• Challenges• Single-Document Summarization o Early worko Machine Learning Methods
Supervised Methods Unsupervised Method
o Deep Natural Language Analysis methods• Multi-Document Summarization• Evaluation• Conclusion
MOTIVATION
• Download 1000 + papers and get the summary..
• You have list of emails about sports event get the summary of those emails in one para…
• You have to study loads of books for the exam and the summarizer gives the key concepts of the books as few pages notes…
Value for researchers• Get me everything Papers say about
“Automatic Text Summarization”
MOTIVATION
DEFINITION
Automatic Summaries
• Should be less than half of original text• Should convey important information• May be produced from single or multiple
documents (Radev et al)
Dipanjan Das, Andre F.T. Martins (2007). A Survey on Automatic Text Summarization. Literature Survey for the Language and Statistics II course at CMU, Pittsburg
APPLICATIONS – MOVIE REVIEWS
TYPES OF SUMMARIES
With respect to content: Indicative: provide an idea what the text is about, but
do not render the content Informative: shortened versions of the text
With respect to the way of creating: Extracts: identify important sections of the text Abstracts: produce important material in a new way
Dipanjan Das, Andre F.T. Martins (2007). A Survey on Automatic Text Summarization. Literature Survey for the Language and Statistics II course at CMU, Pittsburg
TYPES OF SUMMARIES With respect to Input
• Restricted vs. Unrestricted domain• Single-document vs. Multiple-document
With respect to Purpose• Generic vs. Query based• Background vs. just-the-newsnd vs. just-the-news
Eduard Hovy and Lin C. Y. "Automated Text Summarization in summarist", MIT Press
TYPES OF SUMMARIES With respect to Input
• Restricted vs. Unrestricted domain• Single-document vs. Multiple-document
With respect to Purpose• Generic vs. Query based• Background vs. just-the-news
Eduard Hovy and Lin C. Y. (1998 )"Automated Text Summarization and the summarist system", TIPSTER III Final Report (SUMMAC)
TOOLS – WORD SUMMARIZER
Microsoft Word 2007 - AutoSummarize
TOOLS – GNOME SUMMARIZER
TOOLS – SWESUM SUMMARIZER
http://www.csc.kth.se/~xmartin/swesum_lab/index-eng.html
CHALLENGES
• Selecting pieces from the input and concatenating them to yield a summary
• High reduction rates like headline
• Methods for evaluating summaries
• Multiple languages • Multiple Hybrid sources
Hahn U. and Mani I. (2000) “The Challenges of Automatic Summarization”, Computer, IEEE Computer Society
14
EARLY WORK
ATS has its roots in the last 50’s and has been developed continuously…
• A word frequency based ATS [Luhn, 1958].• An ATS based on multiple features [Edmundson,
1969].• ……..• ……. Still unsolved !
WORD FREQUENCY• Content words indicate
topic of a text• Frequency of a content
word – measure of its significance
• Retrieve the top n frequent occurring content words
• Rank a sentence according to the frequency of those words present in it
WORDS
FREQUENCY
E
Resolving power of significant words
Luhn, H. P. (1958). The automatic creation of literature abstracts. IBM Journal of Research Development, 2(2):159-165
Position of a Sentence
• Sentences occurring under certain headings are positively relevant
• Topic sentences tend to occur very early or very late in a document and its paragraphs
• Optimum Position Policya ranked list that indicates in what ordinal positions in the text the high-topic-bearing sentences tend to occur. (Lin and Hovy, 97).
Ex: [T1, P1S1, P1S2, ...] for a News Article
Edmundson, H. P. (1969). New methods in automatic extracting. Journal of the ACM, 16(2): 264-285
Cue words in a Text
Probable relevance of a sentence affected by Cue words:
Bonus words: positively affecting the relevance of a sentence (e.g. “Significant”, “Greatest”)Stigma words: negatively affecting the relevance of a sentence (e.g. “Impossible”, “Hardly”)Null words: irrelevant
Edmundson, H. P. (1969). New methods in automatic extracting. Journal of the ACM, 16(2): 264-285
NAÏVE BAYES
• Let s be a particular sentence, S the set of sentences making up the summary and F1, … Fk be the set of features
• Assume feature independence
– Additional Features• Sentence Length• Presence of Uppercase Words
– Position, Cue features with Sentence Length performed best
)(
)().|(),...,|(
1
121
iki
ikik
FP
SsPSsFPFFFSsP
Kupiec, J., Pedersen, J., and Chen, F. (1995). A trainable document summarizer, In Proceedings SIGIR '95, pages 68-73
Naïve Bayes Contd…
• Richer features– Tf-idf to derive signature words– Named Entity Tagger to retrieve tokens– Shallow Discourse Analysis to maintain cohesion– Synonym and Morphological variants of lexical
terms merged using WordNet.
Aone, C., Okurowski, M. E., Gorlinsky, J., and Larsen, B. (1999), A trainable summarizer with knowledge acquired from robust nlp techniques, In Mani, I. and Maybury, M. T., editors, Advances in Automatic Text Summarization, pages 71-80
Decision Tree• Feature independence assumption not valid in real
world situation• Creation of feature vector
– Baseline– Title– tf & tf-idf scores– Position score– Query Signature– IR Signature– Sentence Length– Average Lexical Connectivity– Numerical Data– Proper Name– Pronoun & Adjective– Weekday & Month– Quotation– First Sentence
Lin, C.-Y. (1999). Training a selection function for extraction, In Proceedings of CIKM '99, pages 55-62
Decision Tree Contd…
• Scores of all the features combined by automated learning process using decision tree and normalized
• Remarks– Decision Tree performs best over all dataset– Naïve combination beats Decision Tree in 3
topics
• Possible Reason ????• Features were Independent
Hidden Markov Model
• Drawbacks of earlier approches– Feature based bag-of-words model– Non-sequential
• Use sequential model to account for local dependencies between sentences
• Features– Position of sentence in the document– Number of terms in the sentence– Likeliness of the sentence terms given the
document terms
Conroy, J. M. and O'leary, D. P. (2001). Text summarization via hidden markov models, In Proceedings of SIGIR '01, pages 406-407
Hidden Markov Model Contd…
• 2s+1 states alternating between s summary states & s+1 non-summary states
• Odd state summary state, Even state non-summary state
• Transition Matrix M whose element (i,j) is the probability of transition from state i to j
• Output function bi(o)= Pr(O|state i) where O is an observed vector of features
• Assumption : features are multivariate normal
• M & O learnt from training data
Log-Linear Models
• Let c be a label, d the item we are interested in labeling, fi the ith feature and λi the corresponding feature weight
• Z(d) = Ʃcexp(Ʃi λifi(c,d)) is the normalization constant
• fw,c’ (d,c)= 0 c≠c’ 1 c=c’• Larger value of λi means fi is a strong indicator of class c• GIS, IIS used to iteratively tune model parameters.• Outperformed Naïve Bayes• DRAWBACK : Overfitting
)()|(
),(,1
,
dZ
edcp
cdf cjk
jcj
Osborne, M. (2002). Using maximum entropy for sentence extraction. In Proceedings of the ACL'02 Workshop on Automatic Summarization
Drawbacks of Earlier Methods
1.Performance might degrade, if the text consists of multiple topics.
2.Anaphors in the extracted sentences might not have any antecedent in the summary.
3.The summary might be incoherent, since the sentences are just extracted from various parts of the text.
Drawbacks Contd …
• Consider the following two sequences:
• 1. “Dr.Kenny has invented an anesthetic machine. This device controls the rate at which an anesthetic is pumped into the blood.”
• 2. “Dr.Kenny has invented an anesthetic machine. The doctor spent two years on this research.”
• “Dr.Kenny” appears once in both sequences and so does “machine”. But sequence 1 is about the machine, and sequence 2 is about the “doctor”.
Cohesion
• Cohesion ( Halliday & Hasan 1976)– “stitching together” different parts of the text– Use of semantically related terms, co-reference,
ellipsis, conjunctions
• Lexical Cohesion – Semantically related words– Reiteration category
• Repetations, synonyms, hyponyms– Collocation category
• Words occurring in same lexical context– Ex: She works as a teacher in the school.
Lexical Chain General Approach• 1. Select a set of candidate words;• 2. For each candidate word, find an appropriate chain
relying on a relatedness criterion among members of the chains;
• 3. If it is found, insert the word in the chain and update it accordingly.
• Relations– Extra-strong (between word & its repetation)
• No restriction– Strong (between 2 words connected by WordNet reln)
• Window of 7 sentences– Medium-strong (path length > 1 hop )
• Window of 3 sentences• Preference : Extra-strong > Strong > Medium-strong
Drawback
• Mr. Kenny is the person that invented an anesthetic machine which uses micro-computers to control the rate at which an anesthetic is pumped into the blood. Such machines are nothing new. But his device uses two micro-computers to achieve much closer monitoring of the pump feeding the anesthetic into the patient. (Morris & Hirst 1991)
• [lex "Mr.", sense {mister, Mr.}]• [lex "person", sense {person,
individual,someone, man, mortal, human, soul}].
• First sense of machine in WordNet – “an efficient person” – a holonym of “person” and thus wrongly disambiguated
Component and Graph Connectivity
Barzilay, R. and Elhadad, M. (1997). Using lexical chains for text summarization. In Proceedings ISTS'97
Component & Graph Connectivity Contd…
Multi Document Summarization
• Multiple sources of information– Similarity between topics– Supplement each other– Occasionally contradictory
• Key Tasks– Identifying Key concepts across documents– Coping with redundancy– Ensuring final Summary is coherent and
complete
• Applications: news clustering systems– Google News, Columbia NewsBlaster, News in
Essence etc
TOPIC-DRIVEN SUMMARIZATION
Carbonell, J. and Goldstein, J. (1998). The use of MMR, diversity-based re-ranking for reordering documents and producing summaries.
TOPIC DRIVEN SUMMARIZATION(contd.)
Carbonell, J. and Goldstein, J. (1998). The use of MMR, diversity-based re-ranking for reordering documents and producing summaries. In proceedings of SIGIR '98.
GRAPH SPREADING
Mani, I. and Bloedorn, E. (1997). Multi-document summarization by graph search and matching. In AAAI/IAAI, pages 622-628.
Example for nodes and links in the graph(mani and bloerdon 97’)
Mani, I. and Bloedorn, E. (1997). Multi-document summarization by graph search and matching. In AAAI/IAAI, pages 622-628.
GRAPH SPREADING(Contd..)
Words and phrases are intialized according to their TF-IDF scores.
For each sentence in both documents, two scores are computed.
One score that reflects the presence of common nodes, which is computed as the average weight of these nodes;
Other score that computes instead the average weights of difference nodes.
the sentences that have higher common and different scores are highlighted and accordingly the ouput is generated.
CENTROID-BASED SUMMARY Does not make use of language generation
model. Documents are modeled as bag-of-words.Topic Detection: Clustering algorithm that uses TF-IDF vector
repesentations of documents Successively add documents and recomputes
centroids.
Centroids are pseudo documents that include words with TF-IDF score above some threshold.
d’-truncated documentC j – jth cluster
Radev, D. R., Jing, H., and Budzikowska, M. (2000). Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies.
CENTROID BASED SUMMARIZATION
Sentence Identification: 2metrics o Cluster Based Relative Utility-how relevant a
sentence is to particular topic of the cluster.o Cross Sentence Informational Subsumption-
measure of redundancy among sentences.
Radev, D. R., Jing, H., and Budzikowska, M. (2000). Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In NAACL-ANLP 2000 Workshop on Automatic summarization, pages 21-30, Morristown, NJ, USA.
MULTILINGUAL MULTI-DOCUMENT SUMMARY
Target language(English) in which summary is written.
Source documents present in both preferred language and foreign language(Arabic).
Use IBM’s translational model to translate documents in source language to target language.
Check for similarity between translated sentences in two documents.
If similarities found, retain documents in source language, since they can be more grammatically correct.
Evans, D. K. (2005). Similarity-based multilingual multi-document summarization. Technical Report CUCS-014-05, Columbia University.
SHORT SUMMARIES
Witbrock, M. J. and Mittal, V. O. (1999). Ultra-summarization (poster abstract): a statistical approach to generating highly condensed non-extractive summaries.
Witbrock, M. J. and Mittal, V. O. (1999). Ultra-summarization (poster abstract): a statistical approach to generating highly condensed non-extractive summaries. In Proceedings of SIGIR '99, pages 315{316, New York, NY, USA.
SENTENCE COMPRESSION Compression of sentence used for
summarization. Uses noisy-channel model which considers
that one starts with a short summary s, according to source model P(s).
Subjected to noisy-channel to make full sentence t, in a process guided by channel model, P(t/s).
Now observing t, recover the summary as:s’=argmaxs P(s/t)= argmaxs P(s).P(t/s)
Advantage of decoupling the goals of grammatical correctness and preserving important information.
Knight, K. and Marcu, D. (2000). Statistics-based summarization - step one: Sentence compression. In AAAI/IAAI, pages 703-710
Knight, K. and Marcu, D. (2000). Statistics-based summarization - step one: Sentence compression. In AAAI/IAAI, pages 703-710.
Evaluation
• Difficult task. (There does not exist an ideal summary for a given document or set of document.)
• Agreement between human summarizers is quite low.
• Difficult to evaluate the summary content.
• Absence of a standard human or automatic evaluation metric.
Evaluation
• Lin and Hovy (2002).
describe and compare various human and automatic metrics to evaluate summaries.
Focus on the evaluation procedure used in the Document Understanding Conference 2001(DUC-2001).
compared manually written ideal summaries with summaries generated automatically by summarization systems and baseline summaries.
Lin, C.-Y. and Hovy, E. (2002). Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Summarization, pages 45-51
Evaluation
• Lin and Hovy (2002). Each text was decomposed into a list of
units (sentences).
stepped through each model unit (MU) from the ideal summaries.
marked all system units (SU) sharing content with the current model unit.
All (4) Most (3) Some (2) or Hardly any (1)
Grammaticality, cohesion, and coherence were also rated
Evaluation
• Lin and Hovy (2002).
The weighted recall at threshold ‘t’ (t=1 to 4).
outline an accumulative n-gram matching score (which they call NAMS),
Taken from: Dipanjan Das, Andre F.T. Martins, A Survey on Automatic Text Summarization2007
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).
Let be a set of reference summary, and let be a summary generated automatically by a system. Let be a binary vector representing n-grams contained in a document d.
The metric ROUGE-N is an n-gram recall based statistic
where denotes the usual inner product of vectors
Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Marie-Francine Moens, S. S., editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004). Example:
d := “Ram helped Shyam to learn German.”r1:= “Shyam learnt German by Ram.”r2:= “Ram taught German to Shyam.”s := “Shyam learn German from Ram.”
Φn(r1) := [10101] Φn(r2) := [10111]Φn(s) := [11101]
= 6/7
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).
ROUGE-N can be used for multiple reference summaries.
An alternative is taking the most similar summary in the reference set.
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004). Example:
d := “Ram helped Shyam to learn German.”r1:= “Shyam learnt German by Ram.”r2:= “Ram taught German to Shyam.”s := “Shyam learn German from Ram.”
Φn(r1) := [10101] Φn(r2) := [10111]Φn(s) := [11101]
= 3/3
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).
Another metric in (Lin, 2004) applies the concept of longest common subsequence (LCS).
Let r1,…,ru be the reference sentences of the documents in R, and s a candidate summary
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004). Example:
s := “Ram pushed Shyam into the Pool.”r1:= “Ram push Shyam into the Pool.”r2:= “Shyam push Ram into the Pool.”
ROUGE-N: r1=r2 (“Ram”, “Shyam into the Pool”)
ROUGE-L:r1: = 5/6 (“Ram”, “Shyam into the Pool”)r2: = 4/6 (“Ram”, “into the Pool”)r1>r2
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).
Yet another measure ROUGE-S, which can be seen as a modified version of ROUGE-N for n = 2
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004). Example: s := “Ram pushed Shyam into the Pool.”r1:= “Ram push Shyam into the Pool.”r2:= “Shyam push Ram into the Pool.”r3:= “Shyam into the Pool Ram pushed.”
ROUGE-N: r3>r1=r2ROUGE-L: r2>r3=r4ROUGE-S:r1: = 10/15 r2: = 9/15 r3: = 7/15r1>r2>r3
Evaluation
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).
The various versions of ROUGE were evaluated by computing the correlation coefficient between ROUGE scores and human judgment scores
Conclusion
• a need to develop efficient and accurate summarization systems.
• attention has drifted from summarizing scientic articles to news articles, electronic mail messages, advertisements, and blogs.
• Both abstractive and extractive approaches have been attempted.
• simple extraction of sentences have produced satisfactory results in large-scale applications.
• This survey emphasizes extractive approaches to summarization using statistical methods.
References• Primary Reference
o Dipanjan Das, Andre F.T. Martins (2007). A Survey on Automatic Text Summarization. Literature Survey for the Language and Statistics II course at CMU, Pittsburg
• Secondary Referenceo Edmundson, H. P. (1969). New methods in automatic extracting. Journal
of the ACM, 16(2):264-285o Luhn, H. P. (1958). The automatic creation of literature abstracts. IBM
Journal of Research Development, 2(2):159-165o Lin, C.-Y. (1999). Training a selection function for extraction. In
Proceedings of CIKM '99, pages 55-62o Ono, K., Sumita, K., and Miike, S. (1994). Abstract generation based on
rhetorical structure extraction. In Proceedings of Coling '94, pages 344-348
o Barzilay, R. and Elhadad, M. (1997). Using lexical chains for text summarization. In Proceedings ISTS'97
o Mani, I. and Bloedorn, E. (1997). Multi-document summarization by graph search and matching. In AAAI/IAAI, pages 622-628
References Contd…• Evans, D. K. (2005). Similarity-based multilingual multi-document
summarization Technical Report CUCS-014-05, Columbia University• Radev, D. R., Jing, H., and Budzikowska, M. (2000). Centroid-based
summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In NAACL-ANLP 2000 Workshop on Automatic summarization, pages 21-30
• Witbrock, M. J. and Mittal, V. O. (1999). Ultra-summarization (poster abstract): a statistical approach to generating highly condensed non-extractive summaries. In Proceedings of SIGIR '99, pages 315-316
• Knight, K. and Marcu, D. (2000). Statistics-based summarization - step one: Sentence compression. In AAAI/IAAI, pages 703-710
• Lin, C.-Y. and Hovy, E. (2002). Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Summarization, pages 45-51
• Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Marie-Francine Moens, S. S., editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81
• Lin, C.-Y., Cao, G., Gao, J., and Nie, J.-Y. (2006). An information-theoretic approach to automatic evaluation of summaries. In Proceedings of HLT-NAACL '06, pages 463-470