Upload
ahmedali-durga
View
104
Download
1
Embed Size (px)
DESCRIPTION
Citation preview
SUB EVENT DETECTION ON SOCIAL MEDIA
Kshitij Kansal Maaz Anwar Nomani
Ahmed Ali Durga
Information Retrieval and Extraction
INTRODUCTION1. Motivation
• Social Media is filled with a lot of information. • Information is shared much before the news gets displayed on the
news websites. • The information shared captures even the minute details which
news websites might ovelook. • This gives us a lot of scope for early news detection with more
diminutive details.
2. Objective • We aim to propose an automatic method for extracting Sub -
Events in the given Social Media feeds.
SUB EVENT• What is a Sub Event?
• Any kind of information which is small to be conveyed as a part of whole event.
• large enough to affect some appreciably large reader's community. • Includes aftermath of an event, real time notifications, responses,
public sentiments and reports.
• Why Sub Event? • Closely related to a particular commuity. • Can be used to enhance the knowledge of an event. • Can measure the public sentiments along the whole course of
occurance of the event.
OUR EXPERIMENT
• Detecting the "sub events" in the Twitter Stream related to the US Presedential Elections.
• Main Event: US Presidential Elections and the Victory of Barack Obama.
• Sub Events: Victory or defeats of some famous candidates, public sentiments across the course of elections, changes in the stock market as the treds start to pour out etc.
• The approach decided is not specific to this dataset only. This can be applied to any dataset in the form of Twitter stream.
APPROACH
We followed an organised approach where we divide the whole process in the following three sub parts which were dealt with separately and later integrated.
• Tweet filtering and Noise Reduction • Sub Event Detection • Sub Event Summarization
TWEET FILTERING AND NOISE REDUCTION
Aim: To eliminate the useless tweets which do not convey much information regarding the event.
• Tweet Stream provided is cleaned using the self defined filter. • Filter takes into account the linguistic aspects of the language and
context filtering. • Remove Diacritic marks • Consider only ASCII characters • Ignore repeatitions • Ignore Multiple Punctuations • Consider only tweets starting with capitals • Remove extremely small and large tweets
SUB EVENT EXTRACTIONAim: To extract tweets that express some defining moments in the event.
• To be applied on the filtered stream available from the noise reduction module.
• Dictionary of the tweets words and generation of Tweet Vector
• Find the distance between the tweets.
• Group together the similar tweets.
• Chunks of relevant tweets will form the sub events.
• Hashing of the tweet stream to increse the speed of the system
EXTRACTION ...Dictionary Creating and Vector Generation
• Dictionary Creation: • Bag of Word Representation. • Stop Word Removal. • Assign unique ID to the words.
• Vector Generation: • Create the n dimension vector • n is the number of words in the dictionary. • Vector value = 1, if word present • Vector value = 0, if word not present • Create sparse vector for space optimization.
EXTRACTION ...Distance and Similarity Measures
• Euclidean Distance: • Simple distance between the tweet vecors. • Similar to finding distance between the points in n dimension space. • n being the size of Tweet dictionary.
• Similarity Measure: • Calculate the no of similar words in the tweets. • If greter than some threshold, assume them to be similar • Threshold(in our case): 50% of the length of smaller tweet. • Takes into account the length of tweets i.e. Normalization.
• Cosine Similarity: • Similar to above method. • Also takes int account the length i.e. Normalization. • Works by finding out the angle between the two tweets. • Tweets are taken to points in n dimension space.
EXTRACTION ...Hashing
• Increases the speed of retrieval module • Locality Sensitive Hashing
• Dimension Reduction of high dimension data • Maximizes the probability of collision of similar
tweets.
• PyLucene • Python extension for using Java Lucene • Apache Lucene is a free/open source
information retrieval software library
SUMMARIZATION
• Related tweets are extracted and stored in separated files. • Need to make extract the sub event from these related tweets. • Some kind of summarization of the colled tweets is required.
• Summarization needs to be in human readble form. • Should able to convey the happeinings in the sub event. • If possble, crawl data from the URL's in the links and use it for
summarization. • Image support will increase its attractiveness and user
acceptability.
SUMMARIZATION ...
• Important for the end user evaluation. • Thus,Summarization forms the crux of the content defined by a
sub-event. • Two approaches to automatic summarization
• Extraction: Works by selecting a subset of existing words, phrases, or sentences in the original text to form the summary
• Abstraction: build an internal semantic representation and then use natural language generation techniques to create a summary that is closer to what a human might generate
SUMMARIZATION ...
• Spanning Phrase approach is used.
• Took into account the most frequent words in the cluster of tweets and club them.
• Choose two to be the maximum frequency of a word is 'w' ccurring in all the tweets.