Consumer sentiment analysis with Twitter

  • Published on

  • View

  • Download

Embed Size (px)


Consumer sentiment analysis with Twitter. Reetta Suonper August 2013. Two months , one csv.gz file per day In total about 1.2 billion tweets - PowerPoint PPT Presentation


Using Social Media Data to Construct a Consumer Sentiment Index for Ireland

Consumer sentiment analysis with TwitterReetta SuonperAugust 2013My datasetTwo months, one csv.gz file per dayIn total about 1.2 billion tweetsIt's always easy for a person to say get over, but you don't feel what heart feels to make that statment|PrettynPinkC215|2011-02-01T04:01:16Z|2011-02-01T04:00:48Z|1296532876139018784|The tools I useGeneral approach: natural language processing (NLP)The Natural Language Toolkit (NLTK)Introduction: the consumer sentiment indexA survey-based indicator of consumer confidence or sentiment

History goes back to 1946 at University of MichiganIrelands consumer sentiment index by the ESRI since 1996ESRI survey questionsQ1:Economic situation in the country (next 12 months)Q2:Unemployment in the country (next 12 months)Q3:Household financial situation (12 months ago)Q4:Household financial situation (next 12 months)Q5:Good/bad time to buy large household itemsAnswers: positive/neutral/negativeThis is what it looks like:The KBC/ESRI consumer sentiment index

We can speculate on what drives sentiment but we cant really knowOn the June 2013 improvement in households assessment of their personal finances:We think that the ECB rate cut in May played some role a combination of lowinflation, early summer sales and increasing signs of improvement in the residential property market could have contributed

On the decline in the July 2013 index:We think reports that the Irish economy had fallen back into recession and a couple of high profile job loss announcements unnerved consumers last month.Motivation: why using Twitter could helpMore timelyContinuous informationSave moneyWhat drives sentimentPrevious researchOConnor et al (2010): From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series An index based on tweets containing the word jobs correlates with the Michigan index and Gallups daily pollIndices with economy or job correlate poorly!The process (simplified)

Initial wordlist topicsGeneral economic situationUnemployment/employmentHousehold financial situationBuying climate major hh itemsGeneral economyJob lossesGeneralAcquire/buyGood timesJob gainsIncomeCostBad timesCreditPriceyEcon policyFeeling brokeBargainFeeling flushUsing WordNet to expand seed wordlistUse WordNet to find synonyms for initial keyword list:Words have many different meaningsInclude part-of-speech tagWord doesnt exist in WordNet?Output does not include tenses or plurals

Pre-processing tasksRegular expressions for more basic tasks:Cleaning, tokenising URLs, usernames

NLTK functionality for more complex tasksStopword removal, stemming, POS-taggingFine selection not there yetDo more filtering using bigrams?I broke pay cutnew jobUse POS tags?Classification?

Finalise fine selectionSentiment classificationVisualisation

The to-do listwww.nltk.orgNatural Language Processing with Python: Text Processing with NLTK 2.0 CookbookResourcesResourcesOConnor et al (2010): From Tweets to Polls: Linking Text Sentiment to Public Opinion Time SeriesBollen et al (2011): Twitter mood predicts the stock marketBollen et al (2011): Modeling public mood and emotion: Twitter sentiment and socio-economic phenomenaGo et al (2009): Twitter sentiment classication using distant supervisionJiang et al (2011): Target-dependent Twitter Sentiment Classification



View more >