Analysis of twitter feeds and blogs

Language and Computation Group18th November 2011

Communications of the ACM, October 2011

Conclusion 1

See also: danah boyd, Kate Crawford, “Six Provocations for Big Data”, September2011: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431

Conclusion 2

Conclusion 3

Conclusion 4

Conclusion 5

For quotessee nextslide…

“relatively high amount of hype”“even when the predictions were better than chance, they

were not competent compared to the trivial method of predicting through incumbency.”

“We simply tried to repeat the (reportedly successful) methods that others have used in the past, and we found that the results were not repeatable.”

“Hoping that the errors in sentiment analysis ‘somehow’ cancel themselves out is not defensible.”

“Spammers and propagandists write programs that create lots of fake accounts and use them to tweet intensively, amplifying their message.”

“Predicting elections with accuracy should not be supported without some clear understanding of why it works”.

“Learn from the professional pollsters … identify likely voters and get an unbiased representative sample of them”

Table onnext slide:

Tweets with searching-related terminology:

Examines relationship between emotional reactions and public opinionSeeks to offer insight into how public

opinion is formedBased on analysis of posts from Usenet online

forumEvaluation of emotional content is based on

counting of words in ANEW – Affective Norm for English Words

Nevertheless, this still begs the question of sample biasHow typical are Usenet users of the general

population?

See nextslide…

SPARQL query language use cases:“Give me a stream of locations where my

product is being mentioned right now.”“Give me all people that have said negative

things about my product.”“Give me all URLs that people recommend with

relation to my product.”“What competitors are being mentioned with

my product.”511,147 tweets about iPad (June 3rd – June 8th

2010):http://wiki.knoesis.org/index.php/Twarql

Use of agent-based prediction market

Each agent extracts users sentiments from a different social medium Reflects it beliefs by trading in the marked Belief-Desire-Intentions paradigm Agent will intend to do what it believes and will achieve its goals given its beliefs

about the world Avoids problems with human agents

Poor estimation at either end of probability spectrum Agents do not manipulate the market Do not require recruitment and incentives

Bothos et al, IEEE Intelligent Systems, November/December 2010

See next slide…..

Presents methodology for predicting individual retweets in Twitter

Input to the model is the tweeter, a retweeter and the content of the tweet

Output of model is the probability of a retweet of a tweet by the retweeter

Probabilistic collaborative filtering prediction models used, called Matchbox

Crawled twitter from June 10th 2010 to July 29th 2010, finding 20,000,000 retweets

Top correlations on next slide.....

Understanding of specific groups useful for commercial and political organisations

Four key tasks:Discover or extract the group itselfDevelop a profile from group descriptors

and defining group characteristicsUnderstand group’s sentiment and ability

to influence other individuals or groupsStudy group composition

Privacy and security are important concerns

END OF SLIDES

One reason for low concordance is use of U for you or rly for really. Also, frequent typos,and use of Internet acronyms such as rly for really. Sentence fragments, and pronoundrops such as busy now instead of I’m busy now.