Upload
irene-samson
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Twitter Catches The Flu: Detecting Influenza Epidemics
using Twitter
Eiji ARAMAKI * Sachiko MASKAWA *
Mizuki MORITA **
* The University of Tokyo** National Institute of Biomedical Innovation
EMNLP2011
Why we developed this system?
Let me show you several existing systems
Centers for Disease Control and Prevention (CDC)
Infection Disease Surveillance Center (IDSC)
European Influenza Surveillance Network (EISN)
Why each country has each surveillance system?
• Influenza epidemics are a major public health concern, because it causes tens of millions of illnesses each year.
• To reduce the victims, the early detection of influenza epidemics is a national mission in every country.
• BUT: These surveillance systems basically rely on hospital reports (written manually).
Two Problems & Recent Approach
• (1) Small Scale – For example, IDSC gathers influenza patient data from
5,000 clinics. But It does not cover all cities (especially local cities).
• (2) Time Delay (Time lag)– For example, the data gathering process typically has a
1–2 week reporting lag
• To deal with these problems – Recently, various approaches that directly capture
people’s behavior are proposed
Recent Approach
• using Phone Call data– Espino et al. (2003) used data of a telephone
triage service, a public service, to give an advice to users via telephone. They reported the number of telephone calls that correlates with influenza epidemics.
• using Drug sale data– Magruder (2003) used the amount drug sales.
Among various approaches…
The State-of-the-ArtWeb based Approach
• Ginsberg et al. (Nature 2009) used Google web search queries that correlate with an influenza epidemic, such as “flu”, “fever”.
• Polgreen et al. (2008) used a Yahoo! query log.• Hulth et al. (2009) used a query log of a
Switzerland web search engine.
This Study
• Web search query is a extremely large scale and real-time data resource.
• BUT: the query data is closed (not freely available), which is available only for several companies, such as Google, Yahoo, or Microsoft.
→ This study examines Twitter data, which is widely available.
OUTLINE• Background
• Objective
• Method
• Experiment
• Discussion
• Conclusion
Detailed Task Definition
Detailed Task Definition
Simple Word Frequency in Twitter“Cold”, “Fever” & “influenza”
Winter Summer
Simple Word Frequency contains various noisesBecause….
Actual influenza curve is more smooth
Negative Influenza TweetNegative Influenza Tweet
Positive Influenza TweetPositive Influenza Tweet
A word “influenza” does not always indicate an influenza patient
Two types of Influenza Tweets
• Negative influenza tweet indicates an influenza patient
• Negative influenza tweet includes mention of “influenza”, but does not
indicate that an influenza patient is present
• Not only the general news, but also various phenomena generate Negative influenza tweet…
Negative Influenza TweetNegative Influenza Tweet
Positive Influenza TweetPositive Influenza Tweet
Various Negative Influenza Tweet (1/2)
• Prevention – You need to get a influenza shot sometime
soon.
• Modality (just suspition)– @John might be suffering from influenza
• Question– Did you catch the influenza ?
Various Negative Influenza Tweet (2/2)
• Influenza of Cat or Dog– Today, I couldn't go
home late. My cat caught the influenza...
• Influenza of TV Character – In the last episode of
that TV Series, Ritsu-chan caught the flu
Research Questions• In total, half of Influenza related tweets are negative,
motivating an automatic filtering.
• RQ1: Could a NLP system filter out the negative influenza tweet?
• RQ2: Could this filtering contributes to the surveillance accuracy?
OUTLINE• Background
• Method
• Experiment
• Discussion
• Conclusion
Basic Idea: Binary Classification• We regard this task as a binary classification task , such as a spam mail
filtering
PositivePositiveNegativeNegative
Training Corpus
Training Corpus
(2) What kind of Feature?
(3) What kind of Machine Learning Method?
(1) What kind of Corpus?
inputinput
See proceeding for detailed Average Annotator Agreement Ratio = 0.85
Corpus (5k Sentences with Labels)
What kind of Feature?
I think the influenza is going aroundR1 R2 R3L1L2L3
• Surrounding Words (BOW, no stemming, no POS)
• Among various settings, Window size = 6 achieved the highest accuracy
Twitter contains many ungrammatical
expressions
Twitter contains many ungrammatical
expressions
What kind of Machine Learning Method?
Classifier F-Measure TimeAdaBoost 0.592 40.192Bagging 0.739 530.310Decision Tree 0.698 239.446Logistic Regression 0.729 696.704Nearest Neighbor 0.695 22.441Random Forest 0.729 38.683SVM (polynomial; d=2) 0.738 92.723
• Among various settings, SVM achieved the feasible accuracy
OUTLINE• Background
• Method
• Experiment
• Discussion
• Objective
Twitter Data (2008-2010)
• First month is used for training corpus• We divides the other data into 4 seasons
– Twitter API sometimes changes the spec, leading to dropout periods.
Season ISeason I Season IISeason II Season IIISeason III Season IV
Season IV
Method Comparison & Evaluation• (1) TWEET-SVM (The proposed method)• (2) TWEET-RAW
– Based on simple word frequency of “influenza”• (3) GOOGLE [Ginsberg 2009]
– Based on Google web-search query– The previous estimation data is available at the Google Flu Trend
website.
• (4) DRUG-SALE [Magruder 2003]• Evaluation is based on – Average Correlation with GOLD_STANDARD DATA that
is the real number of the influenza patients reported by Infection Disease Surveillance Center (IDSC)
Result: Correlation Ratio
TWEET-RAW TWEET-SVM GOOGLE DRUG
Season I 0.683 0.816 0.817 -0.208
Season II -0.009 -0.018 0.232 0.406
Season III 0.382 0.474 0.881 0.684
Season IV 0.390 0.957 0.976 0.130Bold indicates the correlation > statistical significance level.
In most seasons, the proposed method achieved the higher correlation than simple word freq-based method, demonstrating the advantage of the SVM based filtering
In most seasons, the proposed method achieved the higher correlation than simple word freq-based method, demonstrating the advantage of the SVM based filtering
+SVM
Result: Correlation Ratio
TWEET-RAW TWEET-SVM GOOGLE DRUG
Season I 0.683 0.816 0.817 -0.208
Season II -0.009 -0.018 0.232 0.406
Season III 0.382 0.474 0.881 0.684
Season IV 0.390 0.957 0.976 0.130Bold indicates the correlation > statistical significance level.
Except for Season II, the proposed method achieved almost the same accuracy to GOOGLE.
Except for Season II, the proposed method achieved almost the same accuracy to GOOGLE.
+SVM
Why Twitter suffers from Season II? Because it includes Pandemic!
Suggesting Twitter might be biased by News Media
TWEET-RAW TWEET-SVM GOOGLE DRUG
Normal Season 0.831 0.890 0.847 0.308Pandemic Season 0.001 0.060 0.918 0.844
WHO says Pandemic
In 1999 Jul (Season II).
WHO says Pandemic
In 1999 Jul (Season II).
Season ITWEET-SVM ≒ GOOGLERelative number
Season IIRelative number
TWEET-SVM << GOOGLE
OUTLINE• Background
• Method
• Experiment
• Discussion
• Conclusion
Extra ExperimentExtra Experiment
Frequent Question
• Could an Influenza Patient REALLY use a Twitter or Google Search?
• That seems to be un-natural situation!
I’d like to
sleep ...
I’d like to
sleep ...
Due to that, we modified the system assuming as follows:
People use Twitter or Google at the first sign of the influenza
People use Twitter or Google at the first sign of the influenza
( Markov model)≒
Implemented by usingInfectious Model [Kermack1927]
SSSusceptibleSusceptible
II RRInfectiousInfectious RecoverRecover
Catch the flu Recover
• S-to-I transition is observed by Twitter / Google• 38% of Influenza people recover a day
0.38
0.62BEFORE FLU AFTER FLUUNDER FLU
BUT: It ALSO improves Google based Approach
• This model improves correlation of BOTH Twitter & GOOGLE.
• This result suggests that there is a room of collaboration between medical study and web/NLP study
OUTLINE• Background
• Method
• Experiment
• Discussion
• Conclusion
Answer to Research Questions• This study proposed a new influenza surveillance
system using Twitter• RQ1: Could a system filter out the negative influenza?– Yes. But NOT Perfect
• RQ2: Could this accuracy contribute to the surveillance performance?– YES. It increases the correlation (except for pandemic
period).
• We could achieve the almost same accuracy to GOOGLE using freely available data.
Conclusion
• Still now, more than 100 (sometime over 1,000) people die from influenza in Japan
• We hope that this study might help people
Thank you
NLP could save a life!
Eiji ARAMAKI Ph.D.University of Tokyohttp://mednlp.jp
Eiji ARAMAKI Ph.D.University of Tokyohttp://mednlp.jp