Upload
sheamus-mcgovern
View
78
Download
4
Embed Size (px)
Citation preview
Forensic Linguistics with Apache Spark
Kostas Perifanos @k_perifanos
Idiolect, sociolect, intertextuality
What?
- Idiolect: individual’s distinctive and unique use of language
- Sociolect : variety of language associated with a social group (socioeconomic, ethnic, age)
- Intertextuality: the shaping of a text’s meaning by another text
Forensic Linguistics
"Forensic linguistics, legal linguistics, or language and the law, is the application of linguistic knowledge, methods and insights to the forensic context of law, language, crime investigation, trial, and judicial procedure. It is a branch of applied linguistics.” [Wikipedia]
- Authorship Attribution
- Authorship Identification
- Gender/Age classification etc
Dataset
- 8m tweets between 18/06/2015 - 06/08/2015
- 92m words (white space tokenized)
- 190K users
- Key events during this period
- Referendum Announcement
- Capital Controls
- Referendum voting
Toolset
- Apache Spark 1.6.1
- RDD
- DataFrames / Spark SQL
- Word2vec, KMeans
- Apache Zeppelin
- Gephi
Basic Data Exploration - Counting
Check for trends:
- Lowercase vs Uppercase ratios
- Relative frequencies of important (propaganda) words
- Average text length (per day)
- Average word length (per day)
Counting - lowercase / uppercase ratio
Counting - Propaganda
- Build a word2vec model, treat @mentions as vocabulary words
- Find top-N “synonyms” using seed accounts, keep all starting with “@”
- @handle1: @handle2, @handle3, ...
- @handle32: @handle5, @handle3, ...
- Visualize the graph
Similarities & user interactions
Similarities & interactions graph [Gephi]
Similarities & interactions graph [Gephi]
Gephi : Modularity analysis, 9 communities detected
Communities:
- “Yes”, black
- “No”, magenta
- media, red
- celebrities, dark green
- “Romantic twitter”, orange
- ....
- Choose top N most frequent words [1]
- Build frequency vectors for all users
- Compare user signatures [eg Cosine Similarity]
- Identified double-account user among 180K candidates (so much for anonymity)
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3694980/
2. Idiolect : Style signatures
2. Idiolect : Style signatures
- Apply clustering on signature vectors
- KMeans on signatures
- KMeans on word2vec vectors:
- Transform words to vectors, sum and average
- Also works very well for metaphor detection
Sociolect: Clustering
- User generates texts by sampling a number of topics
- “Similar” users will tend to have similar topic distributions
- Given a subset of similar users, identify the most influential, eg the user who enforces writing style. [But that’s another presentation :)]
Challenges
Noise
“Random events”
Opinion shifting: People change their opinions and their writing styles accordingly. Social media tends to amplify this behaviour [one more presentation :) ]
Intertextuality: LDA + signatures
- User - Topic Classification
- Gender classification
- Age
- Personality, stress, anxiety etc
- Try Deep Learning approaches
Next steps
Thank you!
Questions?
@k_perifanos - http://github.com/kperi