Vandalism Detection in Wikidata - IMT · Vandalism Detection in Wikidata Stefan Heindorf1, Martin Potthast2, Benno Stein 2, Gregor Engels1 CIKM 2016 October 25, 2016 12

Vandalism Detection in WikidataStefan Heindorf1, Martin Potthast2, Benno Stein2, Gregor Engels1

CIKM 2016October 25, 2016

1 2

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

Motivation


Motivation


Motivation


Motivation


Motivation


Motivation


Motivation


3Stefan HeindorfVandalism Detection in Wikidata


Item head


Item head

Item body


Item head

Item body

Revisions


(Feb 22, 2013)

(May 13, 2013)

(May 30, 2013)

Item head

Item body

Revisions


(Feb 22, 2013)

(May 13, 2013)

(May 30, 2013)

Item head

Item body

Revisions


(Feb 22, 2013)

(May 13, 2013)

(May 30, 2013)

Item head

Item body

Revisions


Item head

Item body

Revisions

Why is it a problem?

4

Patrolling Reverting Warning Blocking Protecting

• Over 2 Mio manual edits per month• A lot of tedious work• Vandalism is not detected in time

Stefan HeindorfVandalism Detection in Wikidata

Research Question

How to detect damaging changes tocrowdsourced knowledge bases?


Our Approach


Our Approach

1. Label Dataset Vandalism Corpus [SIGIR’15]


Our Approach

1. Label Dataset Vandalism Corpus [SIGIR’15]2. Study Vandalism Characteristics 47 Features


Our Approach

1. Label Dataset Vandalism Corpus [SIGIR’15]2. Study Vandalism Characteristics 47 Features3. Experiment with ML Multiple-Instance Learning


Our Approach

1. Label Dataset Vandalism Corpus [SIGIR’15]2. Study Vandalism Characteristics 47 Features3. Experiment with ML Multiple-Instance Learning4. Compare with state of the art 2 Baselines


Corpus [SIGIR ’15]

Revisions over time

7


Revisions over time

7Month


Revisions over time

7Month


Revisions over time

7Month

103,000 vandalism revisions


Revisions over time

7Month

103,000 vandalism revisions24 million manual revisions


Revisions over time

7Month


0.4% vandalism


Revisions over time

7

Item head(1.3% vandalism)

Month


0.4% vandalism


Revisions over time

7


Item body(0.2% vandalism)

Month


0.4% vandalism


Revisions over time

7


Item body(0.2% vandalism)Training

Month


0.4% vandalism


Revisions over time

7



Validation

Month


0.4% vandalism


Revisions over time

7



TestValidation

Month


0.4% vandalism

Content Features11 Character features (e.g., lowerCaseRatio, digitRatio) 9 Word features (e.g., badWordRatio)4 Sentence features (e.g., commentSitelinkSimilarity)3 Statement features (e.g., propertyFrequency)

Context Features10 User features (e.g., userCountry)2 Item features (e.g., logItemFrequency)8 Revision features (e.g., revisionTag, revisionLanguage)

Features (47 in total)

Stefan Heindorf 8Vandalism Detection in Wikidata



revisionTag



revisionTag Vand. Total Prob.Rev. with tags 52 T 8,619 T 0.60%

By abuse filter 49 T 122 T 39.90%By editing tools 3 T 8,496 T 0.03%

Rev. w/o tags 52 T 15,386 T 0.34%

revisionTag





Rev. w/o tags 52 T 15,386 T 0.34%

revisionTag





Rev. w/o tags 52 T 15,386 T 0.34%

revisionTag





Rev. w/o tags 52 T 15,386 T 0.34%

revisionTag

Multiple-Instance Learning



• Observation: Vandalism seldom occurs in isolation





22:35, 11 September 2013 184.19.64.111 (talk) . . (Changed English label: Barack Obama Aloha)

22:35, 11 September 2013 184.19.64.111 (talk) . . (Added English alias: Lulu:):):):):):):))

12:05, 11 September 2013 MatmaBot (talk | contribs) . . (Changed Polish description: imported













Session 1

Session 2



• Idea: Apply Multiple-Instance Learning





Session 1

Session 2








Session 1

Session 2








Session 1

Session 2

WDVD vs. Baselines

• WDVD (our approach)Wikidata Vandalism Detector

10Vandalism Detection in Wikidata Stefan Heindorf

WDVD vs. Baselines


• FILTER (baseline)Wikidata Abuse Filter


WDVD vs. Baselines



• ORES (baseline)Objective Revision Evaluation Service


WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision

RecallVandalism Detection in Wikidata Stefan Heindorf

Test Dataset (0.2% vandalism)

WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision


FILTER


WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision


ORES

FILTER


WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision


ORES

FILTER


WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision


ORES

FILTER


WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision


ORES

FILTER

PR-AUC: 0.491ROC-AUC: 0.991


WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision


Detect and revert 30% vandalismfully automatically

ORES

FILTER


WDVD vs. Baselines




10

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

ision


Detect and revert 30% vandalismfully automatically

ORES

FILTER

• Reduce workload by factor 10(precision 2% instead of 0.2%)

Still find 98.8% of all vandalism


Conclusion and Outlook



Conclusion• Vandalism: Concentration on item heads (currently)• Features: Content & Context• Model: Multiple-Instance• PR-AUC: 0.491• ROC-AUC: 0.991





Code + Data: http://www.heindorf.me/wdvd.html



Outlook• Goal: Better detection (on item bodies)• Idea: Double-check with other sources








Join the competition:Vandalism Detection @WSDM Cup 2017http://www.wsdm-cup-2017.org/




Acknowledgement• German Research Foundation (DFG)• SIGIR Student Travel Grant







Acknowledgement• German Research Foundation (DFG)• SIGIR Student Travel Grant




Thank you!

Documents

Vandalism Detection in Wikidata - IMT · Vandalism Detection in Wikidata Stefan Heindorf1, Martin Potthast2, Benno Stein 2, Gregor Engels1 CIKM 2016 October 25, 2016 12