Towards Healthier Online Communities

Jure Leskovec (@jure) Joint work with J. Cheng, C. Niculescu-Danescu-Mizil

Today’s computing applications: Social interactions User generated content Voting/reputation mechanisms

One user’s experience depends on the experience of the other user

Complex feedback loops Jure Leskovec, Stanford 2

In many situations, we observe human behavior, but the underlying principles driving it may not be obvious

Basic models provide a vocabulary for dissecting the fundamental ingredients

Jure Leskovec, Stanford 3

To build better applications… Healthier online communities Better social networking services More useful social media applications

…we need to understand what are the fundamental patterns of user behavior so we can design for them.

Jure Leskovec, Stanford 4

Plan for the talk: Investigate social feedback loops in online discussion forums Applications: Detecting antisocial/trolling behavior in online communities

Jure Leskovec, Stanford 5

Jure Leskovec, Stanford 6

[ICWSM ’14]

Jure Leskovec, Stanford 7

Jure Leskovec, Stanford 10

Jure Leskovec, Stanford 11

Jure Leskovec, Stanford 12

Jure Leskovec, Stanford 14

Jure Leskovec, Stanford 15

Positively Evaluated

Negatively Evaluated

?

?

Jure Leskovec, Stanford 16

Related work includes Quantifying social influence (e.g., voting) Muchnik, L., Aral, S., & Taylor, S. J. (2013). Social influence bias: A randomized experiment. Science.

Predicting evaluations (e.g., helpfulness) Ghose, A., & Ipeirotis, P. G. (2011). Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics. KDE.

Examining what people talk about Chen, Z., & Berger, J. (2013). When, why, and how controversy causes conversation. JCR.

Jure Leskovec, Stanford 17

How do we measure community feedback?

Number of up-votes

Up-votes minus Down-votes

Fraction of up-votes Jure Leskovec, Stanford 18

Doesn’t account for down-votes

15

Number of up-votes

0 15 15 vs

Jure Leskovec, Stanford 19

Doesn’t account for proportion of up-votes

5

Difference in up-votes and down-votes

0 50 45 vs

Jure Leskovec, Stanford 20

Doesn’t account for total number of votes

4

Proportion of up-votes

1 40 10 vs

Jure Leskovec, Stanford 21

To measure how users perceived votes, we asked crowd workers to rate, on a 7 point scale, if they felt more positive, or negative about receiving a certain number of up/down-votes.

Jure Leskovec, Stanford 22

7 = Positive

1 = Negative

0 20

20

# of up-votes

# of

dow

n-vo

tes

10

10

Jure Leskovec, Stanford 23

Number of up-votes

P-N Difference in up/down-votes

P/(P+N) Proportion of up-votes

0.410

0.879

0.920

R2

Jure Leskovec, Stanford 24

P/(P+N) = 9/(9+1) = 0.9 ≥ High Threshold

Jure Leskovec, Stanford 25

defined by the proportion of up-votes

Jure Leskovec, Stanford 26

Four large comment-based news communities with

1.2M articles, 1.8M registered users, 42M posts, 140M votes, 1 year

CN

N

IGN

Bre

itbar

t

allk

pop

Jure Leskovec, Stanford 27

Evaluations can affect

Post quality (How well you write)

Community bias (How people perceive you)

Voting behavior (How you vote on others)

Posting frequency (How regularly you post)

Jure Leskovec, Stanford 28

Do users improve?

Operant conditioning predicts that feedback would guide authors towards better behavior (i.e. up-votes are “reward” stimuli, and down-votes are “punishment” stimuli) Skinner, B. F. (1938). The behavior of organisms: An experimental analysis.

Jure Leskovec, Stanford 29

Or do they get worse?

Feedback can have negative effects. People given only positive feedback tend to become complacent. Also, bad impressions are quicker to form and more resistant to disconfirmation. Brinko, K. T. (1993). The practice of giving feedback to improve teaching: what is effective? Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Jure Leskovec, Stanford 30

0.45

0.55

0.65

0.75

0.85

Pro

port

ion

of u

p-vo

tes

{ {

Positive

Negative *

Before After

Jure Leskovec, Stanford 31

… …

…

What happens after you give a user a positive, or a negative evaluation? Wish:

Randomized controlled trial

Jure Leskovec, Stanford 32

Compare similar pairs of users who were evaluated differently on similar content

…

… ≈ ≈

…

…

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects.

3 posts before 3 posts after

Jure Leskovec, Stanford 33

Down-voting because of post content

I hate Obama…

How much of an evaluation can be explained by textual, or community effects?

Down-voting because the community dislikes the author

We don’t like you. Jure Leskovec, Stanford 34

Disentangling textual vs. community effects

Text quality determined by training a machine learning model using text features, validated using crowd workers.

Jure Leskovec, Stanford 35

p < 0.05 in all communities

To learn more about these types of effects, see Kanouse, D. E., & Hanson Jr, L. R. (1987). Negativity in evaluations.

Negativity bias

Jure Leskovec, Stanford 36

How does community perception of a user change after an evaluation?

Evaluations can affect Community bias (How people perceive you)

Jure Leskovec, Stanford 37

Community Bias

Actual Evaluation P/(P+N)

Text Quality

Text Quality Up-votes Down-votes

0.9

0.8

0.9-0.8

= +0.1

Community Bias?

Jure Leskovec, Stanford 38

User A

User B

…

…

Compare community bias before and after the evaluation

}

Jure Leskovec, Stanford 39

p < 0.05 in all communities

Halo Effect

Jure Leskovec, Stanford 40

Before

Mor

e P

ositi

ve

After

Similar Text Quality

Similar History

Positive Eval.

Negative Eval.

Worse Perception

Worse Text

Jure Leskovec, Stanford 41

Does feedback regulate post quantity?

Evaluations can affect Posting frequency (How regularly you post)

Jure Leskovec, Stanford 42

… … Before After

Time

Jure Leskovec, Stanford 43

Jure Leskovec, Stanford 44

Does feedback result in subsequent backlash?

Evaluations can affect Voting Behavior (How regularly you post)

Jure Leskovec, Stanford 45

0.6

0.625

0.65

0.675

0.7

Pro

porti

on o

f up

-vot

es g

iven

Before After

Positive

Negative *

Jure Leskovec, Stanford 46

Negatively-evaluated users write worse (and more!), are themselves evaluated worse by the community, and evaluate other community members worse.

Positively-evaluated users, on the other hand, don’t do any better.

Jure Leskovec, Stanford 47

Jure Leskovec, Stanford 48

0.16

0.18

0.2

0.22

0.24

Jan Feb Mar Apr May Jun Jul Aug

Pro

port

ion

of d

own-

vote

s

0.8m down-votes

1.7m down-votes

Jure Leskovec, Stanford 49

0.18

0.23

Jan Mar May Jul

0.10

0.20

Jan Mar May Jul

0

0.12

Jan Mar May Jul0.05

0.11

Nov Jan Mar May

Breitbart allkpop

CNN IGN

Jure Leskovec, Stanford 50

Jure Leskovec, Stanford 51

After

Before

Jure Leskovec, Stanford 52

53

Jure Leskovec, Stanford 54

[ICWSM ’15]

55

Girls are focused on their image way too much. This isn't new. And when they talk about other girls 'fat', 'ugly', and 'b*tch' are recurring words.

56

It also shows that Islam and Christianity teaching women to dress modest could be right afterall.

57

Religious nut alert

Clearly that is the only logical conclusion to this article. Now if you'll excuse me, I need to iron my tarp. I have work on Monday, and I want to appear 'modest'.

fail at life. go bomb yourself.

It also shows that Islam and Christianity teaching women to dress modest could be right afterall.

58

Both of these little skanks are ugly to the bone.

59

Both of these little skanks are ugly to the bone.

Troll

60

Baker, P. (2001); Donath, J. S. (1999); Herring, S., et al. (2011); Shachaf, P. and Hara, N. (2010)

62

63

Large-scale Longitudinal Predictive

Adler, B. T., et al. (2011); Wang, W. Y. and McKeown, K. R. (2010)

Jure Leskovec, Stanford 64

How do trolls differ from non-trolls?

How do trolls’ behavior change over time?

How do we predict antisocial behavior?

Over 18 months, ~1.7 million users generated ~40 million posts and ~100 million votes.

Post Deletion Rate 2.0% (>500k)

2.3% (>180k)

2.7% (>110k)

User Ban Rate 3.3% (>37k)

1.7% (>5k)

2.2% (>5k)

Many definitions: Engaging in negatively marked

online behavior Taking pleasure in upsetting others Not following the rules Disrupting a group while staying

undercover

Donath, J. S. (1999); Hardaker, C. (2010); Kirman, B., et al. (2012); Schwartz, M. (2008)

Troll User subsequently banned in the future. Non-troll User who was never banned, but is similarly active.

Jure Leskovec, Stanford 70

Facebook is for losers with no friends in real life.

Less positive

Facebook: The encyclopedia of beauty?

71

Suck a f****ng c@ck u d@uchebag

More profanity

Soccer star Muamba still 'critical' after on-pitch collapse

72

Penny, once again you show why you are one of the best of the league. Always a class-act. Always a good example.

Fantastic story. Kudos to Penny Hardaway. This is what you call giving back.

*sniff* This had me tearing up in the office. Great job Penny! You were always a class act on and off the court. Your Grandmother is applauding you again in heaven!

Ex-NBA star returns to inner city, brings hoop dreams

73

stop reading then. Just sayin.. CNN not making you read...you chose to.

How many white NBA players grew up

I have a question for CNN.......why as a reader do I not see any articles similar to this about white NBA basketball players?......every single touchy feely story is about a black ball player and then all you do is blast Tim Tebow............YOU GUYS MAKE ME SICK AS A READER !

Penny, once again you show why you are one of the best of the league. Always a class-act. Always a good example.

y g g

Less similar to previous posts

74

many stories of charitable efforts by white NBA players, but this isn't a story about a Black NBA player...it's a story about someone who came back to their roots to contribute.

b\c young black men need to see examples from their own race. They need to see that even minorities can succeed and give back instead. They connect with people of their own race. It allows for mentorship and guidance.

A story about a millionaire helping kids in his poor neighborhood personally, and being a positive role model makes you sick as a reader. Gotta love conservatives.

Gets more replies from other users

75

If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican. If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican. If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican.

If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican.

If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican.

If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican.

Posts more frequently

Posts more in the same thread

cut Social Security, you re a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican.

If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, Republican. If you claim you want less government but want to control the bedroom,you're a Republican; If you want to cut Education, you're a Republican; If you want to cut Social Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a Republican If you claim you want less government but want

76

0

0.2

0.4

0.6

0.8

1

0.1 0.3 0.5 0.7 0.9

Post

Del

etio

n R

ate

Time (Normalized)

Trolls

Non-Trolls

0

0.2

0.4

0.6

0.8

1

0.1 0.3 0.5 0.7 0.9

Post

Del

etio

n R

ate

Time (Normalized)

Trolls

Non-Trolls

Are trolls writing worse?

Is the community becoming less tolerant?

1

2

Mean Evaluated Post Appropriateness

All Posts First 10%

Last 10%

Trolls 2.7 3.0 2.3

Non-Trolls 3.3 3.5 3.2

Mean Evaluated Post Appropriateness

All Posts First 10%

Last 10%

Trolls 2.7 3.0 2.3

Non-Trolls 3.3 3.5 3.2

Mean Evaluated Post Appropriateness

All Posts First 10%

Last 10%

Trolls 2.7 3.0 2.3

Non-Trolls 3.3 3.5 3.2

-0.7

-0.3

0

0.2

0.4

0.6

0.8

1

0.1 0.3 0.5 0.7 0.9

Post

Del

etio

n R

ate

Time (Normalized)

Trolls

Non-Trolls

Are trolls writing worse?

Is the community becoming less tolerant?

2

First 10% of Life Last 10% of Life

≈

Similar Textual Quality

Trolls

Non-Trolls

Similar Textual Quality =

=

0

0.1

0.2

0.3

0.4

0.5

First 10% of Posts Last 10% of Posts

Text

ual Q

ualit

y

Trolls

Non-Trolls

Similar Textual Quality =

=

0

0.1

0.2

0.3

0.4

0.5

0.6

First 10% of Posts Last 10% of Posts

Post

Del

etio

n R

ate

~19k users CNN, balanced dataset, first 10 posts

Post # of words …

Activity posts/day …

Community % upvoted …

Moderator deletions …

90

0.70

0.74

0.5 0.6 0.7 0.8 0.9

Bag of Words

Deletion Rate

ROC AUC

91

0.70

0.74

0.62

0.5 0.6 0.7 0.8 0.9

Bag of Words

Deletion Rate

Post

ROC AUC

92

0.70

0.74

0.62

0.73

0.83

0.84

0.5 0.6 0.7 0.8 0.9

Bag of Words

Deletion Rate

Post

+ Activity

+ Community

+ Moderator

ROC AUC

ROC AUC

0.84 0.74 0.76

0.69 0.83 0.74

0.74 0.75 0.78

Trained on

Test

ed o

n

Towards healthier online communities.

Large-scale longitudinal analysis of feedback loops and antisocial behavior in discussion forums.

The effects of negative feedback are more pronounced than those of

positive feedback. Jure Leskovec, Stanford 94

Negatively-evaluated users write worse (and more!), are themselves evaluated worse by the community, and evaluate other community members worse.

Positively-evaluated users, on the other hand, don’t do any better.

Jure Leskovec, Stanford 95

Trolls differ from non-trolls in their language and behavior in a community.

Trolls mature over time, and the community plays a role in incubating antisocial behavior.

We can predict whether a user will subsequently get banned, just by looking at a user’s initial posts.

Future work could look at communities that support only

positive feedback (e.g., likes),

the effects of user reputation,

deeper linguistic analysis, and

experiments to establish causality.

How Community Feedback Shapes User Behavior J. Cheng, C. Danescu-Niculescu-Mizil, J. Leskovec. AAAI International Conference on Weblogs and Social Media (ICWSM), 2014.

Antisocial Behavior in Online Discussion Communities by J. Cheng, C. Danescu-Niculescu-Mizil, J. Leskovec. AAAI International Conference on Weblogs and Social Media (ICWSM), 2015.

Jure Leskovec, Stanford 98