Methods and guidelines for the

design and analysis of online citizen

science*

Elena Simperl, University of Southampton, UK

*with slides by Ramine Tinati, Markus Luczak-Rösch, and others

sociam.org

@project_sociam

Overview

• Background to citizen science

• Citizen science & human computation

• Citizen science & online communities

• Citizen science & science

• Methods and design guidelines

What is citizen science

Example: Zooniverse

Example: Pybossa

Example: Volunteer Science

Example: EyeWire

Studying citizen science: human computation

• Task design

• Task assignment

• Answer validation and aggregation

• Contributors’ performance

• Motivation and incentives

Studying citizen science: online community

• Roles and activities

• Patterns of participation

• Motivation and incentives

• Evolution in time

Studying citizen science: eScience

• New workflows

• New best practices

• New publishing models

A footer

When is a citizen science

project successful?

Levels of engagement

0

2

4

6

8

10

12

14

16

18

20

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Acti

ve u

sers

in

%

Month since registration

~1% of participants contribute 72% of Talk & 29% of Task

Beyond data collection and analysis

• Discussion and community engagement are

integral part of the experience

Work vs talk

40.5%

Classifications

Talk

co

ntr

ibu

tio

ns

Classifications

Chat and instant messaging

Microposts

PH SG SW NN GZ CC PF SF AP WS

91% 2

0

6

4

10

8

Discussion profiles

Deeply

engaged

volunteers, few

threads but

multiple posts

within them

9 0.1

%

Content

producers,

posting across

many boards

and threads

7 0.1

%

Thread

followers and

PM (one-to-

one) talkers

8 0.4

%

First to

respond and

question

answerers 4 1%

Highly active

thread starters

and answerers

across a wide

range of topics

1 2.8

%

Infrequent

volunteers,

single thread

posts, no

personal

messages

5 5.5

%

Watcher and

starter of

many threads,

but not first to

reply

3 6.5

%

Highly active

thread

starters and

first to reply

back

2 14.6

%

Long active

volunteers (the

core group),

posting

sporadically

6 69.0

%

Talking does not mean less work

How long do games take to complete depending on when the chat messages are made?

Games took longer to complete when chat messages were being made during the

gaming session, compare to games that had messages before or after.

Ecosystem effects

Project A

Project B

Project C

Participant X

Part. Y

Designing platforms

Task specificity

Community development

Task design PR and

engagement

Bootstrapping the

community

Serendipitous

scientific discovery

Engaging with people,

supporting profession

team

Supporting individuals,

finding new scientific

discoveries

Obtaining new

citizen scientists

Retaining people

Supporting people,

improving task

completion

Obtaining new citizen

scientists

Reinvigorating old

users

What’s next?

• Human computation

• Task assignment: what tasks are interesting/relevant for whom?

• Peer review, collaborative approaches

• The role of gamification: is science a game?

• Online community

• Mapping design patterns to online community design frameworks: do CS form a community?

• Making discussions more effective

• Science

• Citizen science platforms that everyone can use

Publications

R. Tinati, M. Van Kleek, E. Simperl, M. Luczak-Rösch, R. Simpson and N. Shadbolt. Designing for citizen data analysis: a cross-sectional case study of a multi-domain citizen science platform. Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2014 (CHI2015), pages 1-10, 2015. M. Luczak-Rösch, R. Tinati, E. Simperl, M. Van Kleek, N. Shadbolt and R. Simpson. Why Won't Aliens Talk to Us? Content and Community Dynamics in Online Citizen Science. Proceedings of the International Conference on Weblogs and Social Media (ICWSM2014), 2014.

A footer