Crowdsourcing Scientific Work: A Comparative Study of Technologies, Processes, and Outcomes in Citizen Science

Crowdsourcing Scientific Work

A Comparative Study of Technologies, Processes, and Outcomes

in Citizen Science

Andrea Wiggins11 April, 2012

Kevin Crowston (Advisor)Rick BonneyJian QinSteve Sawyer

Geof Bowker (External Reader)Murali Venkatesh (Internal Reader)John Burdick (Chair)

Tuesday, May 15, 12

Citizen Science

• Projects involving the public with scientists in collaborative research.

-‐ Crowdsourcing scientific work of data collection and processing

-‐ Increasingly ICT-‐mediated

• As citizen science gains in popularity, scientists need a better understanding of how design and management influence scientific outcomes, particularly for ICT-‐enabled participation.

• Research goals

-‐ Describe the phenomenon of citizen science.

-‐ Develop an empirically-‐grounded framework that describes the conditions, processes, and products of citizen science projects.

Tuesday, May 15, 12

Related Research

• Public participation in science

-‐ Purposes and forms of engagement

-‐ Informal science education, policy, STS

•Irwin; Bonney et al; Cooper et al; Wilderman

• Scientific collaboration

-‐ Broader context of practice

•Sonnenwald; Finholt; Lawrence et al

• Online communities

-‐ Participation in virtual environments

•Crowston; Haythornthwaite; Preece & Shneiderman

= citizen science*

*volunteer

monitoring

cybe

r-in

frast

ruct

ure

crowd-sourcing

publicparticipationin science

scientificcollaboration

onlinecommunities

Tuesday, May 15, 12

Research Questions

How do virtuality and technology alter organizing in citizen science?

How do virtuality and technology shape participation in citizen science?

How do organizing and participation influence scientific outcomes in citizen science?

Tuesday, May 15, 12

Cases

Tuesday, May 15, 12

Mountain Watch

•Monitoring alpine climate change

-‐ Participation involves:

•Finding monitoring plots

•Identifying target plants and their phenophases

•Recording observations on paper

•Dropping off data sheet at facilities or entering online

-‐ Started in 2004 by the Appalachian Mountain Club

•Primarily in White Mountains of New Hampshire

•Combines citizen science with other research efforts

•Intensive study of factors influencing data quality

Tuesday, May 15, 12

Great Sunflower Project

• Collecting data on pollinator service (bees!)


•Planting sunflowers

•Creating garden description on Drupal website

•Recording 15-‐minute observation samples on data sheet

•Online data entry

-‐ Started in 2008 by a single academic researcher

•Collects data across North America

•Very successful in attracting volunteer interest

Tuesday, May 15, 12

eBird

• Collecting bird abundance and distribution data


•Choosing observation methods

•Recording bird observations

•Entering observations and metadata online

-‐ Launched in 2002 by Cornell Lab of Ornithology (with National Audubon Society)

•World’s largest biodiversity data set

•Receives between 2.5M -‐ 3M observations/month

•Data used in both research and decision-‐making for policy and land management

Tuesday, May 15, 12

Comparative Case Selection

CriterionCriterion Mountain Watch Great Sunflower eBird

PurposeMission Conservation,

education, recreationResearch, education

Research, education, conservation

PurposeScientific interests

Climate change effects on alpine habitats

Plant-‐bee relationshipsBird abundance &

distribution

Intended Community

Hikers Gardeners Birders

EnvironmentInstitutions Single nonprofit Academic Nonprofit partnership

EnvironmentResources 1.5 FTE, $15K 0.5 FTE, $13K 4.5 FTE, $300K

Technologies

Paper Structured data sheet Structured data sheet Variable & optional

Technologies Digital Organization website section

Open source CMS website

Purpose-‐built software systemTechnologies

Data access Limited Very limited Extensive

Tuesday, May 15, 12

Methods

Tuesday, May 15, 12

Data Collection

• Semi-‐structured interviews with project organizers

-‐ Sampled for maximum diversity of roles and perspectives, with individuals from 7 organizations

-‐ Some longitudinal interviews, additional informal interviews

• Participant observation

-‐ 300+ hours of birding, 3 years of sunflowers, 6 days in the White Mountains

-‐ Listservs, forums, beta testing interfaces & mobile application

-‐ Extensive involvement in citizen science organizer community

• Secondary data, documents, & artifacts

Tuesday, May 15, 12

• Concurrent with data collection and theory development

-‐ Iterative deductive and inductive coding

-‐ Rich process models

-‐ Concept diagrams

• Research Quality

-‐ Interviewees reviewed transcripts

-‐ Key informants reviewed case chapters

-‐ Expert and peer review of findings

-‐ Audit trail, ongoing memos

-‐ Data triangulation

Analysis

Participation

Satisfaction

Individual Development

Sustainability

Scientific Knowledge

Organizing

Design

Institutions

Scientific Interests

Technologies

Contributions

Community

Broader Impacts

Resources

Commitment

Mission

Skills

Biography

Networks

Personal Interests

Science

Tuesday, May 15, 12

Findings

Tuesday, May 15, 12

Theoretical Framework

• Iteratively developed

-‐ Initial version based on literature, used to guide study

-‐ 16 versions over 3 years

• Inputs-‐Moderators-‐Outputs-‐Inputs structure

• Example of a relevant flow:

-‐ Design & Organizing -‐> Participation -‐> Contributions -‐> Scientific Knowledge

Organizational Inputs Individual

InputsIndividual Outputs

Individual Processes

Individual Emergent

States

Organizational Outputs

Organizational Processes

Organizational Emergent

States

Organization Design

Technology Design

Community

Sustainability

Knowledge

Communication

Data Management

Volunteer Management

Scientific Research

Demographics

Skills

Motivation

Commitment

Roles

Joining

Contributing

Learning

Satisfaction

Contributions

Innovation

Task Design

Organizational Inputs Individual

InputsIndividual Outputs

Individual Processes

Individual Emergent

States

Organizational Outputs

Organizational Processes

Organizational Emergent

States

Organization Design

Technology Design

Community

Sustainability

Knowledge

Communication

Data Management

Volunteer Management

Scientific Research

Demographics

Skills

Motivation

Commitment

Roles

Joining

Contributing

Learning

Satisfaction

Contributions

Innovation

Task Design

Tuesday, May 15, 12








Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions

CommunityBroader Impacts

Resources

Mission

Individual InputsSkills

BiographyNetworksPersonal Interests

Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Tuesday, May 15, 12








Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Tuesday, May 15, 12








Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Tuesday, May 15, 12








Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Tuesday, May 15, 12








Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Environment

Products

Outputs

Outcomes

Inputs



Project Inputs

Institutions


Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design

Science

States

Satisfaction

Sustainability

Commitment

Tuesday, May 15, 12








Environment

Products

Outputs

Outcomes

Inputs


?

Project Inputs

Institutions

?

Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design?

States

Satisfaction

Sustainability

Commitment

Environment

Products

Outputs

Outcomes

Inputs


?

Project Inputs

Institutions

?

Technologies

Contributions


Resources

Mission



Processes

Participation

Organizing

Design?

States

Satisfaction

Sustainability

Commitment

Tuesday, May 15, 12

Emergent Themes

1. Project design approaches that favor science versus hobbies for participation design

2. Design and organizing implications of engaging communities of practice

3. Relationships between physical environment, technologies, participant experiences, and data quality

4. Information technology tradeoffs: helpful for scale and communication, challenging for usability and resources

5. Resources and sustainability relate to institutions and scale of participation

Tuesday, May 15, 12

How do virtuality and technologies alter organizing in citizen science?

• Virtuality is inherent and a key benefit, but leads to questions about quality

-‐ “People would gravitate towards the really charismatic species, which in the White Mountains is diapensia. So people would go out with these diapensia-‐tinted glasses, and they’d see it everywhere and pass over the least well-‐known species.”

• Enables large-‐scale research that is more like crowdsourcing than other forms of scientific collaboration

-‐ “If technology makes new things available, you change your focus to exploit it.”

• Reduces coordination costs and improves quality, but ICT often unsuited for use in the field

-‐ “Someone entered in data that said that they saw a bee after 130 minutes, and I think what they were putting in is that it was at 1:30 in the afternoon.”

Tuesday, May 15, 12

How do virtuality and technologies shape participation in citizen science?

• Opens participation opportunities to larger, more diverse population

-‐ “The skill base varies from Master gardeners and beekeepers to amateur first-‐time gardeners. ... Our audience skews a little older. There are far fewer schoolchildren who participate than I thought there might be.”

• Importance of place: geographic biases and autonomy, functional constraints of and emotional relationships to place

-‐ “Folks do have a real connection to these mountains. So to feel like they can do something to help out, and to protect, and get a handle on what is actually happening up here in the mountains, it’s valuable.”

• Leads to usability issues for some, but can also be rewarding and more scalable

-‐ “Some people have difficulty printing out the data form, and writing all this stuff in while they’re observing, and taking it back, and then entering it in.”

-‐ “Let’s give them tools to do what they want, and they’ll give us all of their data.”Tuesday, May 15, 12

How do organizing and participation influence scientific outcomes in citizen science?

•Diverse types of scientific outcomes suggest more holistic criteria for evaluating project success

-‐ “[eBird] is just getting to the point where we are going to see more and more information come out that will help drive policy and decision-‐making.”

• Keep participants happy: greater quality and/or quantity of contributions improve outcomes

-‐ “The more people enjoy the project and get some reward then the better off you’ll be for sustaining it. We’ve seen significant growth that hasn’t slowed down since we turned the switch on and changed the way we think about it.”

Tuesday, May 15, 12

Limitations & Future Work

• Limitations

-‐ Depth rather than breadth

-‐ Focused primarily on organizers

• Future work

-‐ Integrate findings and framework with participant-‐oriented studies

-‐ Compare to entirely online citizen science projects

-‐ Work with organizer community to translate findings into recommendations for practice

Tuesday, May 15, 12

Contributions

• Theoretical framework

-‐ Complements and extends prior models

-‐ Foundation for future research with room for expansion & refinement

• Case studies

-‐ In-‐depth description and comparison

• New prospective best practices

-‐ Sustainability planning in context of organizations and resources

-‐ Aligning scientific and personal interests as much as possible

-‐ Making explicit links between individuals, communities, and organizing

-‐ Engaging non-‐scientist community members as organizersTuesday, May 15, 12

Thanks

• Committee

• Case study projects

-‐ Mountain Watch

-‐ Great Sunflower Project

-‐ eBird

•Writing group

-‐ Mohammad Jarrahi & Jaime Snyder

• Everett Wiggins

•U.S. National Science Foundation Grants 09-‐43049 & 11-‐11107

Tuesday, May 15, 12