How effectively are crowdsourcing websites supporting volunteer participation and quality contribution?

Donelle McKinleyPhD candidate, School of Information ManagementVictoria University of WellingtonNDF Barcamp, Hamilton, 21 June 2013

www.digitalglam.org @donellemckinley

Crowdsourcing: a short version

Crowdsourcing outsources tasks traditionally performed by specific individuals to a group of people or community through an open call (Howe, 2009).

Crowdsourcing: a long version

“Crowdsourcing is a type of participative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, the voluntary undertaking of a task. The undertaking of the task, of variable complexity and modularity, and in which the crowd should participate bringing their work, money, knowledge and/or experience, always entails mutual benefit. The user will receive the satisfaction of a given type of need, be it economic, social recognition, self-esteem, or the development of individual skills, while the crowdsourcer will obtain and utilize to their advantage that what the user has brought to the venture, whose form will depend on the type of activity undertaken.”

(Estellés-Arolas & González-Ladrón-de-Guevara, 2012)

Potential benefits of crowdsourcing for the institution

O Continue tradition of volunteerismO Market and stimulate interest in collectionsO Signal the institution's openess and

approachabilityO Achieve goals otherwise too costly and labour-

intensiveO Better reflect the diversity of visitorsO Tap into expertise outside the institutionO Engage visitors in new waysO Raise the profile of researchO Demonstrate relevanceO Enable new research questions to be explored

Common motivations for volunteer participation

O The size of the challengeO The necessity for volunteer contributionO Collaboration with prestigious institutionsO Contribution to researchO EducationO Mental stimulationO Being part of a communityO Personal research interestsO Enhancing a resource from which they will

benefit

The potential of the crowd

Digitalkoot (National Library of Finland)O First 51 days of the project O 31, 816 visitors to the siteO 15% participated

Transcribe Bentham (University College London)O First 6 months of the projectO 1,207 visitors registered to participate O 21% participated

Three common scenarios

The website either:

O Follows the structure of the underlying

technology or the organisation

O Adheres to familiar conventions

O Is the product of personal preference

Garrett, J. (2011). The Elements of User Experience: User-centered design for the web. USA: Aiga/New Riders.

How effectively does your website…?

O Define its objectives

O Reflect visitor motivations

O Align visitor motivations with

relevant incentives

O Minimize sources of frustration and

concern

How are they impacting on the effectiveness of your website?

O ContentO LanguageO ReadabilityO Website navigationO Arrangement of page elementsO ConsistencyO Visual appearanceO Page load speedO Number and complexity of processes to

complete the desired action

O Prototype: an early sample or model built to test a concept or process

O Pilot: a small-scale preliminary experiment conducted to evaluate feasibility, time, cost, adverse events, and improve design prior to the launch of a full-scale project

O Soft-launch: the release of a website to a limited audience, in order to (beta) test and tweak a design before being launched to a wider audience

O Beta testing: user testing by a limited audience to ensure the website/software has few faults or bugs, and gather user feedback

O Optimization: increasing the percentage of visitors that fulfill the objective of a webpage or website

User evaluation: A different kind of user engagement

OFocus groups

OSurveys

OUsability testing

OOngoing feedback channels

Analytics: How is website design impacting on…?

O The number of online visitors

O Time spent on site

O The number of online visitors who register to

participate

O The number of online visitors who actually

participate

O The number of abandoned and completed

tasks

O The number of return visitors

OCR text correction: Trove, National Library of Australia

Evaluation of the Trove crowdsourcing user interface involved:

O Asking potential volunteers to comment on the prototype

O Sitting potential volunteers in front of the computer screen and asking them how they would complete the task

O A soft launch and beta testingO Gathering feedback on the beta version from over

600 users over the course of five months, via a survey, an online contact form, direct observation of user activity, analytics, online comments, and direct contact with users via email and phone

Evaluation of the Transcribe Bentham task interface involved:

O Beta testingO A user surveyO Website analyticsO Analysis of user interaction statisticsO Comparisons with studies on

crowdsourcing volunteer behaviour

UK Reading Experience Database (UK-RED)

Evaluation of the UK-RED task interface involved:

O Heuristic evaluation O A survey of current and potential

contributorsO Comparison with other

crowdsourcing task interfacesO Comparison with recommended

practice as outlined in crowdsourcing and HCI literature

Usability and functionality requirements for a NZ-RED task interface

O Minimize user effortO Support integration of the task with research

processesO Enable new visitors and contributors to

understand what the task involves quickly and easily

O Support accurate and controlled data entry O Be easy to use for people reasonably confident

with the WebO Support flexible, structured data entryO Support bilingual data entry

Challenges impacting on evaluation

OFunding

OExpertise

OOrganisational culture

What could a more effective website mean for the project?

O More online visitors participateO Tasks are completed more efficientlyO Tasks are completed with greater

accuracyO The task is more enjoyableO Volunteers participate more oftenO The project is more cost-effectiveO More volunteers are willing to participate

in future projects

Thanks!

For references and other great reads visit http://www.digitalglam.org/crowdsourcing/books/

Presentation and slides will be available athttp://www.digitalglam.org/crowdsourcing/talks/

For crowdsourcing research updates follow www.digitalglam.org @donellemckinley