Amazon Mechanical Turk (Mturk)

• What is MTurk?– Crowdsourcing Internet marketplace that utilizes

human intelligence to perform tasks that computers are unable to do

– Mturk samples are…• Fast • Cheap• Reliable• Diverse

First Things First…• Go to www.mturk.com

Click here

First Things First…• Sign in with existing Amazon.com account or

create one

Getting StartedYou are called a requester

Design

• The space that hosts your Qualtrics survey link is called a HIT

• Your HIT is published to the Mturk site for workers to see

• The first step is to design the layout of your HIT

Properties

The title and description are seen by workers

Keywords categorize your HIT and make it searchable

PropertiesSet time restrictions

Chose Mturk Masters option

Set qualifications

Choose qualification requirement

Properties

Choose payment amount per HIT

Choose number of participants

The next step will be to design the layout of the HIT

Layout

• Uses html code • Give instructions and brief description of study

to workers • Provide survey link

Layout

Template name should identify your study. This is not displayed to workers.

Keep any of this you want and delete the rest.

Layout

Code word and study instructions

Qualtrics link

Confirmation code

User comments

Layout• Use a code word to prevent workers from repeating similar surveys• Lets say you create a survey about prejudice and use code word

ALPHA for the HIT• Inform workers before they begin that the HIT uses code word

ALPHA• After data collection you would like to re-run the study after making

some changes on Qualtrics• You don’t want workers who completed the first round to do the

second round also• Again, inform workers that the HIT uses code word ALPHA and that if

they had taken an ALPHA HIT before, they should not continue

Layout

• Other ways of controlling for repeats– In Qualtrics, create an item at the beginning of the survey

that asks the participant if they have completed a HIT from this requester before• If no, okay• If yes, ask if the HIT they’ve done uses undesired code word (in

this case, its ALPHA)– If yes, send participant to end of survey message– If no, okay

– Ask at end of survey if participant had completed a similar survey in the past• Yes, no, I don’t remember

Layout

Confirmation code

Layout

• Confirmation Code– Create a space for workers to enter the

confirmation code you provide at the end of Qualtrics survey

– Code can be made up– Used to check that workers completed entire

survey

Layout

Layout

SpaceLink

Space

Layout

SpaceInstructions

Text box

Layout

SpaceInstructions

Text box

Layout – html code<p><a href="http://kuclas.qualtrics.com/SE/?SID=SV_5pfmnKdhILnpOfi" target="_blank">Click here to take the survey.</a></p> !survey link<p>&nbsp;</p> !space<p>At the end of the survey, you will be provided with a confirmation code. Please enter the code below to verify that you have completed the task:</p><p><textarea rows=“1" cols="80" name=“Code"></textarea></p> !code box<p>&nbsp;</p> !space<p>Please provide any comments you may have below, we appreciate your input!</p><p><textarea rows="3" cols="80" name="comment"></textarea></p> !comment box

Preview and Finish

Publish

Publish

Publish HIT

Name the batch

Negative balance

Publish

• Note: – If your Amazon account is funded, the HIT will be

published right away– If not, you will be notified of this and be able to

click a link which will allow you to fund your account using a variety of methods

Manage

• Watch completion rate• Accept or reject work• Cancel batch

Manage Click here to accept work (pay participants)

ManageConfirmation code

Marking worker and clicking approve will initiate payment

Clicking worker ID allows you to block worker and pay bonuses

Approved 1 HIT out of 1 for this worker

MTurk – Things to consider

• Blocking workers is BAD• Average pay anywhere from $.10 to $1.00• Command line tools• Professionalism– Turkernation.com

• SPSP Seminar on MTurk

Thank you!

• Matt Baldwin: mwbaldwin@ku.edu• Lucas Keefer: lkeefer1@ku.edu