Easy Mechanical Turk Integration with Houdini

HoudiniOn Demand Labor API

By: Chris Conley and Jameson Detweiler

Philly AWS December Meetup

12/13/2010

Date: December 13, 2010

Tuesday, December 14, 2010

12/13/2010

Easy Mechanical Turk Integration with Houdini

• Marketplace for work

• ~ 500k workers

• Accessible via API

Philly AWS December Meetup

Mechanical Turk

12/13/2010

Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup

Task Examples

• Moderation - images, comments, reviews, video

• Categorization - products, best image, tagging

• Ads - moderation, keywords

• Audio Transcription / Video Captioning

• Cleanup - proofreading, contact reconciliation, review editing

• PDF/Image/Handwriting to Text

• Research - contact info, document search

12/13/2010

Turk Overview• Workers and Requesters

• HITs

• Questions

• Assignments

• Qualifications

• Notifications

• Blocking Workers

12/13/2010

Worker View

12/13/2010

Creating HITs

• Easy

• Great for testing the waters

• No integration

• More Flexible

• But Complicated

• Programmatic Integration

WYSIWYG API

12/13/2010

WYSIWYG

12/13/2010

WYSIWYG

12/13/2010

http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html

12/13/2010

HIT Questions

• XML

• External Question

12/13/2010

Turk API Libraries

• ruby-aws http://rubyforge.org/projects/ruby-aws/

• rTurk https://github.com/mdp/rturk

• Turkee https://github.com/aantix/turkee

12/13/2010

Questions about Turk?

12/13/2010

Houdini

• Simple

• Qualified results

• Automatic

• Trusted Workers

12/13/2010

Houdini APIs

• Simple

• Exact

• Multi-answer

• Image Moderation

• Ad Moderation

• Data Reconciliation

• Data Extraction

• ...

Generic Optimized

12/13/2010

Simple API

• One answer per task

• Cheap

• Image Moderation, Low profile content

12/13/2010

Simple API Example

12/13/2010

Exact API

• Multiple workers

• Ensures all answers match

• Returns the “final answer”

• If not, escalated to trusted worker group

• More expensive than Simple

• Image Moderation, Data Harvesting, etc

12/13/2010

Exact API Example

12/13/2010

Multi-answer API

• Ask multiple turkers same question

• All answers are returned

• Surveys, feedback, answer qualification on your end

12/13/2010

Image Moderation API

• No need to build form_html

• Just send the image url

• Optionally send custom instructions and possible responses

12/13/2010

Image Moderation Example

12/13/2010

Image Moderation DemoSource: https://github.com/chrisconley/cat_or_not

12/13/2010

Admin Console• Review Work

• Perform “Trusted” tasks

• Worker Insights: Avg Completion Time, # of workers

• Remove bad performing workers from specific task groups

12/13/2010

Getting Started

• Get an API Key - http://houdiniapi.com/sign-up/

• Tunnlr - http://tunnlr.com

• Sandbox - http://houdini-sandbox.heroku.com

12/13/2010

Pricing Example

Monthly Fee Images Per Month

$20 200

$99 1500

12/13/2010

Easy Mechanical Turk Integration with Houdini

Technology

CheckMarket Mechanical Turk Getting Started Guide

Best Practices for Mechanical Turk

Mechanical Turk Under the Hood

Amazon Mechanical Turk Overview

Getting Started with Mechanical Turk

Amazon Mechanical Turk Workers Can Provide Consistent and

Using Amazon Mechanical Turk for Product Term Annotation

How Public Sector is using Mechanical Turk

Crowdsourcing using Mechanical Turk: Quality Management and Scalability

Amazon Mechanical Turk : Les petites mains du web

Amazon Mechanical Turk Participation Agreement

Amazon mechanical turk intro to bpo's v3

Quality control questions on Amazon’s Mechanical Turk

Alex Sorokin: OpenCV/Mechanical Turk Presentation

Using Mechanical Turk to Obtain and Analyze English …colala.berkeley.edu/papers/gibson2011using.pdf · Using Mechanical Turk to Obtain and Analyze English Acceptability Judgments

Data Processing with Mechanical Turk

Engagement Effects of Player Rating System-Based ...€¦ · Banner Ad Banner Consent Instructions Game Survey Mechanical Turk Mechanical Turk Consent Payment Code. Paid Voluntary

Amazon Mechanical Turk User Interface ... - s3.amazonaws.coms3.amazonaws.com/awsdocs/MechTurk/latest/amt-ui.pdf · Welcome This is the Amazon Mechanical Turk Requester User Interface

Amazon Mechanical Turk Getting Started Guide

Amazon mechanical turk intro to govt partners v2