Easy Mechanical Turk Integration with Houdini

HoudiniOn Demand Labor API

By: Chris Conley and Jameson Detweiler

Philly AWS December Meetup

12/13/2010

Easy Mechanical Turk Integration with Houdini

Date: December 13, 2010

Tuesday, December 14, 2010


12/13/2010

Easy Mechanical Turk Integration with Houdini

• Marketplace for work

• ~ 500k workers

• Accessible via API

Philly AWS December Meetup

Mechanical Turk



12/13/2010

Easy Mechanical Turk Integration with HoudiniPhilly AWS December Meetup

Task Examples

• Moderation - images, comments, reviews, video

• Categorization - products, best image, tagging

• Ads - moderation, keywords

• Audio Transcription / Video Captioning

• Cleanup - proofreading, contact reconciliation, review editing

• PDF/Image/Handwriting to Text

• Research - contact info, document search



12/13/2010


Turk Overview• Workers and Requesters

• HITs

• Questions

• Assignments

• Qualifications

• Notifications

• Blocking Workers



12/13/2010


Worker View



12/13/2010


Creating HITs

• Easy

• Great for testing the waters

• No integration

• More Flexible

• But Complicated

• Programmatic Integration

WYSIWYG API



12/13/2010


WYSIWYG



12/13/2010


WYSIWYG



12/13/2010


API

http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html





12/13/2010


HIT Questions

• XML

• External Question



12/13/2010


Turk API Libraries

• ruby-aws http://rubyforge.org/projects/ruby-aws/

• rTurk https://github.com/mdp/rturk

• Turkee https://github.com/aantix/turkee


http://rubyforge.org/projects/ruby-aws/

http://rubyforge.org/projects/ruby-aws/

https://github.com/aantix/turkee





12/13/2010


Questions about Turk?



12/13/2010


Houdini

• Simple

• Qualified results

• Automatic

• Trusted Workers



12/13/2010


Houdini APIs

• Simple

• Exact

• Multi-answer

• Image Moderation

• Ad Moderation

• Data Reconciliation

• Data Extraction

• ...

Generic Optimized



12/13/2010


Simple API

• One answer per task

• Cheap

• Image Moderation, Low profile content



12/13/2010


Simple API Example



12/13/2010


Exact API

• Multiple workers

• Ensures all answers match

• Returns the “final answer”

• If not, escalated to trusted worker group

• More expensive than Simple

• Image Moderation, Data Harvesting, etc



12/13/2010


Exact API Example



12/13/2010


Multi-answer API

• Ask multiple turkers same question

• All answers are returned

• Surveys, feedback, answer qualification on your end



12/13/2010


Image Moderation API

• No need to build form_html

• Just send the image url

• Optionally send custom instructions and possible responses



12/13/2010


Image Moderation Example



12/13/2010


Image Moderation DemoSource: https://github.com/chrisconley/cat_or_not



12/13/2010


Admin Console• Review Work

• Perform “Trusted” tasks

• Worker Insights: Avg Completion Time, # of workers

• Remove bad performing workers from specific task groups



12/13/2010


Getting Started

• Get an API Key - http://houdiniapi.com/sign-up/

• Tunnlr - http://tunnlr.com

• Sandbox - http://houdini-sandbox.heroku.com


http://tunnlr.com

http://tunnlr.com

http://houdini-sandbox.heroku.com

http://houdini-sandbox.heroku.com


12/13/2010


Pricing Example

Monthly Fee Images Per Month

$20 200

$99 1500



12/13/2010


Other Solutions

• Crowdflower

• Microtask

• CrowdCloud



12/13/2010


Resources

Houdini Docs: http://houdiniapi.com/documentation

Mturk Docs: http://docs.amazonwebservices.com/AWSMturkAPI/2008-08-02/index.html


http://houdiniapi.com/documentation

http://houdiniapi.com/documentation




12/13/2010


Thanks!Chris Conley Jameson Detweiler

[email protected] [email protected]

@chrisconley @jaymstr


mailto:[email protected]




Technology

Easy Mechanical Turk Integration with Houdini