Who We Are

Jim Jones ( jim.jones1@gmail.com )

Creator of the Turkee gem, a Rails form library that allows adeveloper to easily integrate with Mechanical Turk using thestandard Rails form helpers. The gem also makes it easy toretrieve the data submitted to Mechanical Turk and map thosevalues back to your existing models.

Mark Percival ( m@mdp.im )

Creator of the RTurk gem, a simple wrapper and library forAmazon’s Mechanical Turk API.

What is Mechanical TurkThe Amazon Mechanical Turk (MTurk) is one of the suites ofAmazon Web Services, a crowd-sourcing Internet marketplace thatenables computer programmers (known as Requesters) to co-ordinate the use of human intelligence to perform tasks whichcomputers are unable to do. The people that perform these tasksare informally referred to as “Turkers”.

Mechanical Turk Demographicshttp://waxy.org/2008/11/the_faces_of_mechanical_turk/

More Demographicshttp://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html

United States: 46.80%India: 34.00%Miscellaneous: 19.20%

The Mechanical Turk InterfaceWorker Interface https://www.mturk.com/mturk/findhits?match=false

Requester Interface https://requester.mturk.com/bulk/batches

Common UsesTagging of imagesCollection of survey dataSentiment analysis

Creative Uses of Mechanical Turk10,000 Sheep All drawn by Turkershttp://www.thesheepmarket.com/

Dr. Suessify the Newshttp://groups.csail.mit.edu/uid/deneme/?p=671

Russia to launch 520-day mock mission to Mars ! Oh, ThePlaces Russia Will Go!Fisherman’s wife breaks the silence ! One Fish Two Fish SickFish Who FishBP tries again to cap well; protests set to start ! BP tries tocap with a hat while protesters start to raise hell about the well

“Take a Bow” http://nowtakeabow.com/

This isn’t created with Mechanical Turk, but you can envision thepossibilities of utilizing Mechanical Turk workers for the creation ofa piecemeal video such as this.

Iterative Uses for Mechanical TurkExploring Iterative and Parallel Human ComputationProcesses Greg Little1, Lydia B. Chilton2, Max Goldman1, RobertC. Miller1

We are interested in human computation processes whichcoordinate small contributions from many humans to achievelarger goals. For example, an algorithm might coordinate manyworkers to write a description for an image.

Ruby IntegrationRTurk gem http://github.com/mdp/rturk

Creating HITs

h = RTurk::Hit.create(:title => hit_title) do |hit| hit.assignments = num_assignments hit.description = hit_description hit.reward = reward hit.lifetime = duration.days.seconds.to_i hit.question(f_url, :frame_height => HIT_FRAMEHEIGHT)

hit.qualifications.add :approval_rate, { :gt => 80 } end

Reviewing HITs

hits = RTurk::Hit.all_reviewableputs “#{hits.size} reviewable hits. \n”unless hits.empty? puts “Reviewing all assignments” hits.each do |hit| hit.assignments.each do |assignment| puts assignment.answers[‘tags’] assignment.approve! if assignment.status == ‘Submitted’ end endend

Rails Integration Turkee GemTurkee gem http://github.com/aantix/turkee

The Turkee gem builds on top of RTurk providing a simple formhelper for posting/representing model data on Mechanical Turk’sexternal servers.

The create_hit method takes a model,creates a Mechanical Turk hit pointing it atthe new resource URL for the modelpassed in. e.g. new_iteration_url

task = Turkee::TurkeeTask.create_hit(host, turkee_task[:hit_title], turkee_task[:hit_description], 'Iteration', turkee_task[:hit_num_assignments], turkee_task[:hit_reward], turkee_task[:hit_lifetime])

The form helper directs the form actiontowards the external Turk url and expandsthe fields in the given block in the samemanner that Rails’s form_for helper does.

<form accept-charset="UTF-8" action="https://workersandbox.mturk.com/mturk/externalSubmit" class="new_survey" id="new_survey" method="post"> <input type="hidden" id="assignmentId" name="assignmentId" value="2G2UC26DG67D103D8QB0FOK2JTAD66"/> <input type="hidden" id="hitId" name="hitId" value="29E6NG1FS3NYNMBGNC6ZVZL21YQLZU"/> <input type="hidden" id="workerId" name="workerId" value="A3DWBLF5GT7ACT"/> <input type="hidden" id="turkSubmitTo" name="turkSubmitTo" value="https%3A%2F%2Fworkersandbox.mturk.com"/> <input id="iteration_turkee_task_id" name="survey[turkee_task_id]" type="hidden" value="9" /> <p><textarea cols="40" id="survey_value" name="survey[value]" rows="20"></textarea></p> <p><input name="commit" type="submit" value="Create" /></p></form>

Retrieving Data from MechanicalTurkThe process_hits method retrieves the user posted data fromMechanical Turk, reconstructs the data into a param hash (like aController), determines the model for which the data represents,and creates an entry for that model in the database.

Turkee::TurkeeTask.process_hits(@turkee_task)

rake turkee:get_all_results

Turkee Iterator (updated for Rails3.1)Turkee Iterator is a demo built on top of Turkee to sample some ofthe capabilities of Mechanical Turk. Allows for flexible HIT requestsfrom Mechanical Turk.

https://github.com/aantix/turkee_iterator_rails31

Christmas Demos

Gift For WifeDirections : Should be under $25 Should include a link to theitem on Amazon Should look like I really care.

Either vote for your favorite submission OR give me your own giftidea (and Amazon link) for a chance to win a bonus 50 cents.

Thanks for the help!

http://turkee31.herokuapp.com/iterations?id=9

Difficult DilemmaDirections: I’m having difficulty telling my kids that Santa doesn’treally exist. On top of that, they’e been terrible this past year.

Please write a short letter to my kids from Santa explaining why hecan no longer give them gifts anymore(any reason will do). I willtranscribe the best one it and leave it in their stockings onChristmas morning.

Merry Christmas!

http://turkee31.herokuapp.com/iterations?id=11

Let’s Do a Realtime Example, SayHellohttp://turkee31.herokuapp.com/turkee/turkee_tasks/new

Please say “Hello” to my colleagues and tell us where you’refrom.

I am demo’ing Mechanical Turk to the Ruby SF meetup group.Please tell send us a creative “hello” and where you’re from.

A one or two line introduction would be nice. Tell us how theweather is in your city. Tell us about your family. Or just simplysay hello.

0.05

100

5

It’s mostly sunny at 65 degrees. Hello from San Mateo, CA!

More Content Creation

Mechanical Turk is an untappedmarket in terms of getting contentcreated for your application. Startthinking outside the context ofscientific surveys and fact findingand start thinking of this resourceas a pool of human workers withemotions, opinions, and stories totell.E.g.

Iterative story telling where each Turker adds a piece to anever ending story.

Have the Turker give their opinion on a topic or product andthen have other Turkers respond to that opinion.

Have the Turker describe some emotionally happy or painfulpoint in their life. Ask emotionally charged question to have theTurker reveal the essence of who they are.

Example: Obesity Testimonials(Design of Questions)

Upload a picture of yourself of when youfirst considered yourself “fat”.

When the above picture was taken, were you aware thatyou had been gaining weight? Describe the feeling you hadwhen you first glanced at the picture and noticed yourweight gain. Did anyone else notice or make snidecomments? How did you feel?

“As a child I was pretty average but I was lead to believe that Iwas fat. I would request simple items, things that are statussymbols to the average kid, like jeans. These small requests wouldbe shot down, my mother telling me that I was too big to fit into apair of jeans. I remember her telling me that when I was eight. Itstarted a cycle in my life where I felt that I was too large, too fat,for the things other children took for granted.”

Other Concepts

Micro BonusesThe concept where the user is awarded a varying amountdepending upon how much time they take to answer a question orhow accurate it (relative to the other user answers)

E.g. Maybe there’s a trivia game where you record how long ittakes each user to answer a question and the fastest response forthe day gets 10 cents instead of 5.

GamblingAllow users to gamble their potential pay.

E.g. Sentiment analysis.

Show the user all of the recent headlines for a given stock, stockcharts, etc.

You state that they can either take their payout now, OR, if theythink that the stock will close out higher than it is now at the endof the day, they can “gamble”, and will be awarded the fee PLUS abonus at the end of the day. If it closes down from the point atwhere it is, they receive nothing.

Does a future, larger payout increase the quality of data analysis?

Interesting Research Findings

Financial Incentives and thePerformance of CrowdsWinter Mason, Yahoo! Research

Figure 2 reveals two main findings: first, that across all difficultylevels participants chose to complete more tasks on average whenthe pay was higher (F(3,607) = 15.73, p < 0.001); and second,that across all payment levels, the number of completed tasksdecreased with increasing difficulty.

As Figure 3 indicates, however, increasing compensation didnot improve accuracy, which we measured in two ways…

Toward Automatic Task Design: AProgress ReportEric Huang, School of Engineering and Applied SciencesHarvard University

Task Search in a HumanComputation Market

Lydia B. Chilton, University of Washington **

We found strong evidence that Turkers sort by the most number ofHITs available (so they can find one task, and then do 100instances of them in a row) and the most recently posted HITS (sothey get the latest and greatest HITs).