Embed Size (px)
DESCRIPTION
AWS World Wide Public Sector Mechanical Turk workshop session 3, best practices for converting workflows to Mechanical Turk
Citation preview
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Transformational Impact of Cloud Labor
John Hoskins & Daniel [email protected]
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Crowdsourcing Best Practices
amazonweb services
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Crowdsourcing myths
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
• It’s cheaper– It’s actually more efficient
• It’s faster– It’s actually more scalable
• It’s not accurate– It’s actually more accurate
The Myths[ ]
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
5
Crowdsourcing Best Practices
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
• Consider the question carefully– Workers answer what you ask
• Select your workers– Perspective and skills vary
• Iterate and Optimize– Adjust for optimal results
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
The question[ ]7
You will get an answer to the question that you ask. Focus on asking the right question
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Choosing your workers[ ]8
Workers are different– from language and cultural differences – to varying skills. Test and monitor.
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Monitor and Improve[ ]9
Monitor key metrics, adjust and measure key attributes impact on those metrics.
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
• Accuracy– Know your current
• Throughput– Understand both turnaround and scale requirements
• Cost– Measure against a budget – as cost can impact the other two
Key Metrics[ ]10
“Great service, Good food, Friendly staff – you can choose two”
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Cost[ ]11
Cost is impacted most by the efficiency of the other two metrics.
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Accuracy[ ]12
Error has two sources: human and systematic. Isolating human error and solving for systematic error gives a better chance for long term success.
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Accuracy[ ]13
After solving for systematic error choosing the best workers and monitoring those workers provides the next step towards high accuracy and lowering costs.
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Throughput[ ]14
Many factors impact throughput;
Reputation
Ergonomics
Clarity
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
http://www.mturk.com
15
John Hoskins, Amazon Mechanical Turk
amazonweb services
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Cost[ ]16
Cost is impacted most by the efficiency of
the other two metrics.
Optimization of task and workers lowers
both the cost of getting it done and
adjudicating a result.
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Thank You
