Systematic Analysis of Output Agreement Games: Effects of Gaming Environment, Social Interaction, and Feedback
Shih-Wen Huang and Wai-Tat Fu
GWAP Today – ESP Game
[Von Ahn and Dabbish, CHI 2004]
GWAP Today - Foldit
[Cooper et al., Nature 2010]
Two Unanswered Questions
1. Can GWAP collect outputs with high quality? Why?
2. Which component of GWAP motivate volunteers to work?
• To answer these questions, a systematic analysis that tests the effects of different components of GWAP needs to be done
Output Agreement Games
Game Interface
Database
1. Randomly match multiple strangers and provide them with the same input
Output Agreement Games
Game Interface
Database
1. Randomly match multiple strangers and provide them with the same input
Output Agreement Games
2. Players are asked to generate outputs that are related to the input
Game Interface
Database
1. Randomly match multiple strangers and provide them with the same input
Output Agreement Games
3. Players will be rewarded if the outputs are matched. The matched outputs would be used to describe the input
2. Players are asked to generate outputs that are related to the input
Three Major Components
• Gaming Environment (G): Encouraging players to generate matched answers using scoring system, leaderboard, etc.
• Social Interaction (S): Players need to cooperate with other players
• Feedback (F): Players can implicitly learn the answers of other players
Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
Teammate searching page Teammate waiting page
Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction
Experimental Design
• 150 workers (30 per interface) were recruited from Amazon Mechanical Turk within two weeks.
• Each worker earned $0.05 for generating 20 labels
• A label was considered as a high-quality label if its distance to a gold standard label < 1
Gaming Environment Helps Collect More High-Quality Labels
Interfaces with Gaming Environment collect significantly more high quality labels than other three interfaces
Amount of high-quality labels collected by each interface
Social Interaction Motivates More Volunteers
Interfaces with Social Interaction motivate more volunteers than other three interfaces
Percentage of workers reported that they
would love to do the task again voluntarily
Implications for Human Computation
1. Gaming Environment allows output agreement games to collect more high-quality outputs
2. Social Interaction helps motivate more volunteers in output agreement games
Questions?