Describes paper presented at IEEE Social Computing 2011 conference on novel "serious game" called Odd Leaf Out to identify errors in classified image sets. See http://www.cs.umd.edu/localphp/hcil/tech-reports-search.php?number=2011-17 for the paper.
Citation preview
1. Odd Leaf Out:
Improving visual recognition with games
IEEE Social Computing
Oct 10, 2011
2. BioTracker Team
L to R: ArijitBiswas, Jennifer Preece, Cynthia Parr, Dana Rotman,
Erin Stewart, Darcy Lewis.
Front Row: David Jacobs, Derek Hansen, Jen Hammond, Anne
Boswer
Missing: Eric Stevens
3. BioTrackers Research Questions
How can a socially intelligent system be used to
direct human effort and expertise to the most
valuable collection and classification tasks?
What are the most effective strategies for
motivating enthusiasts and experts to voluntarily
contribute and collaborate?
4. BioTrackers Research Questions
How can a socially intelligent system be used to
direct human effort and expertise to the most
valuable collection and classification tasks?
What are the most effective strategies for
motivating enthusiasts and experts to voluntarily
contribute and collaborate?
5. Goal: Identify Errors in ImageClassification Datasets
Augmented
Plant identification
Citizen Science
Data Collection
Scientifically Validated Information
6. Games with a Purpose
7. Odd Leaf Out Game
8. Odd Leaf Out Game
9. Key Game Characteristics
Single Player
+ No problems with collusion strategies
+ No need for 2 players at a time
- Lack of excitement of live interaction
Learns from Players Wrong Answers
+ Gaming the system is harder
- Player frustration when they are actually right
10. Game Variations
11. Constructing Leaf Sets
Goals
Generate useful data
Right level of difficulty
Process
Calculate distance between each pair of leaves using features
identified via curvature-based histograms
Select an initial leaf at random
Select 4 others from the same species including the most dissimilar
one
Select the odd leaf from another species with varying levels of
distance from the initial leaf
12. Evaluation
Seed dataset of 120 image sets with 12 errors
Difficult errors created by comparing erroneous leaf to the mean
species distance of other leaves in same species
Recruited two groups to play online:
Family, friends, colleagues, students, alumni
Experienced botanists, plant scientists, ecologists
Players randomly assigned to regular or skip version
After first game, players rated difficulty & gave suggestions
for game improvement
13. Identifying Errors
Two Odd Leaf Error Sets (8)
Find most incorrectly selected images
No Odd Leaf Error Sets (4)
Find odd leaves that were in hard sets
14. Results
15. Results
16. Results
17. Results
Errors detected just as well based on novice and experts
Skipped rounds dont necessarily include errors
18. Results
19. Results
20. Discussion
Images other than leaves
Test other variations of game
Education as motivator
Other demographic groups? (children)
21. Questions and Discussion
Derek L. Hansen [email protected]
www.biotrackers.net
