Upload
luca-galli
View
635
Download
1
Embed Size (px)
Citation preview
A Draw-and-Guess Game to Segment Images
Luca Galli, Piero FraternaliDavide Martinenghi, Marco TagliasacchiPolitecnico di Milano, ItalyEmail: [email protected]
Jasminko NovakEuropean Institute for Participatory Media,GermanyEmail: [email protected]
Luca Galli
● CUbRIK is a research project financed by the European Union
● Goals:
● Advance the architecture of multimedia search
● Exploit the human contribution in multimedia search
● Use open-source components provided by the community
● Start up a search business ecosystem
● http://www.cubrikproject.eu/
The CUbRIK Project
Luca Galli
Humans in Multimedia Information Retrieval
● Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features
● Solution: humans have superior capacity for understanding the content of audiovisual material
● State of the art: humans replace automatic feature extraction processes (human annotations)
● Our contribution: integration of human judgment and algorithms
– Goal: improve the performance of multimedia content processing
Luca Galli
4
● Problem statement: segment fashion images for mining trends based on visual features of garments (e.g. color and texture)
● Use case: identifying trends in collections of images of people and garments
● Applications: retrieving similar garments, inspect clothing trends in image collections, analyzing trends change in the years
The Fashion Trend Mining Scenario
Color descriptors
Texture descriptors
coarse(sub-)image similarity
Luca Galli
5
Problems in automatic feature detection: Body part and feature recognition is affected by the posture
of the portrayed subject
Problems arise when part of the body is occluded or in pictures with complex human poses
Issues of traditional approaches
?
Luca Galli
6
● Body part recognition
● Garments segmentation
● Gender identification
Solution: Human Contribution Issues are solved by augmenting the capabilities of
algorithms with human intervention. Humans have superior capacity to understand complex
content and to perform high level abstractions on visual representations
For these reasons, they are suitable for tasks such as:
Luca Galli
The CUbRIK Architecture
Content and users registration in the system
Managing tasks and processing contents
Provides front end for issuing queries and viewing results
Search Engines used to access the content and annotations
Luca Galli
8
Users Motivation: Games with a Purpose
● Users need motivation to complete a task in a human computation platform, such as monetary rewards, prestige, entertainment.
● Games with a Purpose (GWAPs) are digital games that generate useful data as a by-product of play.
● The design of a GWAP requires to create a game so that its structure encourages computation, correctness of the output and players retention.
● Introduced by Luis Von Ahn with the ESP Game, several GWAPs have been developed with a number of different purposes
Luca Galli
9
The Fashion Trend Mining Pipeline Male, 24 Female, 22 Female?, ??
Luca Galli
10
Gwap Design: Sketchness!
● Multiplayer Puzzle Game
● Established genre: Guess and Draw(Pictionary, iSketch…)
● Inversion Problem Mechanic
● Players take turns into drawing the shapes of objects inside an image in order to make the other players guess the underlying object.
● Two different roles:SketcherGuesser
● Objective: Garments tagging, body partsegmentation, garment segmentation
Luca Galli
11
Player Role: Sketcher● The only player to see the
low confidence image
● “May” be asked to provide atag for the image
● Is asked to draw the contourof the object for which the tag is provided within theallotted time
● Goal of the Sketcher is to let the other players guessthe tag without providingany other hints than thecontour
Tag of the target object
Fashion image provided by the pipeline
Contour of the garment provided by the player
Luca Galli
12
Player Role: Guesser
● Any other player in the game
● His/Her goal is to guess the object for which the Sketcherhas provided the contour
● Not allowed to draw on the whiteboard, just to typein the chat box the probableanswer as fast as possible
● Scoring:
● Sketcher: 10 pts + 1 for each guesser
● Guesser: 10 pts to the first, thendecreasing down to five
Contour of the garment provided by the Sketcher
Luca Galli
13
Task Injection: new approach to GWAPs
● Traditional gwaps developed on ad-hoc basis and tailored for a specific task
● Resulting gwaps may be not fun, unappealing to players or provide an experience that may still perceived by the users as work
● The proposed GWAP exploits suitable modified game mechanics of a well known and appreciated category of games, called “drawing and guessing games” to implicitly solve segmentation problems while playing
Luca Galli
14
● Task Injection:
● Does the task injection approach guarantee the same level of entertainment and participation for the players as the original instance of the game?
● How to measure entertainment anyhow?
● Output Aggregation:
● How the results provided by different players on the same input be processed and grouped together in order to obtain a unique result?
● Majority voting not suitable for lines!
● Integration within CUbRIK’s Gaming Framework:● Exploiting the features provided by CUbRIK in order to manage
players and their abilities and provide retention and reward features through gamification techniques.
Future Directions● The proposed game concept is still under development and there are
still several open questions and future improvements: