CUbRIK Project

Vincenzo Croce Engineering Ingegneria Informatica S.p.a.

“In the same way Stanley Kubrick was one of the most innovative and pattern-breaking Directors in the Cinema industry, so the CUbRIK project will introduce real innovative patterns inside the Multimedia search domain”

CUbRIK – International Workkshop Search Computing

36 month large-scale integrating project started in October 2011

partially funded by the European Commission’s 7th Framework ICT Programme for Research and Technological Development

www.cubrikproject.eu

CUbRIK at a glance

226/9/2012CUbRIK – International Workkshop Search

Computing

The CUbRIK Project Concept: the Cube Metaphor

26/9/2012CUbRIK – International Workkshop Search

Computing 3

project concept is based a on cube metaphor progress proceeds along three dimensions: technological, human and

business ones At the same time, the cube metaphor recalls the open

composability of the Rubik-cube puzzle

Objective: Advance the architecture of multimedia search

Multimedia search: coordinated result of three main processes: Content processing: acquisition, analysis,

indexing and knowledge extraction from multimedia content

Query processing: derivation of an information need from a user and production of a sensible response

Feedback processing: quality feedback on the appropriateness of search results

426/9/2012CUbRIK – International Workkshop Search

Computing

Objective: Advance the Human platform

The CUbRIK vision is to incorporate human capacities and social interaction

CUbRIK aims at harvesting different forms of human computation:

Implicit, such as emotion recognition Individual, such as intelligence and decision making in gaming Social, such as in crowdsearch

526/9/2012CUbRIK – International Workkshop Search

Computing

What achieved so far

Scenario definition History of Europe – entities Fashion Application

Requirements Architecture H-Demos

CUbRIK Pipelines CrowdSearch Frameworks

Q&A GWAP

26/9/2012CUbRIK – International Workkshop Search

Computing 7

Objective: Advance the architecture of multimedia search

Content processing, query processing and feedback processing phases implemented by means of independent components

Components are organized in CUbRIK pipelines Each application defines ad-hoc pipelines that provide

unique multimedia search capabilities in that scenario

826/9/2012CUbRIK – International Workkshop Search

Computing

Objective: Advance the architecture of multimedia search

Example of CUbRIK Pipeline

926/9/2012CUbRIK – International Workkshop Search

Computing

conflict - situation during the analysis where absence or contradictions of facts recorded in Annotations may arise

Conflict Resolutoin Task -> Application – GWAP -> application-performer pair –

Q&A application

CUbRIK architecture

1026/9/2012CUbRIK – International Workkshop Search

Computing

Objective: Humans in the loop

Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features

Solution: humans have superior capacity for understanding the content of audiovisual material

State of the art: humans replace automatic feature extraction processes (human annotations)

Our contribution: integration of human judgment and algorithms

Goal: improve the performance of multimedia content processing

1126/9/2012CUbRIK – International Workkshop Search

Computing

Human activities

26/9/2012CUbRIK – International Workkshop Search

Computing 12

CrowdSearch Framework

26/9/2012CUbRIK – International Workkshop Search

Computing 13

Types of tasks•Automatic tasks•Crowd tasks: tasks that are executed by an open-ended community of performers

Types of tasks•Automatic tasks•Crowd tasks: tasks that are executed by an open-ended community of performers

GWAP Framework

26/9/2012CUbRIK – International Workkshop Search

Computing 14

Objective• extract the core aspects of a media refinement exploiting a videogame.

• task gamification - converting a task into a game• task injection - integrates the task goals inside pre existing games.

Challenges• Drive user’s motivation • Select the proper users

• based on their profile, competence and reliability.

• Validate the results provided by the crowd

Human-powered trademark logo detection demo

26/9/2012CUbRIK – International Workkshop Search

Computing 15

Problems in automatic logo detection: Object recognition is affected by the quality of the

input set of images

Uncertain matches, i.e., the ones with low matching score, could not contain the searched logo

Human-powered trademark logo detection demo

Goal: integrate human and automatic computation to increase precision and recall w.r.t. fully automatic solutions

1626/9/2012CUbRIK – International Workkshop Search

Computing

Human-powered trademark logo detection demo

26/9/2012CUbRIK – International Workkshop Search

Computing 17

Human-powered entities creations

26/9/2012CUbRIK – International Workkshop Search

Computing 18

Problems in automatic entities : Entity creation can be affected by wrong

synset or gloss relation between synset and concept relation between concepts entity names entity attribute attribute value

Human-powered entities creations

26/9/2012CUbRIK – International Workkshop Search

Computing 19

Solution: Word Games Popular Fun Educational use Played with:

Words real world entity names definitions (glosses)

Finalist - crossword: extremely popular fun words, clues, relations (in clues) maximum mistake coverage easy personalization themes or topics

Human-powered entities creations

26/9/2012CUbRIK – International Workkshop Search

Computing 20

Crossword generation select a layout select a slot pick a word check grid consistency

Feedback , check of Etype relations attributes values

Vincenzo Croce

Vincenzo.croce@eng.it

Thanks for your attention

www.cubrikproject.eu

2126/9/2012CUbRIK – International Workkshop Search

Computing