CUbRIK Applications First Release

FIRST CUBRIK APPLICATION DEMONSTRATORS Human-enhanced time-aware multimedia search

CUBRIK

Project IST-287704

Deliverable D10.1 WP10

Deliverable Version 1.0 - 30 September 2013

Document. ref.: cubrik.D101.CVCE.WP10.V1.0

First CUbRIK Application Demonstrators D10.1 Version 1.0

Programme Name: ...................... IST Project Number: ........................... 287704 Project Title:.................................. CUBRIK Partners:........................................ Coordinator: ENG (IT)

Contractors: UNITN, TUD, QMUL, LUH, POLMI, CERTH, NXT, MICT, FRH, INN, HOM, CVCE, EIPCM, EMP

Document Number: ..................... cubrik.D101.CVCE.WP10.V1.0 Work-Package: ............................. WP10 Deliverable Type: ........................ Accompanying Document Contractual Date of Delivery: ..... 30 September 2013 Actual Date of Delivery: .............. 30 September 2013 Title of Document: ....................... First CUbRIK Application Demonstrators Author(s): ..................................... Lars Wieneke, Claudio Massari, Jasminko

Novak, Marilena Lazzaro, Vincenzo Croce Approval of this report ............... Summary of this report: ..............

History: .......................................... Keyword List: ............................... Availability .................................... This report is public

This work is licensed under a Creative Commons Attribution-NonCommercial-

ShareAlike 3.0 Unported License.

This work is partially funded by the EU under grant IST-FP7-287704


Disclaimer

This document contains confidential information in the form of the CUbRIK project findings, work and products and its use is strictly regulated by the CUbRIK Consortium Agreement and by Contract no. FP7- ICT-287704.

Neither the CUbRIK Consortium nor any of its officers, employees or agents shall be responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission herein.

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7-ICT-2011-7) under grant agreement n° 287704.

The contents of this document are the sole responsibility of the CUbRIK consortium and can in no way be taken to reflect the views of the European Union.


Table of Contents EXECUTIVE SUMMARY 1

1 HISTORY OF EUROPE APPLICATION 2

1.1 USER STORY 2 1.2 OVERVIEW 4 1.3 INTERFACES 5

1.3.1 Indexation: Manual upload: Process New folder 5 1.3.2 Social graph and context expander 9

1.4 STATUS AND OUTLOOK Y3 13 1.4.1 Use cases 13 1.4.2 Components 13 1.4.3 Data sources 16

2 SEARCH FOR SME INNOVATION APPLICATION 17

2.1 USER STORY 17 2.2 OVERVIEW 18 2.3 INTERFACES 19

2.3.1 Image crawling from SN 20 2.3.2 Trend Analysis for category 20

2.4 STATUS AND OUTLOOK Y3 31 2.4.1 Use cases 31 2.4.2 Components 32

First CUbRIK Application Demonstrators Page 1 D10.1 Version 1.0

Executive Summary

The following document describes the first release of the demonstrators for the two CUbRIK V-Apps. The document will provide a general overview for each application but will link to the specifications of the applications (in particular D2.2 and D2.3) for further details.

The documentation of the components focuses on the practical use of the components. Whenever possible a connection between the implemented feature and its reference in the requirements document will be given.

Chapter 1 introduces the user story for the History of Europe app, provides an overview about the different use cases and components and underlines the different crowd-sourcing approaches that were used in the app. After this the interfaces of the app are described. The chapter concludes with a status of the different components in Y2 and an outlook on Y3.

The description of the fashion app in chapter 2 follows the structure of the previous chapter and presents the user story, the use cases with their different components, the crowd sourcing approach taken in the app and the different interfaces of the application. The chapter concludes as well with the status of the components and an outlook on their future use.


1 History of Europe application

The breadth and scale of multimedia archives provides a tremendous potential for historical research that hasn't been fully tapped up to know. The History of Europe application demonstrates the integration of human and machine computation that combines the power of face recognition technology with two distinctively different crowd-sourcing approaches to compute co-occurrences of persons in historical image sets. These co-occurrences are turned into a social graph that connects persons with each other and positions them, through information about the date and location of recording, in time and space. The resulting visualization of the graph as well as analytical tools can help historians to find new impulses for research and to un-earth previously unknown relationships. As such the integration of human expertise and machine computation enables a new class of applications for the exploration of multimedia archives with significant potential for the digital humanities.

This document describes the first HoE demonstrator released in M24. For further details about the specifications and detailed requirements please have a look at D 2.2 or D2.3.

The application is accessible at the address: http://cubrik1.eng.it:8084/SMILA/hoe/index.html

1.1 User story

Figure 1: User story "Who is this person? Tell me more"

As outlined in D2.2 and D2.3 the HoE demonstrator builds on the user story "Who is this person? Tell me more"

A newly scanned photo shows a few people at a dining table. Amidst familiar faces Ibrahim notices one person that is unknown to him. He uploads the image to the History of Europe application to find out who this is. He selects the person’s faces in the scan and provides the


names of some of the people he already recognized.

After a while, the image analyser returns suggestions for the identity of the missing persons. One of the suggestions proposes that the unknown person might be a Dutch politician. Based on the reference images that the application offers him Ibrahim is able to confirm this identity.

Ibrahim wants to find out more details about the photo and starts the exploration of the social graph. There, he finds more photos of the same event. Based on the number of co-occurrences of persons in these and other photos, he is able to identify possible candidates who, to his surprise, could also have known the politician he identified in the initial picture. For every person Ibrahim selects in the graph he sees the results of the context expander, which in addition to photos provides him with a list of media items such as newspaper articles, interviews, commentaries and videos.


1.2 Overview

The user story is implemented through four different use cases with dedicated pipelines and components (see figure 2 for an overview).

Acquisition of co-occurrences

Media

harvest ing

and upload

Face detect ion

Face

identifi cat ion

Clickworkers

Crowd Face

posit ion

validat ion

Image Indexation

Social Graph construction

Entitypedia

data

provisioning

Social Graph

Creation

Copyright aware

crawler

Provenance checker

License checker

Content

provider

tools

Metadata Entity

ex tract ion

Data Set Creation Face recognition

Identity reconciliation

Entity

verifi cat ion &

annotat ion

Entitypedia Integrat ion

CROWD pre-

fi ltering

Indexation of textual information

Connection

to the CVCE

collect ion

Entity

anntation and

ex traction

Entity extraction

Expert Crowd

Expert

CROWD

verifi cat ion

Entitypedia Integrat ion

Entity reconciliation

Uncovering who a person is

Visualization of the social graph

Graph

Visualizat ion

Query for entities

Graph

Visualizat ion

Query

execution

Social Graph Network Analysis Tk

Graph

Visualization

Analysis of

the social

graph

Expert CROWD Research Inquiry

Expert Crowd

CROWD

Research

Inquiry

Expanding the context […]

Expansion

through

documents

Expansion

through

videos

Expansion

through

images

Expansion

through

related

ent it ies

Figure 2: Overview HoE app in M24, grey boxes mark components for Y3

Overall the CUbRIK platform envisages three different kind of crouwdsourcing approaches depending on the peculiarities of the crowd, e.g. through cultural predispositions and common background knowledge and according to the specific kind of interaction mechanism that is put in place for the execution of the task. The crowds that the CUbRIK platform can address to outsource tasks can be classified as:

• Games With A Purpose (GWAPs) Players: persons engaged in playing with the different games supported by the CUbRIK game framework, including Sketchness, for object detection in images, SteamPilot, for image tagging, and WordGames, such as Hangman and Crosswords, for entity refinement.


• Crowd Workers: a crowd of paid workers. The Amazon Mechanical Turk1 and

Microtask Crowdsourcing2 markets have been used for a variety of tasks, ranging from face detection to feedback collection in videos playout.

• Expert Crowd: a crowd of reference domain experts that interact with CUbRIK platform according to a Query & Answer (Q&A) mechanism. In HoE the Q&A interaction mechanism is exploited to post queries and retrieve answers from a crowd of experts to solve the task of people identification; the expert crowd is addressed with multiple mechanisms, including email invitation and message posting on social networks.

Figure 3: CUbrIK Crowd typologies

In the context of HoE two of those crowdsourcing approaches were exploited. In particular CrowdWorkers and Expert Crowd were involved as depicted in Figure 1: User story "Who is this person? Tell me more". For the first release of the demonstrator, a subset of components has been implemented. Further components will be integrated in Y3 and the existing components will be refined based on the evaluation of the first demonstrator.

In the following we will present the interfaces that where developed for the different pipelines and components.

1.3 Interfaces

1.3.1 Indexation: Manual upload: Process New folder

Overview

A user starts the upload of new images through the "Process new Folder" function (see Figure 4: HoE app "Process new folder"). Once the images are uploaded, the Face detection component starts and returns face positions in the different images.

1 https://www.mturk.com/mturk/

2 http://www.microtask.com/


Figure 4: HoE app "Process new folder"

A crowd of clickworkers on the Microtask platform reviews these face positions in two ways: to identify false positives the crowd members evaluate whether the bounding boxes placed on an image actually cover a face or not (see Figure 5: Crowd Face position validation). For false negatives the crowd is able to annotate missing faces in the image. Overall a face position is verified after three independent confirmations from different clickworkers.

Figure 5: Crowd Face position validation

The following Face identification component returns a list of potential identities for the faces in the image. Due to the difficult dataset (non-normalized gaze directions, different levels of image quality, persons are in different ages showing therefore significant changes in appearance) the results of the identification process are not reliable enough and require manual confirmation.


Figure 6: CROWD Research Inquiry

Figure 7: Entity verification & Annotation

Two levels of confirmation are introduced: in the Expert CROWD Research Inquiry component an external crowd of experts is invited to participate in the confirmation task and either votes on one of the identities already provided in the list or adds a new identity in case it was not available before (see Figure 6: CROWD Research Inquiry. The user who uploaded the images is also able to annotate images, faces and identities and to verify the votes of crowd users (for face position annotation see Figure 7: Entity verification & Annotation).


Components

Figure 8: Process New Folder

Name Illustration Functionality

Media harvesting and upload

Figure 4: HoE app "Process new folder"

Upload of images, minimal implementation of the data set creation workflow

Face detection - Detects face positions in uploaded images

Crowd Face Position validation

Figure 5: Crowd Face position validation

Click-workers validate the detected face positions

Face identification - Algorithm identifies persons in verified face positions

CROWD Research Inquiry

Figure 6: CROWD Research Inquiry

Expert users are invited to give their opinion on the identity of the identified faces

Entity verification & annotation

Figure 7: Entity verification & Annotation

App users can add annotations and verify information

Entitypedia Integration - Write annotations to Entitypedia

Face detect ion

Face

identifi cation

Clickworkers

Crowd Face

position

validation

Face recognition

Expert CROWD Research Inquiry

Expert Crowd

CROWD

Research

Inquiry

Identity reconciliation

Entity

verifi cation &

annotation

Entitypedia Integration

Media

harvesting

and upload


1.3.2 Social graph and context expander

The social graph is visualized by means of a force-directed graph layout that groups nodes with stronger link weights closer together than nodes with weaker links in such a way that a global optimum is reached. The visualization is performed by the JavaScript components

Crossfilter and D3.js3 that display the graph in the form of an interactive visual network of persons and connections between them. Thereby, each node represents a person and each edge represents the frequency of their co-occurrences in the given photo collection. The width of an edge represents the number of co-occurrences, in such way that a wider edge means that both persons appear simultaneously in a high number of photos (see Figure 8: Visualization of the social graph). The integration of the social graph on the ENG platform is currently on-going.

Figure 8: Visualization of the social graph

Such network visualizations have the potential to highlight unexpected connections and patterns in all kinds of relations, be they social or between other nodes such as documents. In order to further evaluate any such finding it is however important to be able to interactively explore the relationships between the nodes, manipulate the parameters determining their visualization and re-contextualize the graph based on specific data subsets that are of interest to the user.

Users can interact with the History of Europe social graph in different ways, e.g. clicking on a node results in an ego-graph of the selected person (see : Ego graph) and clicking on an edge displays documents that relate to the selected relationship between two persons (see Figure 10: Connection between two persons and related persons). As the documents stored in the collection very often come with a date of creation, the graph can be filtered by date with the timeline, displaying only the connections of documents created within this timespan. This timeline also shows the amount of photos per date that are contained in the collection. Another filtering option is the number of connecting documents, which allows the visualization of those relationships that are only included in an interval of a minimum and maximum

3 see http://d3js.org/


number of documents. This feature is useful to highlight highest co-occurrences. Finally, the number of appearances of a person in the processed collection enables users to identify people who appear particularly often in any given time frame (see Figure 9: Ego graph).

Figure 9: Ego graph

Figure 10: Connection between two persons and related persons

Filtering of images used for the construction of the graph is also possible by defining the number of persons that were recognized in the images. This filter is mainly used to show/hide those group pictures that include a high number of people on the same picture.


After selecting a person, the graph turns into an ego-graph (see Figure 9: Ego graph), centring on the selected person and showing all relationships for the person. All photos in which the selected person appears are displayed below the graph, sorted by date and in consideration of other filters. Users can review who is tagged in each of these photos by moving the mouse over the pictures, this highlights the different persons that appear in the selected photo in the graph. In addition to these photos, a list of links to text documents that contain references to the selected person is displayed below the photos (see Figure 11: Expansion through documents, display of related documents for specific persons).

Figure 11: Expansion through documents, display of related documents for specific persons

Another powerful feature is the inspection of relationships. Whenever a user clicks on an edge between two persons, the selection shows all images and documents where both persons appear in and that therefore constitute the relationship between the persons (see Figure 10: Connection between two persons and related persons).


Components

Figure 12: Visualization, network analysis and context expander

Name Illustration Functionality

Graph visualization Figure 8: Visualization of the social graph

The social graph of co-occurrences is visualized.

Analysis of the social graph

Figure 8: Visualization of the social graph, Figure 9: Ego graph, Figure 10: Connection between two persons and related persons

The user is able to analyse the social graph with a set of filters.

Expansion through documents

Figure 11: Expansion through documents, display of related documents for specific persons

Once the user selects a person or the edge between two persons, related documents from the CVCE collection and other websites are shown


Visualization of the social graph

Graph

Visualization

Social Graph Network Analysis Tk

Graph

Visualization

Analysis of

the social

graph

Expanding the context […]

Expansion

through

documents


1.4 Status and outlook Y3

1.4.1 Use cases

Use case Y2 Y3

Indexation Boot strapping of the social graph using the CVCE dataset (3k images), initial implementation of Copyright aware crawler, provenance and license checker as well as content provider tools. Indexation of Europeana content.

Integration of the EC Audiovisual library, Integration of all components in the pipeline

Indexation of textual information

- Co-occurrence analysis in CVCE documents and interview transcripts

Social graph construction

Inital version ready Refinement and full integration in SMILA


Visualisation of the social graph, initial set of analytical tools, initial integration of Expert Crowd Research Inquiries

Refined visualization and analysis, full integration of all components in the SMILA pipeline

Expanding the contex […]

Initial implementation of the Expansion through documents component

Implementation of all other components and integration in SMILA.

1.4.2 Components

For guidelines on how to access and install the individual components please see D9.4

Component Y2 Y3

Media harvesting and upload CERTH

Crawled Europeana content for given queries

SMILA pipeline for the Europeana API Integration of the EC Audiovisual library

CONTENT PROVIDER TOOL FRH

- Reliably bind arbitrary license information to local content - Uses digital signatures - Implemented as command-line tool

- Various usability improvements related to content preparation and upload (based on feedback from content providers)

Object Storage FRH

- Binary (media content) and json doc storage, based on a MongoDB core - Support for user and component authentication,

- Integration of License Checker and Provenance Checker functionalities


Component Y2 Y3

and access control (prerequisite for many content providers) - REST interface and SMILA client (both SMILA pipelines integration and direct integration with UI / apps possible)

Copyright aware Crawler FRH

- Crawling only of content that complies with specific required permissions (e.g., rights to modify and use in a commercial context) - Supports relevant image sources, including Flickr and Wikimedia commons

- Support relevant video sources, including news video podcasts and Vimeo

Provenance checker FRH

- Content duplicate and reuse detection using hashes - Keeping track of content source, creation and publication time

- Including crowd interaction to complete and validate license information - Perceptual duplicate and reuse detection using phash - Integration with object storage (simplifies communication and usage)

License checker FRH

- Retrieval of Creative Commons information, and interpretation into usage permissions - Keeping track of license metadata and derived usage permissions for content

- Support for Europeana and proprietary license information mapping - Integration with object storage (simplifies communication and usage)and usage)

Face detection POLMI

Completed Review of the "Shore" component as an alternative to KeeSquare. Option to extend to video in Y3

Crowd face position validation POLMI

Completed "face validation" and "face add"

-

Face identification POLMI

Completed Integration of co-occurence information (social graph) and textual annotations in the identification process

CROWD pre-filtering L3S

Systematic study to understand annotation difficulties for different target users (generic crowd / expert crowd). Development of candidate features that enable a prediction whether a photo is easy or hard to annotate.

Integration of the full pipeline to distribute tasks to generic crowds and or experts.


Component Y2 Y3

Entity verification & annotation HOMERIA

UI for showing the results of face detector/identification of images. UI for annotating data related to one image (place, date, event...), and data related to a person (name, date of birth...)

UI for enabling verification of entities by voting and adding explanation to the possible values of annotations. (Gamified interface)

Entitypedia integration UTN

Entitypedia entity model extended with HoE entity model. HoE entities are stored in Entitypedia. Basic entity search functionality on top of HoE entities.

Word games on top of HoE entities

Connection to the CVCE collection CVCE

Provision of around 10.000 historical documents containing persons, places, organisations, events names and periods of time.

Connection to the CVCE back-end (for entities extraction of new documents)

Entity annotation & extraction EMPOLIS

- Extraction of entities from CVCE documents and other text inputs also event extraction

Expert CROWD verification CVCE

- Application for the validation of named entities within documents by an expert crowd

Expert crowd entity verification POLMI, CVCE

Implementation of conflict manager (CrowdSearcher) to dispatch entity verification (via email - addresses provided by CVCE). Naive computation of relevance score (output of face identification + crowd feedback)

Improvements to the conflict manager: - Handle different task dispatching mechanisms (e.g., Twitter); - Keep track of identity of expert workers (to build expert profiles); - Targeted dispatching of tasks depending on worker expertise.

Social graph creation QMUL

Creation of the social graph from information stored in the Entitypedia.

Integration with SMILA,

Query for entities EMPOLIS

Initial version Full integration of search for query, connection with Entitypedia, Auto completion of Entity names

Visualisation of the social graph HOMERIA

Visualization of people as nodes of the graph, co-occurrences of people as relationships of the graph and exploration of images related to a selected person or relationship.

Alternative visualization of the graph for events, places, etc.

Social Graph network analysis toolkit CERTH

Basic graph analysis Graph analysis toolkit, that provides local topological features, global features (e.g. graph diameter) and clustering based on modularity. Spectral clustering, calculation of shortest distance


Component Y2 Y3

Expansion through documents EMPOLIS

Complex Query document expansion phase 1

Complex Query document expansion phase 2

Expansion through video FRH

Deferred to Y3 in favor of News History Demo

- Expansion of HoE with News Content History (analysis of shared / reused footage within news videos) - Connection via related events / tags - Possibly connection via face recognition

Expansion through images L3S

First implementation of the component based on a detailed annotations study

Refinement of the component

Expansion through related entities L3S

- First release of the component

1.4.3 Data sources

Y2: Bootstrapping of the application with the CVCE data collection

3924 images, r.a. 1700 persons

Y3: Extension

Integration of material from the EC Audiovisual Service

131k images, unknown amount of individual persons

Fallback

Despite some recent progress, the agreement of the EC Audiovisual Service is still under review. As fallback a large set of images from the Europeana has been crawled that will be used in case no final agreement with the EC Audiovisual Service can be achieved in Y3.


2 Search for SME innovation application

The Search for SME Innovation application is a web application focussed on the fashion domain, a sector constantly searching for new ideas, innovation of products and new trends to be launched in the market, and its purpose is supporting SME’s to exploit the potential of the new technology to get feedback from and learn about the needs of fashion consumers.

Through the tool, the organizations are able to receive information about the opinions and the feedback of their potential customers, their preferences, what they like about clothes and the current trends. In this application, primarily existing content from social networks is used (e.g. fashion pictures that are crawled from Twitter) in order to make this application an efficient tool for market analysis for SME’s.

Each organization involved in the production of new fashion lines and products, from the main producers to distributors and B2C organizations, invest budget in marketing research for a better and deeper understanding of the potential customers‘ preferences. For this reason, the target groups of the tool are both large organizations in the fashion sector and small and medium sized enterprises (SME):

• Large organizations (great buyers, distributors, clothes producers addressing general public) already make use of market analysis, with their own experts or with the support of external ones; these organizations may benefit of the more affordable results provided by the application or they may use both the traditional market analysis and the new one, having another useful tool to retrieve information they need;

• SMEs may not have enough budget to perform market analysis, basing their analysis just on the experience of their personnel; such organizations, through the application, have the opportunity to gain affordable analysis of their potential customers and trends.

This document describes the first Search for SME innovation demonstrator released in M24. For further details about the specifications and detailed requirements please refer to D 2.3.

The application is accessible at the address: http://cubrik1.eng.it/TrendAnalysis/index.html

2.1 User story

In the deliverable D2.3 two user stories have been presented as overview of the desired functionalities of the demonstrator of the Search for SME Innovation application. In the second year the first demonstrator has implemented the following user story:

Margaret, the experienced retail buyer who usually has a notion of consumer preferences but prefers to verify her own ideas before placing an order.

“Fashion4You” is a small clothing company. Margaret, one of their retail buyers, is preparing a new order of women’s casual wear for the upcoming season. She’s trying to decide which models of skirts should be bought from the local distributor. Margaret is very experienced in her job and she already has an idea of how to structure the order. But there are many aspects to be considered, such as cut, colours and textures, and decisions need to be based on trend predictions and consumer preferences.

Usually, the manager of the company would commission a trimestral market research from an external agency that Margaret can rely on when planning her order. This time however, Margaret must rely only on her intuition and experience since it was decided to dispense with market research due to recent economic shortcomings.

Margaret remembers that her friend Kate, who is also a retail buyer, spoke very enthusiastically about CUbRIK Fashion, a new, innovative application she uses to get an overview of current consumer preferences. The application extracts current fashion trends by garment category, e.g. colour and texture preferences, from social network posts. As a result,


it provides statistics depicting the trends over time, as well as examples of the most popular trends. Margaret decides to try out the application to verify if the information extracted from the social networks matches her own notion of customers’ likely preferences.

Using the Social Network Trends functionality, Margaret selects the category “skirts” and asks the application to provide a trend analysis based on social network posts from the last two months. Exploring the results of the analysis, Margaret can confirm that “turquoise” has been a very popular colour, as she thought. Moreover, the Colour Combination Analysis shows that this colour is most frequently combined with “purple” and “dark green”. Viewing the Print & Graphic Trends, she verifies that the texture preferences haven’t changed much in the past six months, which is also reflected in her own analysis of the sales in her store.

With the Popular Photos Gallery, Margaret can browse images of consumers’ most preferred skirt models, compare them over time and as part of outfits. This helps her discard some models from the order and at the same time take into account others she didn’t consider before.

Having retrieved the information she needed, Margaret is satisfied: the order she prepared needs only little adjustment, but her notion about the kinds of models and colours to be ordered was almost correct.

In this document the first demonstrator is presented and the use cases realised in the year 2 are reported and described.

For more details about the development process, the user stories and the derived use cases as well as a functional description of the different components, the D2.2 and D2.3 should be considered.

2.2 Overview

For the first release of the demonstrator and the realization of the user story “Margaret, the experienced retail buyer who usually has a notion of consumer preferences but prefers to verify her own ideas before placing an order” two use cases have been implemented and a subset of components has been used:

• Image crawling from SN: this use case aims at crawling fashion related images from social networks and further processing them in order to extract segments representing different garments and their features. Crowdsourcing is used in the detection of the clothing items present in the images extracted by the social networks, improving the possibility and the quality of the segments detection in low quality images or in images where parts of the body cannot be recognised by the software algorithms. The images are annotated relatively to categories of clothing items, while textures and colours are derived from the images automatically.

This use case is technical and not user related: it aims at crawling and processing the images so that they can be used in the other use cases.

In relation to the full use case as reported in deliverable D2.3, in the first demonstrator the component Feedback filter – Likelines is not yet integrated. It will be part of the demonstrator in the 3

rd year. The component is shown in the overview, but

it drawn in a different colour.

• Trend Analysis for category: this use case aims at providing the SMEs users with a trend analysis of clothing preferences based on the clothing category the SME user has selected. The trend analysis functionality analyses the images extracted from the social network and further processed for the features and segments extraction, in order to provide the users with an insight report of the fashion trends based on the user preferences on the social network.

This use case uses the images extracted and processed by the use case described above.


In the following figure the overview of the use cases implemented in the second year.

Figure 13: Overview of the first demonstrator Search for SME innovation app at M24, grey boxes mark components for Y3

As depicted in Figure 1: User story "Who is this person? Tell me more" in general three kind of crowd are involved in CUbRIK. In the context of Fashion V-App the task of body part detection is partially outsourced to humans in case of images that are too difficult to be processed, in particular when the confidence of the body part detection is considered to be not sufficient. In this case GWAP players were exploited through the Sketcheness GWAP.

Figure 14: Sketchness GWAP

2.3 Interfaces

The Graphic User Interfaces constituting the application is presented in the following paragraphs, where the GUI of the Search for SME Innovation V-App as envisioned for the Fashion SMEs is described both from the user perspective and the technical point of view. The application focuses on the GUI elements required for the “Trend analysis for category” use case.


2.3.1 Image crawling from SN

Overview

As already described in the previous section, the “Image crawling from SN” use case is technical and not user related, thus no part of the GUI is associated to it.

Components

The figure below reports the components implemented in the use case and in the following table the functionality they provide is described.

Figure 15: Image crawling from SN components at M24

Component Name Illustration Functionality

Image extraction from SN

- It crawls the images from the Social Network (Twitter).

Descriptors extraction - It provides the extraction oft he features (colours and textures) of the images and the garments segments

Body part detector and router

- It provides the automatic extraction of the garment segments, by detecting the parts of the body in the images

GWAP Sketchness - It provides the manual extraction of garments segments, by involving the humans in the loop

Accessibility annotation

- It provides the annotation of the images with metadata accessibility-related

2.3.2 Trend Analysis for category

Overview

As explained, the application implements the functionalities needed for the realization of the user story selected for the first demonstrator. The SMEs users can dynamically generate trend analyses based on crawled social network data (Twitter), by selecting a garment category and time span (Figure 17).

The analysis provided to the SMEs users identifies colour trends, colour combination trends as well as prints & graphics trends for the identified timespan and selected garment category.

The Figure 16 shows the main page of the GUI.


As shown in the figure, the SMEs users are provided with three main functionalities, selectable through the main menu:

• The main functionality is the “Social Network Trends” that implements the core of the Trend Analysis for category use case: it presents to the users the trend analysis of the category selected;

• The “Dashboard” that offers respectively the possibility to further access to the analysis already retrieved;

• The “Popular Photos” that offers a different point of view of the trends analysis, showing the most popular images.

The Main page of the GUI is the Dashboard.

Figure 16: Main page of the GUI.

Social Network Trends

This functionality is the core one available to the user and developed in the first demonstrator. It provides the user with an insight on trends for the selected category and the time range set up.

As shown in Figure 17, the user has firstly to select the category he is interested in receiving the Trend Analysis. The garment categories available are:

• Shirt

• T-shirt

• Trousers

• Skirt

• Shorts

• Suit

• Dress

The first two categories belong to the upper body garment typology, the next three to the lower body garment typology and the last two are related to the complete body garments. Such choice on one side ensures the coverage of all three basic typologies of garments


(related to the lower part of the body, the upper part of the body and the complete body) and on the other side represents the most used list of categories as suggested by an organization working in the fashion sector informally interviewed.

In the second release of the demonstrator the user will be able to select also “all categories”, that will provide him a comparison and an analysis of the trend and the preferences of the garment categories implemented. This function in the first release of the demonstrator is disabled.

Figure 17: Social Network Trend – choice of the category

After having selected the category garment, the user can select the timespan and then can ask the system to generate the trend analysis according the parameters selected, as shown in Figure 18.


Figure 18: Social Network Trend – setting up the timespan

The results of the trend analysis are shown in Figure 19.

The trend report provides information about the most preferred colours, presented in a pie chart with an associated percentage, a chart showing the colours trends in the timeline selected, the most preferred combination of colours and the colour trend images for each of the colours reported. Finally, the user can have an insight on the textures of the top images (the most preferred images).

A trend analysis can be run, with timespan and clothing category as filters.


Figure 19: Results of the trend analysis.

The user can also view the most popular social network photos used for the analysis in the “Photo Gallery”. For this gallery, as shown in Figure 20, the samples of the most popular photos are reported, with the information of the timeline considered and the most preferred colours.

The images reported can be displayed as a gallery (Figure 20), or as thumbnails (Figure 21), accordingly to the view selected by the user.

A trend report is provided for colour, colour trends, colour combinations and prints & graphics.


Figure 20: the “Photo gallery” of the trend analysis – gallery view.

Figure 21: the “Photo gallery” of the trend analysis – thumbnail view.

Most popular pictures can be displayed, based on the no. of Tweets

The images can be displayed as gallery


The dynamically generated analyses can be saved so users can review them later. Saved analyses are listed and previewed in the application dashboard (Figure 23). In the first demonstrator the analysis are automatically saved.

Clicking on each image, a pop up comes up and shows its details, as shown in Figure 22.

Figure 22: the “Photo gallery” of the trend analysis – pop up of the image details

Dashboard

Through the Dashboard, the SMEs users can have an overview of the analysis launched in the past. As shown in Figure 23, The Social Network Trend Analysis performed in the past are listed with the information about when a specific analysis was launched and what garment category it was referred to.

The user is provided also with the possibility to delete a specific past analysis by clicking the red “X” at the right of the analysis.


Figure 23: Dashboard – main page

By clicking on the white arrow at the right of a specific analysis, the user can view a resuming of the results of the analysis, as shown in Figure 24.

Clicking on “Details”, the Trend Analysis page will be opened and the information of the stored analysis selected shown as when it was launched.

Saved analyses can be re-visisted by click on the white arrow

Saved analyses can be deleted by clicking on the red “X”

List of the saved analyses


Figure 24: Dashboard – selecting analysis retrieved in the past

Popular Photos

Using the Popular Photos functionality the users can have an overview of the most popular images extracted and analysed in the trend analysis process, performed every day by the technical component.

This functionality allows the users to have an overview of the most popular images with the aim of supporting the users to better understand which are the trends and the most preferred looks in the social network.

The functionality is shown in Figure 25. As explained in the balloons, the user can interact with the application by selecting the timespan and the garment category he is interested in; moreover, the user can change how the results are reported by choosing among different views, so the results can be displayed as a gallery (Figure 25), grouped by garment type (Figure 26), or as thumbnails, as shown for the Social Network Trend Photo Gallery (Figure 21). The results reported can be filtered by popularity and sorted by date or popularity. According the sorting parameter, the horizontal scroll bar shows the time range or the popularity range.

Clicking on each image, a pop up come up and shows its details, as shown in Figure 27.

Saved analyses can be re-visisted by clicking on “Details”


Figure 25: popular photos functionalities

Figure 26: popular photos – images grouped by garment type

The timespan can be changed

The garment categories can be selected

Images can be selected according users’ preferences

Different views of the images are available

Images can be sorted


Figure 27: popular photos functionalities – pop up of the image details

Components

The figure below reports the components implemented in the use case and in the table the functionality they provided is descripted.

Figure 28: Trend Analysis for category components at M24

Component Name Illustration Functionality

Trend Analyser Figure 28: Trend Analysis for category components at M24

It provides an analysis of the trend in the fashion sector for the category of clotes addressed

By clicking on a image, its details are shown


2.4 Status and outlook Y3

2.4.1 Use cases

The following list reports the status of the use cases at Y2 and the foreseen improvements and developments in Y3.

Component Y2 Y3

Image crawling from SN

Crawling fashion related images from Twitter and extraction of the garment segments, colour and textures

Crawling fashion related images from Flickr and integration of Likelines for fashion key frame extraction from videos. Refinement of the feature extraction and segment identification

Trend analysis for category

Trend analysis provided for a set of garment categories.

Analysis of the popularity of the images extracted from the Social Network Twitter based on the numbers of twits and analysis of the most popular colours, combination of colours and textures.

Analysis of “all categories” to be produced to the GUI

Refinement of the algorithms for improving the quality of the results

Trend analysis for sample

- To be implemented.

Analysis of trends provided for images similar to a sample uploaded by the user. The analysis is based on the popularity of the images in the Social Network.

Popularity Assessment


Assessment of the popularity of images uploaded by the user by gathering the preferences of the user’s network on the social media about the images uploaded.

Fashion Matching - To be implemented.

Searching and matching images functionalities for allowing the users to select the possible matches and request users’ assessment using their own network in the social media.

Personalised query suggestion


Suggestion of queries based on the previous users’ behaviours


2.4.2 Components

For guidelines on how to access and install the individual components please see D9.4. The following list reports the status of the component at Y2 and the foreseen improvements and developments in Y3.

Component Y2 Y3

Implicit feedback filter- Likelines

Not integrated in the use case. It extracts key frames from videos fashion-related-

To be integrated into the use case

Image extraction from Social networks

Images crawled in Twitter using predefined categories of garments.

Flickr crawling and integration with the copyright and licensing component

Body part detector and router

Segments selection and routing to crowd based on quality score of the images.

Refining of the algorithms for improving the quality of the detection of the garment segments

Descriptor extractor Colours and textures extraction from complete images and related segments

Refinement of the algorithms for results improving

GWAP Sketchness Game for the manual segmentation of garments

Refinement of the Game and introduction of annotation verification of the images

Trend Analyzer Analysis of the popularity of the images extracted from the Social Network Twitter based on the numbers of twits and analysis of the most popular colours, combination of colours and textures.

Analysis of “all categories” to be produced to the GUI

Refinement of the algorithms for improving the quality of the results

Image similarity detection

- Searching in the internal database of the application images similar to the one chosen by the user, accordingly the category, the colour and the texture parameters.

Relevance feedback - Reordering the images ranking the similarity through the usage of human perception of similarity

Conflict manager - Publication of links of the vertical application in the social network, to reach the network of the demonstrator users

Accessibility Annotation of the images with the metadata describing the accessibility-related feature of the images

The accessibility filtering functionality will be implemented for reordering of the images according the sight


Component Y2 Y3

impairment selected.

Behavior analyzer - Analysis of the users’ behaviours for further query suggestions

User profiling - Register and login functionality

CMS for Human Computation

- Gathering of the users’ actions for the further analysis of the Behaviour Analyser to be used in the query suggestion

Documents

CUbRIK Applications First Release