Emotion-based multimedia browsing (R&D Proposals)

8/14/2019 Emotion-based multimedia browsing (R&D Proposals)

1/38

Project ProposalsProject Proposals

Emotion-Controlled Digest Generation for MultimediaContents

Emotion-Controlled Variable Multimedia Transmission

Virtual Meeting Place

Agent Technology for Entertainment/Consumer Services

An Agent Digest Distribution System for Audio and VideoContents


2/38

EmotionEmotion--ControlledControlled

Digest GenerationDigest Generation

for Multimedia Contentsfor Multimedia Contentsby

A. Barletta


3/38

Multimedia DigestMultimedia Digest

Give an fast preview of multimedia content

Video, audio, books, pictures, etc

Automatic algorithm Who is consuming the digest ?

Are the information contained in a movie

sufficient for generating a digest ?

What about the Human factor ?


4/38

EmotionEmotion--controlled Digestcontrolled Digest

Generation (I)Generation (I)

Digest


5/38


Generation (II)Generation (II)Emotion Engine

DigestGenerator

Model

Bio Sensors:skin,

blood pressure,retinal activity, etc

Emotions: Attention, Stress, Suspense, etc

audio

videoStream:-Video-Audio-Text

Using the inputfrom the emotionEngine we select asubset of the entireMovie, song, book


6/38


Generation (III)Generation (III)networkednetworked --

Emotion Engine

Real TimeDigest

Close Loop

Control

Media Server


7/38

Content AnalysisContent Analysis


8/38

Emotion Engine (I)Emotion Engine (I)

Already available several solutions in terms of emotion detection

Several input methods:

Galvanic skin Response

Electrocardiogram

Elecromyogram

BVP (Blood, Volume, Pressure) Photoplethysmyograph - Respiration Hall effect respiration sensor -

Temperature (using sensors or infrared cameras)

General Somatic Machine (GSM)

Facial Action Coding System

Posture, Gesture, Voice, Face

Pupillary dilation

Stomach motility

Penis size Cortisol level in saliva

Mature models applied to education, Human Machine Interaction, AutomaticMultimedia Selection, CE interface, etc

No current usage in the digest generation


9/38

Emotion Engine (II)Emotion Engine (II)


10/38

Digest Generator ModelDigest Generator Model

It collects emotions and synchronized with content

It uses the data collected for filtering composite media (video,audio, text) and for creating a digest

The multimedia content is subdivided in time quanta and ateach of these quanta is associated the collected humanfeedback

It models the brain activity during the consuming ofinformation: we try to synthesize this activity

With approximated emotion detection and a good empiricalmodel we can simulated the human mind and select only themore important information that needs to be stored


11/38

Possible ApplicationPossible Application

Producing general digest for preview and

eCommerce

Producing real time digest for specific users

(networking close loop case) for late complete

vision

Fast consuming of multimedia contents (no digest)

Video thumbnails

Emotion based CE feedback (gaming)


12/38

ReferencesReferences

http://affect.media.mit.edu/AC_research/sensing.html

http://www.almaden.ibm.com/cs/BlueEyes/index.html[BlueEyes-enabled television ]

web.media.mit.edu/~carsonr/pdf/hcii01.pdf []


13/38


14/38

EmotionEmotion--Controlled VariableControlled Variable

Multimedia TransmissionMultimedia Transmission

by A. Barletta


15/38

MultimediaMultimedia

Fix stream adapted to the available

bandwidth

Who is the consumer ? A machine or ahuman ?

Human have not a constant consuming rate

Variable bandwidth multimedia distribution

dependent on the user attention


16/38

EmotionEmotion--Controlled MultimediaControlled Multimedia

DistributionDistribution

Emotion Engine

Variable

BandwidthUsage

Close Loop

Control

Media Server


17/38

Variable Quality/BandwidthVariable Quality/Bandwidth

The Quality Of Service of the streamedmultimedia content can be changed in a widerange

Human information assumption is not constant: itdepends on the degree of attention

We can adapt the quality of the content to thestatus of the user

Examples: Switching the video stream when the user is notactively focusing on the screen (audio is maintained)


18/38

Possible ApplicationsPossible Applications

Limited bandwidth devices (mobile phone)

Video stream


19/38


20/38

Virtual Meeting PlaceVirtual Meeting Place

How to meet your friends onAmazon.com!

(by A. Barletta)


21/38

Internet ExperienceInternet Experience

Web site are virtual meeting place

No support for communicating with visiting

people

Meeting at the library ? Why not at Amazon ?

Meeting at the News Kiosk ? Why not at

BBC.com ?

Looking for a VAIO ? Ask people visiting

Sony.com !


22/38

Today Internet ExperienceToday Internet Experience


23/38

Internet: A Virtual meeting PlaceInternet: A Virtual meeting Place

Anybody knows S. Lew ?

Did you receive thebook you ordered ?

I did, but I didnt likeit !


24/38

Existing SolutionsExisting Solutions

http://www.groove.net/

Business oriented

virtual shared spaces

Business application already available


25/38


26/38

Agent Technology forAgent Technology for

Entertainment ServicesEntertainment Services

A system for multimedia commerce

by A. Barletta, B. Moser


27/38

The problemThe problem

How to send digital contents ?

How to sell digital content ?

How to exchange digital content ? Which encoding mechanism ?

A Agent framework can solve this problem


28/38

Computing ParadigmsComputing Paradigms

Client/Server

HTTP Server, FTP, etc

Remote execution

postscript

Mobile code

Applet

Agents

Aglets


29/38

Client/Server Paradigm

Agent Paradigm


30/38

Agent ParadigmAgent Paradigm

Reduce terminal-server interactions

Embedded and Proactive Autonomous Behaviors

[Data + Code] is moved to each node of the network

Several prototype solutions for e-Business and networkmanagement

Bartering, Searching, Monitoring, etc


31/38

AgEntAgEnt : Agent + Entertainment: Agent + Entertainment

An Agent Distribution System for Audio

and Video Contents


32/38

An Agent Digest Distribution SystemAn Agent Digest Distribution System

for Audio and Video Contentsfor Audio and Video Contents Autonomous software agents deliver audio and

video digest

Expiration of contents

Close system (security, proprietary encoder)

Flexible: the terminals can be updated automatically

The full chain is under control of the distributor

Content formatting, distribution, decoding

The user control only the selection of the contents(Simple interface: select, try, buy, play)


33/38

System ViewSystem View

Music/Video Agent:- responsible of rendering the content and looking forthe right decoder-Encryption logic (public key, private key)- maintain expiration/control (without calling back the

media server)- in case of purchasing it connects via a secureconnection to the server and retrieve/load/displaycontents- User has not access to the contents itself- It manages group of friends to exchange

Personal Agents Box (PAB)- Under controlled of Sony- provide support formanaging/loading/

activating/destroy Agentsmodule- upgradable decoders- manage security issues

Agent Distribution System (ADS):- it produces Active Agents- it can use a proprietary Sonyprotocol for distributing (usertransparent)- it uses a Sony encodersynchronized with the decodersin the PABs

Music Agent- it contains code and the

digest data


34/38

Use case (I)Use case (I)1

2

3

4

Digest

Song

Agent Distribution System(ADS)

Personal Agents Box (PAB)


35/38

Use case (II)Use case (II)

Antonio has got a VAIO system with a Personal Agents Box (PAB) applicationprovided by Sony

Antonio navigates in a Sony Audio Web sites and he looks for new songs

The Sony Web site offers an Agent Distribution System based on agents

Antonio chooses a collection of MusicAgents he likes [2]

Antonio decides to buy the song: after payment transaction, the content isdownloaded inside the agent, and played. Antonio unplug the VAIO from thenetwork and enjoy the music; after an expiration time the agent disables theplayback and destroy the content; only the digest is available [3].

Antonio decide to send this Agent to a friend, Boris (only the agent with thedigest is sent) [4]

Boris open the agent, listen in the digest and decide to play the song: the agent

after a payment transaction, download the content from the Sony Web site andplays the song


36/38

Secure Agent DistributionSecure Agent Distribution

Each Personal Agent Box has a private andpublic key

The Agent Distribution System (ADS)

receives the public key of each PAB

A group of friends can exchange the Publickeys

Transmission of Agent is secured using thepublic key of the receiving PAB


37/38

Secure Content DistributionSecure Content Distribution

Case 1:

Each MusicAgent has one key

Request of content with key

Server encode with Agent key

Agent decode with its own key

Case 2

Server has a private key

MusicAgent has a public key of the server An asymmetric encryption is done


38/38

ReferencesReferences

http://www.trl.ibm.com/aglets/index_e.htm

Aglets Home Page: IBM framework for building agents

No product available on the market from IBM

Documents

Emotion-based multimedia browsing (R&D Proposals)