Download pdf - Big Media: Multimedia goes Big Data

Technische Universität Darmstadt

Prof. Dr. Max Mühlhäuser

BigMedia:Multimedia goes Big Data

Video as example & driver:

YouTube: 100 h/min.

Internet: >50%, > 40 XB/mo.

LTE advanced!

Cities: >500,000 surveillance cams

e.g., London: 6 months screening effort

Plus: zillions of sensors

Crowd: cf. foto heatmaps

BigMedia @ ISM 2014 © M. Mühlhäüuser 2

A Few BigMedia Facts

www.statista.com/chart/624 ( Cisco Visual Networking Index)

www.sightsmap.com

http://www.statista.com/chart/624

http://www.sightsmap.com/

Multimedia & BigData: Do They Blend?


processinput output

capture interacttransport store

process

multimedia

sense analyze

big data

Volume Velocity

Variety Veracity

conventionaldata processing

Unified BigMedia pipeline:1) capture/sense 2) transport 3) store 4) process 5) interact/analyze

BigMedia

1. CAPTURE/SENSE



processsense analyze

BigMedia

Sensors (CaptureDevices):

1. Hard:1a) „scene capturing“

Camara, Mike1b) „value sensing“

Accelerometer, …

2. Soft:2a) Documents

browse, view, edit2b) Software

contextualcommand-lvl. stroke-lvl. use


2x2 Categories of Sensors

Events in-situ

w/ Smartphone

Desire: link to…

people on site

(+net)

CoStream:


Crowd Media Example: CoStream

Login Awareness Streaming

Portrait upright


CoStream Key Interaction Concepts

Portrait parallelto the ground

Landscape

• Rotate to watch• Tap to stream

Invitations, Sharing:• Push & Pull

(friend list video)• Rating (Like/Dislike)• Vibration to indicate

BigMedia is Multisensory Media AudioVideo + SocialMedia + Location/Motion/…


Another Example 1st General Trend

+ conventional multimedia“media type”: tweetsDomain:Small-ScaleIncident Detection

BigMedia is Blended Media:

Generation (Sources): User Generated

Authoriative

Automatic

Consumption (Targets):@Site

@Net

@Home / @Mobile


2nd General Trend

BigMedia is “Mass” Media: (user generated, automatic, authoritative) selection @ receiver?

social & personal pref’s

automatic (BigData)

or: “summaries” for massive reduction

e.g., emotion metering

e.g., hotspot / trend indicators

e.g., event analytics


3rd General Trend

BigMedia means …

mass amounts of (streams of)

blended (-source/-target)

multi-sensory (hard + soft)

media


Intermediate Summary

2. TRANSPORT

1. net/edge processing

(2./3. basically skipped)




BigMedia

The Issue: Transport vs. Store&Process


Reality: “Field” Stakeholders: “POI”Infrastructure: “Net”



?!

Cloud?


Net/Edge Processing & Latency Needs

real scenery encounteredby mobile user (here: outdoor!)

real scenery captured by camera

processed: 3D, semantics, location, orientation, …

virtual scenery overlayed3D, in real time, as user moves

Dual Reality (DR)

Goal: dual reality DR (100% VR + 100% reality)

+ universal semantics, in- & outdoor, 3D overlay

Grand challenge:

universal semantics (“my phone can see!”)- geometry, location, domains …

3D robust real-time registration~today: popular buildings only


Net/Edge Processing & Latency Needs

Junaio, Layar, Wikitude, TK

indoorexample

photo courtesy of TUD GRIS & FhG IGD

We are facing the age of latency DR (see above) + speech

federated interaction

realtime media stream analytics

Cloud: not enough!

add ‘local cloud’

required: mobility, handover, replication, self-X

driver: excess processing capacity @net

cf. Microsoft™ Cloudlets, Cisco™ Fog Computing

SWOT?

- pro: low latency, ‘cheap’ & ctx-aware processingownership@origin (see below)

- con: distributed processing


Net/Edge Processing: Cloudlets

2. Resilient Networks:

3. Highly Adaptive i.e. Fluid Networks


Further Key Transport (=Net) Issues

Smart GridIndustrial Facilities

Smart CitiesSmart Transport

Fluctuation:- Density

- Intensity- Mobility

3. STORE

1. The Forgotten Forgetting

2. Distributed Storage (&Processing) Privacy?




BigMedia

BigMedia: always-on recording LiveLogs, cf. Microsoft™ Sensecam

Analogy:

cam+mike eyes+ears

brain ‚intelligent‘ processing

Open Issue: data organization user swamped

Google, FB … surrender, timeline

(cf. Gelernter‘s lifestream)

even less organization!

remember brain: associative & forgetful



Eyes/ears Bionics:

auto-organize + auto-consolidate (‘forget’)

idea: reinforce retrieved data + ‘neighbors’

but: too many neighbors!

idea: leverage user interaction

1. Acquire: meta data

2. Interact: N-dim. space

3D visualization, user picks 3-of-N

neighbors: cf. user’s selection!

3. Consolidate:

‘fade’ least reinforced data



approach


Ad 2: Interlude

processing& storage

@net/edge

latency & bandwidth

limits

excesspower

@net/edge

AlwaysOn

/ LiveLogs

Problems, e.g.:how to process,

to archive, …

Further oppor-tunities, e.g.:privacy thruownership


2. BigMedia Privacy

Privacy rights, laws, or desiresw.r.t. PII: personally identifyable info.

AnonymizationData Thrift

big data

collect more datadon’t throw

any data away

Prof. Narayanan, Princeton: essentially impossible … in a foolproof way w/o losing … utility of data1

1: Privacy and Security: Myths and Fallacies of “Personally Identifiable Information” .CACM 53 (6), 2010, pp. 23-25

1. Cyberphysical Spaces

2. Cyberphysical Humans


2. BigMedia Privacy: Things are getting worse

Consent?

latent PII

photos © economist, siliconangle, ebiz-results, REX, Thinkstock/Ninell_art

remote processingvs. local data

“quantified self”& Assistence

anonymousstore?


Challenge

tustedstore

inte

rfac

e

?

4. PROCESS

Just a brief recap / overview, for the sake of time




BigMedia

Selected: 3rd Processing Category


observablesperceivables

1. hard

2. soft

learning

person group popul.whohow

offline

realtime

conceivables



1 of 3 processcategories:

machine learning

trace movement intentionexample:

BigMedia pipeline unify&standardize 3 paradigms1. conventional & statistical processing

2. machine learning

3. crowd processing

Remember: processing @net/edge (Cloudlets …) requires

modularization, smooth mobility, appropriate algorithms & models, …


Process Categories, @Net Processing

5. INTERACT/ANALYZE

technology proliferation new interaction concepts

Mobile - Natural - Large Scale

(many other trends ignored for the sake of time)


/capture interact

transport storeprocesssense analyze

BigMedia

5A. MOBILE INTERACTION

Resizable Displays

AR DR Displays

On-Body Interaction


Lab equipment:

targets user experiments& controlled user studies:UI concepts for devices-to-be!


Resizable Displays (1): Rollable

‚display size dilemma‘: an innovation driver

Resizable Displays (1): Rollables, contd.

…

(this was just a selection)further UI concepts concern,

e.g.: semantic zoom,visual clipboard,

horizontal scroll, …


OLEDs … thin devices folding?

The question, again: interaction concepts towards UX?


Resizable Displays (2): Foldables


FoldMe Design Space

PicoProjectors? use environment as display

daylight, power efficiency

collaboration? privacy?

empty space / wall

hand-held hand jitter

HOWEVER: Add depth cam tangible information space


Resizable Displays (3): Pico Projectors

©Samsung (Galaxy Beam)

Google glass (note: full computer) hype & $$$

multimodal

‘app ready’ in 2 sec. (vs. 22)

Two sets of open issues:interaction concepts & experience1. degree of ‘AR’

2. advancement of vision & speech

3. direct manipulation!!

4. sharing experience?- SeeWhatISee sufficient?

5. acceptance as eyeware fashion

6. privacy

Note: DR-ready sophisticated glasses 2-6 remain!


ARDR displays: Google Glass lessons

Google folks (today!): “no line-of-sight, no AR”

proliferation of technologies interaction concepts

1. rollable! promising

2. glass! heavy invest vs. open issues

3. pico projector: information spaces?

4. foldable: interaction concepts

not promoted doomed to fail?


Future Mobile Displays: Summary

… 3D displays? games as driver

Use Palm-of-Hand (plus fingers) for interactiona) buttons

b) sliders

c) numbers,

d) ..

On-Body Interaction: Hand

a)

b)

8c)


Wireless Sensor

Arc-shaped Board (12 Touch Points)

Interaction Techniques

Evaluation

here: control audio @ “human audio device ”

38

On-Body Interaction: Ear

BigMedia @ ISM 2014 © M. Mühlhäüuser

5B. MORE NATURAL INTERACTION

implicit

tabletop

paper like

spoken

printed tangible


Basis: user(s)‘ pose, emotion, attitude, …

‚Interaction‘: appearance

actions

Example (1): CouchTV


Implicit Interaction: Idea, Example 1

Implicit control

Implicit suggest

Implicit pause and record

Rollables again, but collaborative

bridge phone tabletop

Implicit interaction: auto-adapt UI to physical ‘connectedness’


Implicit Interaction (2), collaborative setting


Table Based Interaction: Concern

Desktop PC: isolated

immersive tabletop:social

table: social

digitize

tabletop: isolated

augment

awareness/accessibility: interactive halo & icon, gradual & remote access, ‘exposè’ organization: teleporting, hybrid piling / hiding , hybrid binding


Table Based Interaction: ObjecTop

Table Based Interaction: PeriTop

add top projection & depth camera


-digital info atop occluders-about object or else-low resolution OK here

Table Based Interaction: CoMAP


46

Table Based Collaboration: Permulin

personal

shared

personalized

output

personalized

output

personalized

input

personal

shared

personalized

output

personalized

input

personal

sharedshared

personalized

output

personalized

input

personal


…

47


Divide View

Merge Views

Interaction Concepts Divide / Merge


48

View of User A

View of User B

49


SharePeek

Sharing &Peeking…

Interaction Conepts Share / Peek


50

View of User A

View of User B

51

Table Based Collaboration: Permuli

Better Parallel WorkLarger Interaction Area

User A User B

…

Mutual AwarenessTruly Fluid Transition

Kinect for user recog-nition

3D Display

Multi-touchframe

Kinect for hand recog-nition

Modified3D shutterglasses

hardware setup today

… and tomorrow?

evaluation results


Several projects atop Anoto ePen technology!

just one pen for ….


Paper Like (1): Pen&Paper Computing

Paper table (& wall) hybrid: paper+table physical objects

hand writtenannotations

tagging usingmenue cards

printed userinterfaces

folders books

Paper Like (2): Paper like displays


Multiple Paper-like Displays

?

Custom printed objects …

… if interactive(!): boost „tangible interaction“

… were demonstrated by U Saarbrücken

… and by TK (@TU Darmstadt)

… are still at an early stage


Printed Tangible Interaction

a b c

d e

Spoken Interaction?Example Smart-Space Dialogs

Speak to smart spaces -homogeneous UI vs. heterogeneous devices

requires:1.context awareness

2.user awareness (multi-speaker!)

3.mike awareness (array … headsets)

plus (not included here):

federation w/ other modalities

openness (cf. Siri++: for app/service developers)


C. LARGE SCALE INTERACTION

(just briefly touched here)

In-door: e.g., walls

Out-door: e.g., facades

Global: e.g., social or virtual

Overarching challenge:Collaboration of / with a large user base


Still lots of passive walls

Interactive walls still inappropriate

Remember lesson learnt today:

novel technology new interaction concepts

Collaborative wall interaction:

only few concepts known, rarely in use

large walls: real estate reach

quest for mobile device federation!


Wall size interaction

Interactive Walls & Rooms

QUT Brisbane “CUBE”

negative example

City Scale Example: Media Facades

2 approaches:

1. Indirect interaction input “abstract aggregation”

cf. emotion metering, hotspots, …

2.Temporal individual control often as “competition”

user creativity framed by app


Global Interaction

Here, CoStream@Home: in-situ socialNet home


Tv Broadcast

CoStream

User Generated Videos

Notifications

Friend List

by default“compressed” to vibration

Cheering Frustration Clapping

D. ORTHOGONAL ISSUES

(one slide for brevity)


Collaboration, Federation, Intelligence

Collaboration: mentioned above

local distributed, team social

quest for research (cf. our ConCalls!)

Federated Interaction leverage multimodality

challenge: open ad-hoc federation, latency

Intelligent UIs: Proactivity: adjust UI implicitly + in advance

Intelligibility: UI explains itself & its reasoning


Web basedfederated UIs

SUMMARY




BigMedia

Capture / Sense:mass amounts of blended-source/-target multi-sensory media

Transport: processing @edge/net (Cloudlets) + fluid networks + (if 24/7) resilience

Store: (partial) user site storage: novel processing, ‘forgetting’ & privacy opportunities

Process: Unify three BigData/Media pipelines: conventional + ML + crowd processing selected challenge: processing @edge/net

Interact/Analyze: many challenges, selected: proliferating technologies + interaction concepts mobile: resizable displays, AR DR, on-body interaction

(more) natural: implicit, table based, paper like, spoken

large scale: walls, facades, social networks

collaboration, federation, intelligence as orthogonal aspects


For Your Long Term Memory

imagine consequences on industry sectors:

Software industry (every app ready for 25 sets of interaction concepts?)

Media industry (OSN convergence done right?)

Telecom industry (Cloudlets embraced?)

Critical infastructures (500k surveillance cameras plus mobile reports?)


Food for Smalltalk