Multimedia and Big Data are closely related topic. Big data enables solving some important challenges in multimedia and basic principles of multimedia are the key issues in multimedia.
1 BAMMF June 20, 2014
Marshal McLuhan in 1964. A medium itself, not the content it carries, should be the focus of study. A medium aects the society in which it plays a role not only by the content delivered over the medium, but also by the characteris@cs of the medium itself. BAMMF June 20, 2014 2 Media
Mul+ BAMMF June 20, 2014 3 Media + + Digital technology allowed combining and using media leading to a major disrupAon.
Photos are the killer content type on mobile. Quick to consume like text, but easier to produce on a phone. BAMMF June 20, 2014 4 From Mary Meekers presenta@on
Big Data BAMMF June 20, 2014 5
To know our world, (MulAmedia) Data is the Source.
McLuhan was right. UnAl 20th Century -- Medium was the message. BAMMF June 20, 2014 7
BAMMF June 20, 2014 8 What is an Elephant?
BAMMF June 20, 2014 9 Six Blind Men Exploring an Elephant.
BAMMF June 20, 2014 10 An Elephant according to Six Limited PerspecAves. The Blind Mens Elephant
BAMMF June 20, 2014 11 But, an Elephant is an Elephant. A Real Animal.
MulAmedia and Big Data are both the Latest Elephant. BAMMF June 20, 2014 12 Lots of people excited about these but with narrow perspecAves.
BAMMF June 20, 2014 13
Two Major Human AcAviAes CommunicaAons ComputaAons BAMMF June 20, 2014 14 Both require MulAmedia AND Big Data.
CommunicaAon BAMMF June 20, 2014 15
Nature Gave Us Five Basic Senses BAMMF June 20, 2014 16 to Experience the World
We promptly created Silos BAMMF June 20, 2014 17
Storytelling: The Oldest and most powerful Art. A story is a chain of events that begins at one place and ends at another. A story is a selecAve batch of informaAon. It selects details, arranges them, perhaps embellishes them. BAMMF June 20, 2014 18 h6p://janefriedman.com/2011/09/27/what-is-a-story/
Where do Events happen? In Real World. Images, video, audio, all other sensors are just a means to detect and capture some aspect of the event. BAMMF June 20, 2014 19
Events are Connectors Events create Context Event contains Information and Experiences Spatial Causal Experiential Informational Temporal Structural
So What is Digital Storytelling? Prepare a database of all events. Collect all experienAal and informaAonal data about each event. Considering audience and the message, select important events. Render Events For each event decide what informaAon and experienAal data should be used. Render all events and related data in the best format for highest quality of experience. BAMMF June 20, 2014 21
From Events to Stories CollecAng Events SelecAng Events Rendering Events
Oral Storytelling BAMMF June 20, 2014 23
Novels BAMMF June 20, 2014 24
Video BAMMF June 20, 2014 25
Do you edit your videos? Why is video ediAng so dicult? Even a_er so many products? How can we make it usable by consumers? Is Video ediAng a real problem? Is it possible to reinvent video ediAng given all the big data? BAMMF June 20, 2014 26
Virage Video Indexing Lyn Russell 00:08:26:12 00:08:34:29 00:08:40:00 00:08:26:12 00:08:34:29 00:08:40:00 Clinton spoke with reporters When She arrived Older citizens who have Clinton running for senate Speaking at rally Improving Medicare Lyn Russell Hillary ClintonFace ID Speaker ID User Annotation Speech to Text Time Code Keyframes Video Time Hillary Clinton
Dierent Data Tracks Ac+vity Loca+on Manual Visual Audio OTHER Walk Walk Walk Walk Walk Walk Exercise Drive Home Drive Mee+ng Work Exercise Home Drive Mee+ng Work Home Mee+ng Work Exercise Event Track
Personalized Storytelling: Events in a Football Game. BAMMF June 20, 2014 29
All communicaAon is story telling. Even scienAc papers. BAMMF June 20, 2014 30
Fundamental Opportunity in MulAmedia CommunicaAon and CompuAng: UAlize complementary and correlated informaAon from mulAple sources to form holisAc picture of the world. BAMMF June 20, 2014 31
ComputaAon: at least 50,000 years CounAng was primarily used to keep track of social and economic data such as number of group members, prey animals, property, or debts. CompuAng was invented for Resource management. BAMMF June 20, 2014 32
What is Important in Big Data? MulAmedia RealAme Uncertainty BAMMF June 20, 2014 33
Big data is the Rorschach blot of our Ames an incomprehensible shape on to which we project our dreams and nightmares, hopes and fears. BAMMF June 20, 2014 34 Rorschach blot
BAMMF June 20, 2014 35 You can only control what you measure.
The most Fundamental Problem in Society: Connecting People to Resources Effectively, Efficiently, and Promptly in given Situations. 36 BAMMF June 20, 2014
Current Social Networks Important UnsaAsed Needs 37 Time to Focus BAMMF June 20, 2014
What dierenAates: Advanced SocieAes from Primi6ve SocieAes Advanced countries from a Developing countries 38 BAMMF June 20, 2014 How Peoples needs are connected to Resources.
Hunter Gatherer Food delivery where you are. 39 BAMMF June 20, 2014
UnAl 2000, you went to a phone. Now the call comes to you. 40 BAMMF June 20, 2014
UnAl recently, you were a folder. Very Soon You are Your Data. 41 BAMMF June 20, 2014
Needs Need = f (persona, situation) BAMMF June 20, 2014 42
Where do most Data Scientists apply their Magic? We live in a targeted world. AdverAsers need ecient ways to si_ through the volume of data. BAMMF June 20, 2014 43
Using BIG Data to solve Big PROBLEMS Time is right. BAMMF June 20, 2014 44
We Live in Dynamic World. Big Data will keep gegng bigger in Cyber Space. BAMMF June 20, 2014 45
Animals Machines SocieAes Published rst in 1942
Desired state (Goal) System model and Control Signal (AcAons) Current State (using mulAmedia data)
EventShop : E. Situation Detection Predic+ve Situa+on Recogni+on Evolving Global Situation Predic+ve Personal Situa+on Recogni+on Personal EventShop Evolving Personal Situation Need- Resource Matcher Recommendation Engine Persona Database Resources Needs Data Inges+on Wearable Sensors Calendar Location. DataSources . Data Inges+on and aggrega+on Database Systems Satellite Environmental Sensor Devices Social Network Internet of Things Actionable Information BAMMF June 20, 2014 48
CriAcal Requirements: Environmental SituaAon RecogniAon. Detects geo-spaAal situaAons. Space is a key dimension. Helps in idenAfying resources and connecAng to them. Personal SituaAon RecogniAon. Centered around an individual. Uses environmental situaAon. Helps in modeling individuals and idenAfying their needs. BAMMF June 20, 2014 49
Data to Situa+ons Situa+ons Event Streams Mul+modal Observa+ons Low-level Analysis Aggrega+on and Classica+on BAMMF June 20, 2014 50
A Grand Challenge for data has always been: Sense making from mulAmodal massive geo-social data-streams. 51 BAMMF June 20, 2014
A REAL Challenge SituaAon Modeling SituaAon RecogniAon (Object recogniAon, event recogniAon, speech recogniAon, dominated last century.) BAMMF June 20, 2014 52
Challenge: Unifying MulAmodal Big Data Spa+o-temporal-thema+c (STT) real-@me streams E-mage as a unifying GRID representa+on 53 (a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on Hurricanes (source: Twitter stream) d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML) BAMMF June 20, 2014
54 Billions of data sources. Environment for SelecAng, and Combining appropriate sources to detect situaAons. PredicAon for Pro-acAve acAons InteracAons with dierent types of Users BAMMF June 20, 2014
The Creative Destruction of Medicine CompuAng Power, Data Universe Social Networking Internet Mobile connecAvity, Bandwidth InformaAon Systems Imaging Wireless Sensors Genomics New Medicine CreaAve DestrucAon BAMMF June 20, 2014 58 Old Medicine
Health = Genetics + Lifestyle BAMMF June 20, 2014 59
Asthma Triggers Indoor Allergens Outdoor Allergens Smoke Air PolluAon Chemical Irritants Asthma/Allergies BAMMF June 20, 2014 60
Everyones Respiratory Health is Different Disaster Situa+on Assimila+on and Control Environmental Resources and Historic Data Governmental Agencies Internet of Things Social Sources Experts Users All Users are not EQUAL. BAMMF June 20, 2014 62
AcAviAes CommunicaAons Media Persona: Turning Disassociated Data into Meaningful InformaAon AGGREGATED PERSONAL ANALYTICS Sensors 63 Persona BAMMF June 20, 2014
Wearable 2013: Data Data Everywhere, nor any Insight to Use. h6p://www.medhelp.org/ BAMMF June 20, 2014 64
HEALTH PERSONA Logical Sensor Physiological Sensors Fitness Tracking Sensors Life Event Kine+c Event Physiological Event Food Event Personicle (Health) Persona Framework Calendar Ac+vity-level Heart rate Home Drive Mee+ng Work Exercise Walk Walk BAMMF June 20, 2014 65
Time Foursquare Personal Calendar Moves Drive Walk Mee+ng in DBH room 3233 Walk Drive Mee+ng with trainer in gym Ac+vity Level Personicle Personicle Home Drive Walk Mee+ng Work Walk Drive Exercise Work Laleh at University Club Ea+ng dinner BAMMF June 20, 2014 66
Insight: Exercise Triggers My Asthma Red: Severe asthma attack. Orange: High activity level. Intense Exercise triggers asthma! Personicle Ac+vity Stream t1 t2 t3 t4...