2003.04.14 - SLIDE 1IS246 - SPRING 2003 Lecture 21: Research Multimedia Systems IS246 Multimedia Information (FILM 240, Section 4) Prof. Marc Davis UC

2003.04.14 - SLIDE 1IS246 - SPRING 2003

Lecture 21: Research Multimedia Systems

IS246Multimedia Information

(FILM 240, Section 4)

Prof. Marc DavisUC Berkeley SIMS

Monday and Wednesday 2:00 pm – 3:30 pmSpring 2003

http://www.sims.berkeley.edu/academics/courses/is246/s03/

2003.04.14 - SLIDE 2IS246 - SPRING 2003

Today’s Agenda

• Feedback on Last Time

• Final Projects Update

• Research Multimedia Systems

– FotoFile

– Informedia

– VideoMAP and VideoSpaceIcon

– Discussion Questions

• Action Items for Next Time

2003.04.14 - SLIDE 3IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 4IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 5IS246 - SPRING 2003

Final Project Ideas

• iCoach– A hi-fi prototype of an interactive multimedia system

to assist job seekers improve their speaking skills in a job interview by streamlining the audio/video capturing, editing, audio analysis, and performance report generating processes

• Media Annotation in Mobile Phones– A detailed design specification of a system in which

the user can input metadata information about a media element she has captured using her mobile phone camera or microphone

2003.04.14 - SLIDE 6IS246 - SPRING 2003

Final Project Ideas

• Active Capture Authoring– A prototype to support the creation of automated

directing applications by developing a set of higher-level state machine primitives to facilitate the design of Active Capture interactions

• Public Domain Content– A paper reviewing technology and technology policies

that contribute to broadening the content in the public domain and facilitating access to public domain content

2003.04.14 - SLIDE 7IS246 - SPRING 2003

Final Project Ideas

• Picture Book Storytelling over Video Phone– A prototype of a consumer application which

incorporates both live video and automatic editing of the live interaction for on-demand viewing that structures the interaction around a “picture book” activity, in which an adult will read the child a story, accompanied by pictures

• Interactive Multimedia Maps – An interactive low-fi prototype of a software

application that gives users the ability to retrieve, use, produce, and distribute multimedia files using GPS mapping technology and an online multimedia database

2003.04.14 - SLIDE 8IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 9IS246 - SPRING 2003

Multimedia Research: ACM MM CFP

• The technical program will consist of plenary sessions and talks with topics of interest in three main tracks: – Content Processing

• Multimedia analysis, processing, and retrieval, including multimedia content description and indexing, multimedia semantics modeling, audio/video/multi-modal processing, multimedia digital rights management, content-based retrieval, MPEG-7/-21 standards.

– Systems• Multimedia networking and system support: including peer-to-peer

streaming, audio/video streaming, multimedia content distribution, wireless multimedia, energy-aware streaming, Internet protocols, multimedia servers, operating systems, middleware and QoS.

– Applications• Multimedia tools, end-systems, and applications: including

hypermedia systems, user interfaces, authoring, multi-modal interaction and integration, multimedia in education, entertainment, and virtual environments.

• We specifically solicit submissions in new and emerging areas in the field.

2003.04.14 - SLIDE 10IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 11IS246 - SPRING 2003

Building a Multimedia Album in FotoFile

2003.04.14 - SLIDE 12IS246 - SPRING 2003

“Scraplets” in the Story-Editing Window

2003.04.14 - SLIDE 13IS246 - SPRING 2003

Face Recognition and Annotation

2003.04.14 - SLIDE 14IS246 - SPRING 2003

Automatic Video Album Generation

2003.04.14 - SLIDE 15IS246 - SPRING 2003

Hyberbolic Tree for Content Index

2003.04.14 - SLIDE 16IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 17IS246 - SPRING 2003

Informedia Project (CMU)

• http://www.informedia.cs.cmu.edu/Informedia-I: Integrated speech, image & language understanding for creating digital video libraries

Informedia-II: Video information summarization & visualization

2003.04.14 - SLIDE 18IS246 - SPRING 2003

Content-Based Thumbnails

2003.04.14 - SLIDE 19IS246 - SPRING 2003

Filmstrip and Video Playback Windows

2003.04.14 - SLIDE 20IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 21IS246 - SPRING 2003

VideoMAP

2003.04.14 - SLIDE 22IS246 - SPRING 2003

VideoSpaceIcon

2003.04.14 - SLIDE 23IS246 - SPRING 2003

Views of VideoSpaceIcon

2003.04.14 - SLIDE 24IS246 - SPRING 2003

Construction of VideoSpaceIcon

2003.04.14 - SLIDE 25IS246 - SPRING 2003

Making a Moving VideoSpaceIcon

2003.04.14 - SLIDE 26IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 27IS246 - SPRING 2003

Discussion Questions

• Catherine Lai on “FotoFile”– FotoFile incorporates some automatic image analysis

to help with annotations, for example, face recognition. But can we and how can we take more advantages of the context to determine appropriate situation for annotation? For example, if one picture is about a karaoke contest, there is a good chance that subsequent pictures taken within a three-hour span and close to the same location will describe the same event.

– How is the use of narrative structure to help organize content so much different from annotating content manually? Does annotating contents mean to do data entry of predefined metadata like date and location as opposed to entering captions?

2003.04.14 - SLIDE 28IS246 - SPRING 2003


• David Warthen on “FotoFile”– In the short section on the “Use of Narrative

Structure” the authors assert they can use the “narrative structure… as a source of their [photos] organization and annotation”, thereby turning annotation into a “more enjoyable” activity. However, they do not really explore this further.

• How might the system exploit this approach to capturing annotation? (e.g. from an actual narrative presentation)

• How might the system facilitate one of the envisioned usages, where “the storyteller shares groupings of photos and annotations over the internet”?

2003.04.14 - SLIDE 29IS246 - SPRING 2003


• David Warthen on “FotoFile”– The authors pay a great deal of attention to

annotating personal photos, which they note requires significant effort. However, they do not attempt any cost/benefit analysis to see whether this effort pays off with reuse, or potential reuse, of the photos. If one considers the task from this point of view:

• How would one measure the ROI of investing in annotation?• Might a “80/20” rule apply where it is only beneficial to invest

in annotating the (loosely) 20% of pictures accessed 80% of the time?

• Might this imply introducing workflow annotation opportunities to the user upon first attempted re-use rather than upon entry in the system (beyond the basic “trip to Greece” or “Sam’s baby pictures” type of annotation)? Or, alternatively, would this present an insurmountable barrier to accessing the pictures for re-use at all?

2003.04.14 - SLIDE 30IS246 - SPRING 2003


• David Warthen on “FotoFile”– The authors have organized FotoFile to use scrapbook/photo

album metaphors (enhanced with keyword retrieval and browsing). These metaphors have the benefit that consumers are familiar and comfortable with them, and root the use of FotoFile in the context of existing human activities. However, such metaphors can also be limiting. Let’s explore alternatives, including:

• Why separate limit FotoFile to pictures, video, and audio, rather than all data on the computer? These might include written documents, e-mail, contact lists, etc.

• Why limit FotoFile to the personal pictures on the computer, as opposed to tying them into online resources as well (encyclopedias, picture searches and picture archives, etc.)?

• What alternatives to scrapbook narrative presentation might be compelling? Perhaps a hyperlinked approach? Or a free-form associative approach?

2003.04.14 - SLIDE 31IS246 - SPRING 2003


• Lily Chen on “VideoMAP and VideoSpaceIcon”– Since the current version of VideoSpaceIcon

cannot be simply applied when the camera operation is tracking or dollying, in which the camera position changes, I am very curious at how the "3D space model" the author defined for this approach will be constructed--how will they represent this 3D icon?

2003.04.14 - SLIDE 32IS246 - SPRING 2003


• Lisa Wang on “VideoMAP and VideoSpaceIcon”– How has spatial mapping of video advanced

in the last ten years?– Has there been further research on three

dimensional space modeling?

2003.04.14 - SLIDE 33IS246 - SPRING 2003

Today’s Agenda




– FotoFile

– Informedia




2003.04.14 - SLIDE 34IS246 - SPRING 2003

Readings for Next Time

• Wednesday 04/16 Guest Lecture on Commercial Media Asset Management Systems– Bradley Horowitz, Virage Founder and Chief Technology Officer

• Bradley Horowitz co-founded Virage in 1995 and currently serves as Chief Technology Officer, determining and managing the technical direction of the company. As director of Virage's Advanced Technology Group, he leads research efforts that result in new technologies, products and services for Virage.

• Prior to joining Virage, Bradley was a PhD. candidate at the MIT Media Lab. While at the Media Lab, he worked on a number of topics related to computer vision, computer graphics and image processing. His work there resulted in a patented new technique for the recovery of structure, motion and camera parameters from video sequences. He received an MS in Media Science from MIT in 1991 under advisor Alex P. Pentland, and a BS in Computer Science from the University of Michigan in 1989.

• Check out Virage product materials at– http://www.virage.com

Documents

2003.04.14 - SLIDE 1IS246 - SPRING 2003 Lecture 21: Research Multimedia Systems IS246 Multimedia Information (FILM 240, Section 4) Prof. Marc Davis UC