Pervasive PixelsPervasive Pixels(Columbia University(Columbia UniversityDept. of Computer Science)Dept. of Computer Science)

Henning Schulzrinne (PI)Steven K. Feiner

Gail KaiserJohn Kender

Kathleen McKeown

Proposed ResearchProposed Research

Goal: seamless mobile multimedia collaboration across distance

Integrate advances across fields Collaborative work Graphical and visual interfaces Spoken language understanding and

generation Vision sensing and understanding Networking and security

ContributionsContributions

Contextual information management use workflow to determine display content multimedia summaries of past and present sessions

Harmonizing physical and virtual environments map changing virtual information onto physical displays map layout of physical environment onto virtual space

Network services clear, flexible interface to common services authentication and privacy support infrastructure for persistent large displays

Features of Research Features of Research InfrastructureInfrastructure Large numbers of instrumented

multi-display workspaces Networked mobile devices of

various capabilities Transparent and automatic

adaptability to changes of place, platform or group

Support for a wide range of hardware and software, from commercial to novel

Proposed Research Proposed Research InfrastructureInfrastructure

Outfit informal areas for collaboration Public areas for walk-by interaction Multiple touch displays, cameras, audio Portable units

Stationary setups Multiple displays, video cameras, audio Seminar room, meeting rooms 12 faculty offices

User-based personalization: user location Triangulation on mobile devices Visual tracking Standard methods (e.g., active badge)

Public areas – walk by Public areas – walk by stationsstations

Multiple touch displays, video projectors and cameras, embedded computers, speakers and microphones

IR/RF badge

network PC

proj.

camera

card reader

loudspeaker

ceiling

electronicwhiteboard

microphonearray

Design for Walk-by Collaboration Station

Public Areas – informal Public Areas – informal gatheringsgatherings

Meeting RoomMeeting Room

Remote-controlled pan-tilt video cameras and projectors, Omnicam, conference table microphones, automatic audio mixer, ceiling speakers

Faculty OfficeFaculty Office

Mimio electronic whiteboard, XGA video projector, Ethernet speaker phone, wall-mounted pan-tilt video camera, PocketPCs

Seminar roomSeminar room

Omnicam omnidrectional audience camera, high-resolution DV video camera, 2 pan-tilt speaker cameras, ceiling mounted microphones, electronic whiteboard, XGA high-brightness video projectors

FunctionalitiesFunctionalities Conferencing

Internet conferencing server to mix IP and PSTN audio streams

Interconnection with analog phone Digital hybrid connects digital or analog sound to

existing telephone system in classroom Network voice-over-IP interface attached to Nortel

Meridian PBX for 20 simultaneous conversations Multi-processor servers and IA64 compute and

database server File storage Face, speaker and fingerprint recognition Backup facilities: 2 printers and tape library

system

Initial resultsInitial results HCI: gesture-based user interface for public kiosk

mouse replacement for pointing and selecting uses frontal and side camera

Security: disCFS and WebDAVA secure file systems disCFS: NFS with credentials instead of authorization WebDAVA: grant restricted access to resources using

HTTP and Java applets Web-based collaboration:

content on all kinds of devices pass DOM through a series of filters and transformations

HTML Ubiquitous multimedia communications

infrastructure being commercialized; I2 demonstration input into standardization (IETF)

Ubiquitous ComputingUbiquitous Computing Traditionally, focus on closed environments

proprietary protocols single (trusted) user class single site (room, lab, home, …) stand-alone components (“video conferencing”)

PP focuses on whole system and user experience Pervasive Pixels networking component:

standard protocols: SIP for media configuration, event notification, instant

multimedia messaging SLP for service discovery

integration of presence and user context standardization in the IETF (RPID)

location-based services user context user authorization service location

Mobility in Pervasive PixelsMobility in Pervasive Pixels Terminal mobility

application-layer mobility complements L3 mobility

Session mobility move active sessions to

devices found in the environment

service discovery Service mobility

move configuration to new devices

Personal mobility one user, many devices

Location-based servicesLocation-based services Traditionally, focus on

geospatial location (e.g., GPS) But other aspects as

important: civil location (often more

intuitive) type of place (home vs.

office; outdoors vs. theatre) behavioral: distraction,

privacy, appropriateness Experimenting with low-

complexity location mechanisms:

IR/RF active badges with low installation cost (Ivistar)

BlueTooth location beacons LAN backtracking and DHCP swipe cards and i-buttons

DHCPserver

458/17 Rm. 815458/18 Rm. 816

DHCP answer:sta=DC loc=Rm815lat=38.89868 long=77.03723

8:0:20:ab:d5:d

CDP + SNMP8:0:20:ab:d5:d 458/17

Some initial lessons Some initial lessons learnedlearned Usage: remote presence from UKy during sabbatical

research group meetings departmental site visit thesis proposals and defenses

Perception: “Multimedia collaboration is a mature field” Reality: It doesn’t work much better than in 1992

still fails in hard-to-diagnose ways quality better, but echo, feedback and level issues remain

Integration between synchronous and asynchronous collaboration

integrating documents, minutes, … Transition from call-focused to presence-focused

much larger use of asynchronous collaboration (email, bulletin boards, …)

Working with start-up company: new IP-based departmental communication system to replace PBX

Columbia SIP servers Columbia SIP servers (CINEMA)(CINEMA)

InternalTelephoneExtn: 7040

SIP/PSTN Gateway

Department PBX

Web based configuration

Web server

Telephoneswitch

SQLdatabase

sipd:Proxy, redirect, registrar server

Extn: 7134

xiaotaow@cs NetMeeting

H.323

rtspd: media server

sipum: Unified messaging

Quicktime

RTSP clients

RTSP

Extn: 7136

713x

Single machine

SNMP(Network Management)

sipconf: Conference server

siph323: SIP-H.323 translator

Local/long distance1-212-5551212

Larger lessons for multimedia Larger lessons for multimedia systems researchsystems research Software tool support for multimedia

communications lacking most are applications, not building blocks cross-platform research media tools are getting very old

and creaky (vic, rat, etc.) multi-party support very weak (multicast never

happened) Components designed to be operated by humans

IP phones only have HTTP/HTML interface video projectors just proprietary configuration

Lots of components, but hard to evaluate in real use

still mostly barely demo quality: audio delay, echo, random failures

people will fall back to good ol’ PSTN quickly

ConclusionConclusion

Pervasive Pixels = attempt to integrate multiple modalities into system, not just grouping of components

Evaluation in real usage, not just demos

Spread throughout the department, not just lab