Pervasive PixelsPervasive Pixels(Columbia University(Columbia UniversityDept. of Computer Science)Dept. of Computer Science)
Henning Schulzrinne (PI)Steven K. Feiner
Gail KaiserJohn Kender
Kathleen McKeown
Proposed ResearchProposed Research
Goal: seamless mobile multimedia collaboration across distance
Integrate advances across fields Collaborative work Graphical and visual interfaces Spoken language understanding and
generation Vision sensing and understanding Networking and security
ContributionsContributions
Contextual information management use workflow to determine display content multimedia summaries of past and present sessions
Harmonizing physical and virtual environments map changing virtual information onto physical displays map layout of physical environment onto virtual space
Network services clear, flexible interface to common services authentication and privacy support infrastructure for persistent large displays
Features of Research Features of Research InfrastructureInfrastructure Large numbers of instrumented
multi-display workspaces Networked mobile devices of
various capabilities Transparent and automatic
adaptability to changes of place, platform or group
Support for a wide range of hardware and software, from commercial to novel
Proposed Research Proposed Research InfrastructureInfrastructure
Outfit informal areas for collaboration Public areas for walk-by interaction Multiple touch displays, cameras, audio Portable units
Stationary setups Multiple displays, video cameras, audio Seminar room, meeting rooms 12 faculty offices
User-based personalization: user location Triangulation on mobile devices Visual tracking Standard methods (e.g., active badge)
Public areas – walk by Public areas – walk by stationsstations
Multiple touch displays, video projectors and cameras, embedded computers, speakers and microphones
IR/RF badge
network PC
proj.
camera
card reader
loudspeaker
ceiling
electronicwhiteboard
microphonearray
Design for Walk-by Collaboration Station
Meeting RoomMeeting Room
Remote-controlled pan-tilt video cameras and projectors, Omnicam, conference table microphones, automatic audio mixer, ceiling speakers
Faculty OfficeFaculty Office
Mimio electronic whiteboard, XGA video projector, Ethernet speaker phone, wall-mounted pan-tilt video camera, PocketPCs
Seminar roomSeminar room
Omnicam omnidrectional audience camera, high-resolution DV video camera, 2 pan-tilt speaker cameras, ceiling mounted microphones, electronic whiteboard, XGA high-brightness video projectors
FunctionalitiesFunctionalities Conferencing
Internet conferencing server to mix IP and PSTN audio streams
Interconnection with analog phone Digital hybrid connects digital or analog sound to
existing telephone system in classroom Network voice-over-IP interface attached to Nortel
Meridian PBX for 20 simultaneous conversations Multi-processor servers and IA64 compute and
database server File storage Face, speaker and fingerprint recognition Backup facilities: 2 printers and tape library
system
Initial resultsInitial results HCI: gesture-based user interface for public kiosk
mouse replacement for pointing and selecting uses frontal and side camera
Security: disCFS and WebDAVA secure file systems disCFS: NFS with credentials instead of authorization WebDAVA: grant restricted access to resources using
HTTP and Java applets Web-based collaboration:
content on all kinds of devices pass DOM through a series of filters and transformations
HTML Ubiquitous multimedia communications
infrastructure being commercialized; I2 demonstration input into standardization (IETF)
Ubiquitous ComputingUbiquitous Computing Traditionally, focus on closed environments
proprietary protocols single (trusted) user class single site (room, lab, home, …) stand-alone components (“video conferencing”)
PP focuses on whole system and user experience Pervasive Pixels networking component:
standard protocols: SIP for media configuration, event notification, instant
multimedia messaging SLP for service discovery
integration of presence and user context standardization in the IETF (RPID)
location-based services user context user authorization service location
Mobility in Pervasive PixelsMobility in Pervasive Pixels Terminal mobility
application-layer mobility complements L3 mobility
Session mobility move active sessions to
devices found in the environment
service discovery Service mobility
move configuration to new devices
Personal mobility one user, many devices
Location-based servicesLocation-based services Traditionally, focus on
geospatial location (e.g., GPS) But other aspects as
important: civil location (often more
intuitive) type of place (home vs.
office; outdoors vs. theatre) behavioral: distraction,
privacy, appropriateness Experimenting with low-
complexity location mechanisms:
IR/RF active badges with low installation cost (Ivistar)
BlueTooth location beacons LAN backtracking and DHCP swipe cards and i-buttons
DHCPserver
458/17 Rm. 815458/18 Rm. 816
DHCP answer:sta=DC loc=Rm815lat=38.89868 long=77.03723
8:0:20:ab:d5:d
CDP + SNMP8:0:20:ab:d5:d 458/17
Some initial lessons Some initial lessons learnedlearned Usage: remote presence from UKy during sabbatical
research group meetings departmental site visit thesis proposals and defenses
Perception: “Multimedia collaboration is a mature field” Reality: It doesn’t work much better than in 1992
still fails in hard-to-diagnose ways quality better, but echo, feedback and level issues remain
Integration between synchronous and asynchronous collaboration
integrating documents, minutes, … Transition from call-focused to presence-focused
much larger use of asynchronous collaboration (email, bulletin boards, …)
Working with start-up company: new IP-based departmental communication system to replace PBX
Columbia SIP servers Columbia SIP servers (CINEMA)(CINEMA)
InternalTelephoneExtn: 7040
SIP/PSTN Gateway
Department PBX
Web based configuration
Web server
Telephoneswitch
SQLdatabase
sipd:Proxy, redirect, registrar server
Extn: 7134
xiaotaow@cs NetMeeting
H.323
rtspd: media server
sipum: Unified messaging
Quicktime
RTSP clients
RTSP
Extn: 7136
713x
Single machine
SNMP(Network Management)
sipconf: Conference server
siph323: SIP-H.323 translator
Local/long distance1-212-5551212
Larger lessons for multimedia Larger lessons for multimedia systems researchsystems research Software tool support for multimedia
communications lacking most are applications, not building blocks cross-platform research media tools are getting very old
and creaky (vic, rat, etc.) multi-party support very weak (multicast never
happened) Components designed to be operated by humans
IP phones only have HTTP/HTML interface video projectors just proprietary configuration
Lots of components, but hard to evaluate in real use
still mostly barely demo quality: audio delay, echo, random failures
people will fall back to good ol’ PSTN quickly