Perceptive Context for Pervasive Computing

Trevor DarrellVision Interface Group

Human-centered Interfaces

• Free users from desktop and wired interfaces• Allow natural gesture and speech commands• Give computers awareness of users• Work in open and noisy environments

- Outdoors -- PDA next to construction site!- Indoors -- crowded meeting room

• Vision’s role: provide perceptive context

Perceptive Context

• Who is there? (presence, identity)• What is going on? (activity)• Where are they? (individual location)• Which person said that? (audiovisual grouping)• What are they looking / pointing at? (pose, gaze)

Virtual views

• Frontal •Profile silhouette:Head:

Examples: VH-generated views

• Faces:

• Gait:

Range-based stereo person tracking• Range can be insensitive to fast illumination change• Compare range values to known background• Project into 2D overhead view

IntensityRange

Foreground

Plan view

• Merge data from multiple stereo cameras..• Group into trajectories…• Examine height for sitting/standing…

Audio input in noisy environments

• Acquire high-quality audio from untethered, moving speakers

• “Virtual” headset microphones for all users

Vision guided microphone arrayCameras

Microphones

Audio-visual synchrony detection

• Audio weighting from video (detected face)

+

AVMI Applications• Image localization from audio

Audio associated with left face

Audio associated with right face

image variance AVMI

Brightness and depth motion constraints

I tI t + 1

I

Z

Z tZ t + 1 yt = yt-1

Parameter space

Closed-loop 3D tracker

Track users head gaze for hands-free pointing…

Head-driven cursor

Related Projects:• Schiele• Kjeldsen• Toyama

Current application for second pointer or scrolling / focus of attention…

“Look-to-talk”

Subject not looking at SAMASR turned off

Subject looking at SAMASR turned on