Upload
earl
View
51
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Multimedia and Human Computer Interfaces. Hao Jiang Computer Science Department Boston College Nov. 20, 2007. Outline. Human Computer Interface. Multimedia interfaces and applications. Vision based interfaces Systems using vision based interfaces. Some basic methods in vision interfaces - PowerPoint PPT Presentation
Citation preview
CS335 Principles of Multimedia Systems
Multimedia and Human Computer Interfaces
Hao Jiang
Computer Science Department
Boston College
Nov. 20, 2007
CS335 Principles of Multimedia Systems
Outline
Human Computer Interface. Multimedia interfaces and applications. Vision based interfaces
– Systems using vision based interfaces.– Some basic methods in vision interfaces
• Background subtraction.
• Color blob detection and tracking.
• Face detection (Viola and Jones face detector).
CS335 Principles of Multimedia Systems
Human Computer Interface
Different meanings of HCI– The study of people, computer technology and the ways
they interactive with each other.– The design, evaluation and implementation of computer
systems for human use.– The research to study how to make computer systems more
useable.
CS335 Principles of Multimedia Systems
Aspects of HCI
Design– Intuition, experiences and design principles.
Human factors– Experiments testing the perception, memory and cognition
models of human being.
Devices– Physical devices such as mouse, keyboards and other fancy
HCI hardware.
Software and systems– Such as windows GUI, shell commands in Unix etc.
CS335 Principles of Multimedia Systems
HCI Studies Computers
– What are they good at and bad at?– Input, output, processing, networking, etc.
People– What are people good at?– What is the limitation of human abilities?– Input, output, memory, perception and cognition model.– No average answer. Must accommodate diversities.
Usability– More than the size of widgets, where are the buttons and
syntax of commands. Context
– Environment, physical constraints, cognition constraints, task, etc.
CS335 Principles of Multimedia Systems
The Quality of HCI
There are fives factors that affect the design and implementation of HCI– Time to learn.– Speed of performance.– Rate of errors.– Retention over time.– Subjective satisfaction.
CS335 Principles of Multimedia Systems
Multimedia Interfaces
Multimedia interface combines different sorts of media modalities to achieve more natural and more efficient human computer communication.– Speech, audio, touch,
video input (patterns, posture, gesture) etc.
“Put that there” is an example multimedia HCI that combines speech, gestures and graphics.
http://www.media.mit.edu/speech/sig_videos.html
Put that there, MIT 1981
CS335 Principles of Multimedia Systems
Early and Late Integration
Mode 1
Mode 2
ClassificationFeatures Output
Mode 1
Mode 2
Features
Classification
Output
Classification
Early integration
Late integration
CS335 Principles of Multimedia Systems
Advantages of Multimedia Interface
More alternative methods. The flexibility of combining different schemes. More efficient for specific tasks. Greater precision of information. Help handicapped people to use computers.
http://www.youtube.com/watch?v=d7eGypGOlOc&feature=related http://www.youtube.com/watch?v=Gnq-8iaOcXk
Pen gesture interface Virtual Reality
CS335 Principles of Multimedia Systems
Vision Based Interfaces
Vision based interfaces uses computer vision techniques to achieve more effective HCI.
Vision based interfaces are based on techniques such as:
Hand tracking,Hand gesture,Arm gestureHand Pointing
Face detectionHead trackingGazeExpressionLip reading
Posture, body gesture and action recognition
CS335 Principles of Multimedia Systems
Example Systems
MotionCapturing
CS335 Principles of Multimedia Systems
Example Systems
The MIT Kidsroom
Posture recognition
Actiondetection
CS335 Principles of Multimedia Systems
Methods in Vision Interfaces
Background subtraction.
Estimating the background
B(n) = B(n)*alpha + I(n)*(1-alpha)
Foreground Map = |Image – Background|>threshold
alpha is a positive floating point number close to 1.
CS335 Principles of Multimedia Systems
Color Blob Detection
We would like to compare the template color histogram witheach location in the image.
We first convert each image pixel by its bin number.s(x,y) = s(x,y-1) + i(x,y)h(x,y) = h(x-1,y)+ s(x,y)
(x,y)
h(x,y) is the color histogram
a b
c d
CS335 Principles of Multimedia Systems
Blob Tracking (Filtering Scheme)
Search window
t1t2 t3
t4
: Predicted location. : detected object.
: Final estimation.p
x
y
x = p * alpha + y * (1-alpha)
CS335 Principles of Multimedia Systems
Blob Tracking (DP Approach)
t2 t3 t4
AB
BC
BD
t1
AB
C
D E
F
CF
CE
DF
DE
t
cost(F) + alpha ||2C-B-F||
const
CS335 Principles of Multimedia Systems
Mean Shift Tracking
Instead of exhaustive search in a window, if object moves slowly, we can use a more efficient location updating scheme – Mean shift.
In the following equation, x(i) are points in the search window and y is the location of object, w(i) are weights.
CS335 Principles of Multimedia Systems
Mean Shift Tracking Result
http://www.caip.rutgers.edu/~comanici/Papers/KernelTracking.pdf
CS335 Principles of Multimedia Systems
Face Detection
Viola and Jones face detector
These features can be computed using integral images similarly
to the color histograms discussed before.
CS335 Principles of Multimedia Systems
Face Detection
It uses boosting scheme to automatically choose the optimal weight for the classifier based on each kind of feature.
To further improve the efficiency, a cascaded classifier is used.
CS335 Principles of Multimedia Systems
Summary
We only touch the very surface of multimedia interfaces.
Currently, the dominant HCI is still GUI with keyboard and mouse.
Multimedia interfaces are beginning to attract more and more interests.
We need more robust and efficient media recognition schemes to support effective multimedia interfaces.