Social Signal Processing: UnderstandingNonverbal Communication in Social Interactions
Alessandro Vinciarelli1,2 and Fabio Valente21University of Glasgow - Sir A.Williams Bldg, G12 8QQ (UK)2IDIAP Research Institute - CP592 Martigny (Switzerland)
e-mail: [email protected], [email protected]
Outline
• Part I - What is SSP?
• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior
• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action
• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action• Analyzing conversations
• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives
• The SSPNet• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives• The SSPNet
• Challenges ahead
Slide 2 of 25
Outline
• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior
• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts
• Part III - Future Perspectives• The SSPNet• Challenges ahead
Slide 2 of 25
Part I - What is SSP?
Slide 3 of 25
Social Signals and Social Behaviour
Our attention focuses on words, but we are immersed in a richnon-verbal world influencing not only the meaning of words, butalso our perception of the social context.
Slide 4 of 25
Social Signals and Social Behaviour
forwardposture
forwardposture
vocalbehaviour
mutualgaze
interpersonaldistance
NonverbalBehavioural
Cues
height
gesture
Our attention focuses on words, but we are immersed in a richnon-verbal world influencing not only the meaning of words, butalso our perception of the social context.
Slide 5 of 25
Social Signals and Social Behaviour
Social Signalforwardposture
forwardposture
vocalbehaviour
mutualgaze
interpersonaldistance
NonverbalBehavioural
Cues
height
gesture
Our attention focuses on words, but we are immersed in a richnon-verbal world influencing not only the meaning of words, butalso our perception of the social context.
Slide 6 of 25
Nonverbal Communication
Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.
Slide 7 of 25
Nonverbal Communication
clothes, attractivenesssomatotype, etc.
self touching
facial expression
prosody, pitch,
postural congruence, etc.
gaze behaviour, etc.
rythm, etc.
distamce, seating
Behaviouralcues
Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.
Slide 8 of 25
Nonverbal Communication
PhysicalAppearance
GesturesPostures
Face and EyesBehaviour
VocalBehaviour
SpaceEnvironment
clothes, attractivenesssomatotype, etc.
self touching
facial expression
prosody, pitch,
postural congruence, etc.
gaze behaviour, etc.
rythm, etc.
distamce, seating
CodesBehavioural
cues
Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.
Slide 9 of 25
Nonverbal Communication
PhysicalAppearance
GesturesPostures
Face and EyesBehaviour
VocalBehaviour
SpaceEnvironment
clothes, attractivenesssomatotype, etc.
self touching
facial expression
prosody, pitch,
postural congruence, etc.
gaze behaviour, etc.
rythm, etc.
distamce, seating
forming impressions
deceiving anddetecting deception
sending messages ofpower and persasion
managing interaction
expressing emotion
Behaviouralcues Functions
Codes
sending relationalmessages
Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.
Slide 10 of 25
Social Signal Processing
DataCapture
PersonDetection
MultimodalBehavioural
Streams
CuesBehavioural
ExtractionSocial SignalsUnderstanding
ContextUnderstanding
BehaviouralCues
SocialBehaviours
Raw Data
Preprocessing
MultimodalBehavioural
Streams
Social Interaction Analysis
- A.Pentland, “Social Signal Processing”, IEEE Signal Processing Magazine,
24(4):108-111, 2007.
- A.Vinciarelli, M.Pantic, H.Bourlard, “Social Signal Processing: Survey of an
Emerging Domain”, Image and Vision Computing, 27(12):1743-1759, 2009.
Slide 11 of 25
Part II - SSP in Action
Slide 12 of 25
A Conversation Analysis Framework
[...] the most widely used analytic approach is based onan analogy with the workings of the market economy. Inthis market there is a scarce commodity called the floorwhich can be defined as the right to speak. Havingcontrol of this scarce commodity is called a turn. In anysituation where control is not fixed in advance, anyonecan attempt to get control. This is called turn-taking.
G.Yule,“Pragmatics”, Oxford University Press (1996)
Slide 13 of 25
Turn-Taking
t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1
The turn taking pattern can be represented with a sequenceS = {(s1,∆t1, ), . . . , (sT ,∆tT )} where si ∈ A = {a1, . . . , aG} is aperson label, and ∆ti the length of the i th turn.
• Among the most robustly detectable behavior evidences,
• but how far can we go by just modeling who talks when andhow much?
Slide 14 of 25
Role Recognition in Broadcast Material
• People play functional roles (Anchorman, Guest, etc.)
• Adjacent speakers are supposed to interact
• Around 85% of data time correctly labeled in terms of role
A.Vinciarelli, IEEE T-Multimedia, 9(6):1215-1226 (2007)
H.Salamin et al., IEEE T-Multimedia, 11(7):1373-1380, 2009
Slide 15 of 25
Role Recognition in Meetings
x1 = (1,1,1,1) x2 = (0,0,1,1) x3 = (1,1,1,0)
w1 w2 w3 w4
a1 2a a3
t1 t 2 t3 t4 t 5 t 6 t7
w1 w2 w3 w4
ts =a2 3
t
s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1
actors
events
• Social networks suitable for functional roles, lexical analysissuitbale for semantic ones
• Affiliation networks are suitable for small groups
• Around 75% of data time correctly labeled in terms of role
N.Garg et al., Proc. of ACM-Multimedia, pp. 693-696 (2008)
Slide 16 of 25
Roles and Prosody
t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1
R∗ = arg maxR∈R
p(R|X , ~α)
• Conditional Random Fields allow the combination ofturn-taking and prosody
• Entropy of main prosodic features
• Results up to 89%, but combination does not always lead tosignificant improvements
H.Salamin et al., Proc. of ACM-Multimedia, to appear (2010)
Slide 17 of 25
Story Segmentation
t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1
Story 1 Story 2 Story 3
The problem consists of finding the following story sequence H:
H = arg maxH∈H
p(X |H)p(H) (1)
• The purity is around 0.75
• Longer stories are better recognized
A.Vinciarelli et al., Proc. of ACM-Multimedia, pp. 261-264 (2007)
Slide 18 of 25
Conflict Analysis
t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1
Q = arg maxQ∈Q
πq1
N∏n=2
p(qn|qn−1) (2)
where qi is one of the two groups or the moderator.
• People tend to react to someone they disagree with
• Groups correctly reconstructued in 66% of the cases (randomgrouping has an average 6.5% performance).
A.Vinciarelli, IEEE Signal Processing Magazine, 26(5):133-136, 2009
Slide 19 of 25
Part III - Challenges Ahead
Slide 20 of 25
Open Issues and Challenges
• Getting psychology and engineering closer
• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary
• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches
• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous
• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data
• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data• Artificial settings are sometimes too simple
• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications
• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications• Applications link research to reality
• Applications provide realistic benchmarks
Slide 21 of 25
Open Issues and Challenges
• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines
• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity
• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data
• Identifying relevant applications• Applications link research to reality• Applications provide realistic benchmarks
Slide 21 of 25
The SSPNet
Universita’ di Roma Tre
Queen’s University BelfastUnversity of Edinburgh
Imperial College London
University of TwenteDelft University of Technology
DFKI
Idiap research instituteUniversity of Geneva
CNRS
Slide 22 of 25
Human-Human and Human-Machine Interaction
SSP in Human Machine Interaction
Behavior AnalysisBehavior Modeling
DFKI, CNRS, U. of Twente
Delft, U. of EdinburghIdiap, Imperial College, TUQueen�’s U. Belfast, U. of Roma
Tre, U. of Geneva
Behavior Synthesis
SSP in Human Human Interaction
2 Research Foci: Human-Human and Human-Computer Interaction3 Scientific Domains: Behavior Modeling, Analysis and Synthesis5 Years to go: From 2009 to 2014
Slide 23 of 25
SSPNet: The Portal
The most tangible aspect of the SSPNet will be the web portal:
http://www.sspnet.eu
• lowering the entry barrier of SSP, i.e. to reduce significantlythe effort required to start research in the domain,
• providing common benchmarks for rigorous performanceassessment and comparison between different approaches,
• disseminating literature, data and tools relevant to SSP.
Our ambition is to make of the portal THE reference for SSP.
Slide 24 of 25
Thank You!
Slide 25 of 25