Roman10 mazzei

The FACE of AutismDaniele Mazzei University of Pisa

IEEE Ro-Man 2010 Viareggio 12th-15th sept 2010

Mindblindness• People with autism are known to

have great difficulties in inferring and analysing mental states, both their own and of others.

• S. Baron-Cohen has defined mindblindness as the inability to develop an awareness of what is in the mind of another human

ASD and Robotics• In recent years, robots have been

proposed for the treatment and rehabilitation of people with cognitive disorders.

• Human-like robots which embody emotional states expression, empathy and non-verbal communication have also been proposed for autism therapy

Our HypothesisTherapy in which a robot is able to adapt its behaviour in response to a patient’s postulated mental states can help ASD subjects to improve their social skills.We call this adaptive therapy.

FACET platform

Face Control Unit(Bodymap and brain)

Face tracking and facial

expression recognition

Gaze tracking

hat

Sensorized

T-Shirt

Inertial platform

Room motorized cameras Environmental

microphone

Face configurator and control

interface

FACE android

Phase 1 Phase 2 Phase 3

Limited expressivityAesthetically not appreciated

1 ASD and 1 control subject tested

Well accepted by ASD but not comfortable for the control subject

Improved aesthetically and in expressivity

4 ASD and 3 control subjects tested

Better accepted by ASD and control subject

Improved empathic link

Current phase

FACE (Facial Automaton for Conveying Emotions)

• Pioggia G, et al. 2005, IEEE Transactions

• Pioggia G. et al. 2008, Cybertherapy and Rehabilitation

FACE3

Sensorized Shirt• Based on e-textiles• Gathers, computes and

transmits HR, HRV, RR, skin conductance, skin temperature and respiratory rate

• All this signals can be correlated with subject emotional states

Made by Smartex Srl, Prato, Italy

HATCAM Eye Tracker• Wearable and specifically

designed to investigate early attention disorders in infants

• Hat or Band mounted system • fast and easy calibration

procedure • Mirror allows to use images of

both eyes (3D gaze point identification)

• A.Armato, et al. GNB 2010• Patent: TO2008A000862 2008

Robotics4.NetThe framework proposes a software architecture inspired to the human nervous system architectureo The Brain: The core of control system

o The Bodymap: A sort of black board used to send and receive messages

o The Roblets: The appendix of the system, as the parts of our nervous system, read data from the environment and convert the Brain signal into an action

EACH ACQUISITION OR ACTUATION SYSTEM IS AN INDEPENDENT ROBLET

The Joker Effect

• The Joker effect consists in the generation of unsuitable expressions for the therapeutic context and could induce fear or discomfort in the subject.

• It is a emphatic misalignment typical of sociopaths

We need to modulate in intensity the expressions

Adaptive Control Paradigm

FACET Signal

processing algorithm

+1

0

-1

Base line signals

FACE operator GUI

FACE terapist GUI

Happiness +3Surprise +2Sadness +1Neutral 0Anger -1Disgust -2Fear -3

FACE Android

FACE mood state

Expressions selection and intensity calculation algorithm

0 +1

Happy +3Surprised +2Sad +1Neutral 0Angry -1Disgusted -2Fearful -3

Subject physiological signals

Adaptive Control Paradigm

FACET Signal

processing algorithm

+1

0

-1

Base line signals

FACE operator GUI

FACE terapist GUI

Happiness +3Surprise +2Sadness +1Neutral 0Anger -1Disgust -2Fear -3

FACE Android

FACE mood state

Expressions selection and intensity calculation algorithm

0 +1

Happy +3Surprised +2Sad +1Neutral 0Angry -1Disgusted -2Fearful -3

Subject physiological signals

Adaptive Control Paradigm

FACET Signal

processing algorithm

+1

0

-1

Base line signals

FACE operator GUI

FACE terapist GUI

Happiness +3Surprise +2Sadness +1Neutral 0Anger -1Disgust -2Fear -3

FACE Android

FACE mood state

Expressions selection and intensity calculation algorithm

0 +1

Happy +3Surprised +2Sad +1Neutral 0Angry -1Disgusted -2Fearful -3

Subject physiological signals

Adaptive Control Paradigm

FACET Signal

processing algorithm

+1

0

-1

Base line signals

FACE operator GUI

FACE terapist GUI

Happiness +3Surprise +2Sadness +1Neutral 0Anger -1Disgust -2Fear -3

FACE Android

FACE mood state

Expressions selection and intensity calculation algorithm

0 +1

Happy +3Surprised +2Sad +1Neutral 0Angry -1Disgusted -2Fearful -3

Subject physiological signals

Expressions Creation

• 3D simulator of FACE allows to study and design facial expressions

• The final expression intensity is modulated merging the motor position of the selected expression with that of the neutral expression

Preliminary results• At the moment the therapy/paradigm

was tested with 10 ASD subjects and 5 controls

• No subjects showed discomfort or fear during the therapy

• Preliminary signal analysis showed a correlation between heart rate variability and intensity of interaction

Conclusions and Future Works• FACET system could have a wide

range of applications in the study of non-verbal communication and expression.

• On going experiments on control and ASD subjects

• More tests of the adaptive control algorithm and of the FACET hardware/software infrastructure

• Using of the FACET platform to perform generic studies on the human-robot empathic link

Thanks For Your Attention

IEEE Ro-Man 2010

Questions?