Attention detection in driver simulator

project #4

INPG : L. Bonnaud, A. CaplierUCL : D. trevisan, B. Macq

Participants : A. Benoit, G. Chanel, P. Ngo,V. Levacic, C. Thillou

Guest star : L. Lawson

04/18/23 2

Objectives

• Multimodal detection of user state– Hypo vigilance– Stress level– In the context of driving simulator

• Alarm => multimodal feedback to the user

• Test of OpenInterfaceas an integration platform

04/18/23 3

Signals used foruser state assessment

Video(camera)

Physiological (biopac)

Head Eyes Mouth ECG GSR

Hypovigilance ? ?

Stress ?

04/18/23 4

General architecture

Significative rotation ?

High frequency ?

Both eyes closed ?

Video

Mouth opening detector

ECG analysis

Head rotation estimation

Eyes blinks detector

GSR analysis

GSRECG

Yawn ?

Fusion (redundancy, equivalence)

High heart rate variation ?

High GSR ?

Local Peak ?

Fission (redundancy, equivalence)

Sonic feedback

Graphical feedback

Haptic feedback

Screen loudspeaker Vibrating wheel

Dialog controller

04/18/23 5

Face motions analysis

04/18/23 6

Physiological signals “database”

• 3 subjects : thanks to participants!• 3 situations :

– rest / relaxation– Stress stimuli while reading a newspaper

• Hand clapping• Light in eyes• Answering simple question• Telephone call, “your dead” (killer game)

– Playing with the driving simulator (difficult tracks)

04/18/23 7

GSR signal

04/18/23 8

GSR signal

04/18/23 9

GSR signal

04/18/23 10

ECG signal : preprocessing

04/18/23 11

ECG : stress assessment

04/18/23 12

Simplified architectureof 1st prototype

Significative rotation ?

High frequency ?

Both eyes closed ?

Video

Mouth opening detector

ECG analysis

Head rotation estimation

Eyes blinks detector

GSR analysis

GSRECG

Yawn ?

Fusion (redundancy, equivalence)

High heart rate variation ?

High GSR ?

Local Peak ?

Fission (redundancy, equivalence)

Sonic feedback

Graphical feedback

Haptic feedback

Screen loudspeaker Vibrating wheel

Dialog controller

Network messages (XML)

1st PC, Matlab, Linux

2nd PC, Simulator, Windows

04/18/23 13

Realization of 1st prototype

Significative rotation ?

High frequency ?

Both eyes closed ?

Video

Mouth opening detector

ECG analysis

Head rotation estimation

Eyes blinks detector

GSR analysis

GSRECG

Yawn ?

Fusion (redundancy, equivalence)

High heart rate variation ?

High GSR ?

Local Peak ?

Fission (redundancy, equivalence)

Sonic feedback

Graphical feedback

Haptic feedback

Screen loudspeaker Vibrating wheel

Dialog controller

Network messages (text)

04/18/23 14

Conclusion

• First working prototype (demo available)– Firewire camera Matlab interface under Linux– Modification of the simulator (multi threading, TCP server,

message display)– TCP client in Matlab

• First try with Open Interface integration problem with Matlab integration to be solved by Lionel next week

• Usefulness of physiological signals• Reliable online face motions analysis

Still a lot of work to do !!!