Upload
emmeline-shaw
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Center for Intelligent Systems Research
GW Transportation Research InstituteThe George Washington University,
Virginia Campus, 20101 Academic Way, Ashburn, VA 20147
NDIA 3rd Annual Intelligent Vehicle Systems Symposium
Driving Simulator Experiment:Detecting Driver Fatigue by Monitoring Eye
and Steering Activity
Dr. Azim Eskandarian, Riaz Sayed (GWU)
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Research Objective
Conduct Simulator Experiment and Analyze the Data, to search for a system for automatic detection of drowsiness based on driver’s performance
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Significance of the Problem• Drowsiness/Fatigue Related Accident Data:
• NHTSA Estimates 100,000 drowsiness/fatigue related Crashes Annually
• FARS indicates an annual average of 1,544 fatalities
• Fatigue has been estimated to be involved in 10-40% of crashes on highways (rural Interstate)
• 15% of single vehicle fatal truck crashes
• Fatigue is the most frequent contributor to crashes in which a truck driver was fatally injured
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
• A drowsy/sleepy driver is unable to determine when he/she will have an uncontrolled sleep onset
• Fall asleep crashes are very serious in terms of injury severity
• An accident involving driver drowsiness has a high fatality rate because the perception, recognition, and vehicle control abilities reduces sharply while falling asleep
• Driver drowsiness detection technologies can reduce the risk of a catastrophic accident by warning the driver of his/her drowsiness
Significance of the Problem
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Driver Drowsiness Detection Techniques 1. Sensing of driver physical and physiological phenomenon
– Analyzing changes in brain wave or EEG
– Analyzing changes in eye activity and Facial expressions
• Good detection accuracy is achieved by these techniques
• Disadvantages:
– Electrodes have to be attached to the body of the driver for sensing the signals
– Non-contact type sensing is also highly dependant on environmental conditions
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
2. Analyzing changes in performance output of the vehicle hardware
– Steering, speed, acceleration, lateral position, and braking etc.
• Advantages:
– No wires, cameras, monitors or other devices are to be attached or aimed at the driver
– Due to the non-obtrusive nature of these methods they are more practically applicable
Driver Drowsiness Detection Techniques
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Approach for Drowsiness Detection and Driver Warning
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Experiment• Conducted in the Vehicle Simulator Lab of the CISR.
GWU VA Campus, Ashburn VA.
• Twelve subjects between the ages of 23 and 43
• Test Scenario consisted of a continuous rural Interstate highway, with traffic in both directions Speed limit of 55 mph.
• Morning session 8 – 10 am
• Night session 1 – 3 am
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
CISR Driving Simulator
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Eye Tracking Equipment
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Sample Data From Simulator
RUN# ZONETIME SPEEDLIM CRASHB CRASHV LANEX BRAKEFOR BRAKETAP
1 0 35 0 0 0 0 0
1 2.1 35 0 0 0 0 0
1 4.2 35 0 0 0 0 0
1 6.2 35 0 0 0 0 0
1 8.3 35 0 0 0 0 0
STEERPOS STEERVAR LATPLACE LATPLVAR SPEED SPEEDVAR SPEEDDEV
-0.1 0 -0.09 0 53.71 0 -4.65
0.2 0 -0.22 0 53.71 0 -4.65
0.4 0 -0.31 0 53.71 0 -4.65
0 0 -0.35 0 53.71 0 -4.65
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Lateral Position of VehicleD a y 1
- 3
- 2
- 1
01
2
3
T i m e
La
tera
l P
os
itio
n
D a y 4
- 3
- 2
- 10
12
3
T i m e
La
tera
l P
os
itio
n
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Power Spectrum Density for Vehicle Lateral Position
0
0.1
0.2
0.3
0.4
0.5
0.6
0 500 100 0 1500 2000
TIME
PS
D
DAY-1
DAY-2
DAY-3
DAY-4
ak
2
k1
n
T
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Steering Anglefilter correction for curves
- 10
- 5
0
5
10
15
20
Time
Ste
eri
ng A
ngle
Curvatu reNo Curva ture
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Hypothesis• The hypothesized relationship between
driver state of alertness and steering wheel
position is that under an alert state, drivers
make small amplitude movements of the
steering wheel, corresponding to small
adjustments in vehicle trajectory, but under
a drowsy state, these movements become
less precise and larger in amplitude
resulting in sharp changes in trajectory
(Planque et al. 1991).
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
A Hybrid Artificial Neural Network Architecture
Wj1
Unsupervised Layer : Clustering Competitive Algorithm
Supervised Layer: ClassificationFeedforward Algorithm
28 X 8
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Hybrid Artificial Neural Network Architecture
Unsupervised Supervised
Adaptive Network
W
Input Output
Desired Output Error
Adaptive Network
W
Input Output
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Training for Unsupervised Competitive Layer
1. Initialize the weight vector randomly for each neuron. 2. Present the input vector X(n) .3. Compute the winning neuron using the Euclidean distance
as a metric.
Where Wi = [w1, w2, …. w8]T is the weight vector of
neuron i.
bi is the bias to stop the formation of dead neurons.
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Training Competitive Layer Continued
• N number of time a neuron wins in competitive layer
and are learning constants and o(n) is the outcome of the present competition (=1 if neuron wins & else = 0).
• Ci initially set to small random value
4. Update the weight vector of the winning neuron Wi* only.
5. Continue with step (2) two until change in the weight vectors reaches a minimum value.
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Training Competitive Layer Continued
• The competitive algorithm moves the weight vectors of all the neurons closer to the center of the clusters.
• Each neuron (or set of neurons) of the competitive layer represents a cluster.
• The Output of the neuron is 1 if it wins the competition and 0 if it losses.
• The Output of the Competitive layer is an
n-dimensional binary vector T(n) = [t1, t2, …….., tn]T .
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Training for supervised feed forward layer
• Step 1: Initialize the synaptic weights and the thresholds to small random numbers.
• Step 2: Present the network with an epoch of training exemplars
• Step 3: Apply Input vector X(n) to the input layer and the desired response d(n) to the output layer of neurons. The output of each neuron is calculated as
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Training Continued
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Training Continued
• N = No. of training sets in one epoch = Learning rate parameter = Momentum constant
• Step 5: Iterate the computation by presenting new epochs of training examples until the mean square error (MSE) computed over entire epoch achieve a minimum value. MSE is given by:
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Training Parameters
• Hybrid architecture using an unsupervised clustering algorithm and a classifier (Back propagation learning algorithm in batch mode)
• Tanhyperbolic activation function, with output range from –1 to 1
• Variable learning rate and momentum were used
• Cross validation during training
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Input Discretization of Steering Angle
Steering. Angle(deg)
r1 r2 r3 r4 r5 r6 r7 r8
1.1 0 0 0 0 0 1 0 0 -3.1 1 0 0 0 0 0 0 0 -2.2 0 1 0 0 0 0 0 0 0.8 0 0 0 0 1 0 0 0 -1.7 0 0 1 0 0 0 0 0 3 0 0 0 0 0 0 1 0
Algorithm to select r (ranges) for each driver to compensate performance variability between drivers
Discretized steering angle for one driver :
pkk i
4 ri pk
k i -1
4 for i 1 4 (1)
pkk 9 i
4 ri pk
k 8 i
4 for i 5 8 (2)
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
• Some drivers are more “sensitive” to vehicle lateral position and make very accurate corrections to the steering for lane keeping while other are less “sensitive” and make less accurate corrections.
• The result is a low amplitude signal (steering angle) for more “sensitive” drivers and relatively high amplitude signal for less “sensitive” drivers.
• Larger values for Pk will make the descritization ranges
wider to accommodate large amplitude while small values will make them shorter for small amplitudes.
• Therefore, same ANN (8-dimensional descritization) can be used
Accounting for Individual Driver Behaviors
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Eye closure data is recorded at 60 Hz
Ci = No. of zero’s in 1 second of data
Ci is further discretized according to the following scheme
Input Discretization of Eye closures
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Algorithm to select r (ranges) for each driver to compensate eye closure variability between drivers
P values are representative of variability of eye closures (blinking) for each driver
129ifor9
i17k kpir9
i-18k kp
Sample of a few seconds of Discretized Eye closures for one driver :
Input Discretization of Eye closures
E(T) = [e1, e2, e3, e4] Time T sec
Ci
e1 e2 e3 e4
1 4 1 0 0 0 2 7 0 1 0 0 3 0 1 0 0 0 4 18 0 0 1 0 5 0 1 0 0 0 6 1 1 0 0 0 7 6 0 1 0 0 8 1 1 0 0 0
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Input Vector
The two vectors are combined to form a 12 dim
vector J(T)
Vector J(T) is summed over 15 sec time interval to
get the input vector X(n)
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
X(n) D(n)
x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 SLEEP WAKE
0 0 1 4 8 2 0 0 12 3 0 0 0 1
0 1 0 14 0 0 0 0 9 2 3 1 1 0
2 0 5 4 3 1 0 0 0 1 5 9 1 0
0 0 2 3 9 1 0 0 11 4 0 0 0 1
0 0 0 10 5 0 0 0 11 3 1 0 0 1
0 5 3 6 1 0 0 0 8 3 2 2 1 0
1 4 1 3 4 0 1 1 7 3 2 3 1 0
1 5 2 0 5 1 1 0 10 1 1 3 1 0
Input and Desired Output Vector
Each row represents the sum of discretized input over a selected time interval, e.g., 15 sec.
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Performance During Training
MSE vs Epoch
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 201 401 601 801 1001 1201 1401 1601 1801
Epoch
Ave
rag
e M
SE
Training
Cross Validation
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
ANN Test Data• Driving data from 12 subjects available
• 1 subject night session not recorded due to equipment error.
• 1 subject morning data not available, software error.
• Remaining 10 were used for training ANN and testing results,
• NOTE: training data and testing of the ANN were not the same, Testing data selected randomly from the sets not used in the training
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Results
Performance SLEEP WakeMSE 0.0550 0.0554NMSE 0.2205 0.2218MAE 0.1259 0.1245Min Abs Error 0.0000 0.0000Max Abs Error 0.9857 0.9806r 0.8851 0.8840Percent Correct 92.3000 93.0000
Actual Totals Network OutputWake Sleep
Wake 193 179 14Sleep 207 16 191Mis-classified
False Alarm
Actual Totals Network OutputWake Sleep
Wake 193 179 14Sleep 207 16 191Mis-classified
False Alarm
Crash Prediction: All crashes that occurred due to driver falling asleep during the experiment were predicted before the crash occurred.
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Morning and Night session results
Subject 01 two day driving
0
1
1 11 21 31 41 51 61 71 81 91 101
111
121
131
141
151
161
171
181
191
201
211
221
231
241
251
261
271
281
291
301
311
321
331
341
15 sec time intervals
St + Eye
Drowsy Wake Crash
0
0.2
0.4
0.6
0.8
Eye
Fraction of time eye is closed
0
1
Steering
Morning Night
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Morning and Night session results
Subject 02 two day driving
0
1
1 13 25 37 49 61 73 85 97 109
121
133
145
157
169
181
193
205
217
229
241
253
265
277
289
301
313
325
337
349
361
373
385
397
409
15 sec time intervals
St + Eye
Drowsy Wake Crash
0
0.2
0.4
0.6
0.8
Eye
Fraction of time eye is closed
0
1
Steering
Morning Night
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Morning and Night session results
Subject 06 two day driving
0
1
1
11 21 31 41 51 61 71 81 91
101
111
121
131
141
151
161
171
181
191
201
211
221
231
241
251
261
271
281
291
301
311
321
331
341
15 sec time intervals
St + Eye
Drowsy Wake Crash
0
0.2
0.4
0.6
0.8
Eye
Fraction of time eye is closed
0
1
Steering
Morning Night
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Morning and Night session results
Subject 07 two day driving
0
1
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103
109
115
121
127
133
139
145
151
157
163
169
175
15 sec time intervals
St + Eye
Drowsy Wake Crash
0
0.2
0.4
0.6
0.8
Eye
Fraction of time eye is closed
0
1
Steering
Morning Night
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Morning and Night session results
Subject 08 two day driving
0
1
1 10 19 28 37 46 55 64 73 82 91 100
109
118
127
136
145
154
163
172
181
190
199
208
217
226
235
244
253
262
271
280
289
298
15 sec time intervals
St + Eye
Drowsy Wake Crash
0
0.2
0.4
0.6
0.8
Eye
Fraction of time eye is closed
0
1
Steering
Morning Night
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Time Before Crash When the ANN Generated a first Warning
0
0.5
1
1.5
2
2.5
3
3.5
4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Crash No.
Tim
e be
fore
Cra
sh (
min
)
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Conclusions• A non-intrusive method of drowsiness detection using
steering data is possible
• A method using ANN is developed and successfully predicts drowsiness (91% Success Rate)
• Method is solely based on driver’s (Vehicle) steering performance
• Same method may be applied to detection of fatigue or other related driver performance
• Further refining and validation of the algorithm is recommended
• Capturing individual driver’s steering while drowsy requires additional research
CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI
Recommended Additional Research• Additional Simulator Experiments
– Validate the Developed Algorithm– Additional Road Conditions– More Diversified Group of Drivers
• Road (Experimental) Tests in an Instrumented Vehicle
• Further Refining the Algorithm Based on the Road Test Data
• Testing of Other Fatigue Related Scenarios• Research on Warning Systems Integrated With This
Detection System