A Combined Eye States Identification Method for Detection of Driver Fatigue

Feng Yutian, Hu Dexuan, Ning Pingqiang

School of Communication and Information Engineering, Shanghai University Shanghai, P.R China, 201800

Email:fengyt@shu.edu.cn

Key words: Driver fatigue detection, eye state identification, template matching recognition, upper-eyelid curvature calculation, projection for closed eye detection

AbstractFor driver fatigue detection system, the correct identification of open and shut eye states matters much to the subsequent calculation of driver fatigue degree. A combined eye state detection algorithm is proposed after a detailed introduction of three algorithms: template matching recognition, upper eyelid curvature calculation, and projection for closed eye detection. First the algorithm of template matching recognition is used as the main method to identify the eye states. Then, before outputting the final result, with an advanced threshold, some images would have a second or third detection for eye states by the use of upper eyelid curvature calculation and projection for closed eye detection. An accuracy of 95.67% for eye states identification in our experiments shows that this algorithm is time saving and robust to illumination and has good prospect of application in a fatigue warning system for drivers.

1 Introduction Fatigue driving has become one of the most important causes of traffic accidents. And the identification of eye states has high correlation with the computing of drive fatigue degree which could reflect driver drowsiness reliably. Besides eye location, eye states detection can be of vital importance as the output of open or shut states would be used in the subsequent processing.

To detect the eye states, many efforts have been made in the past years and some algorithms have been proposed. On the whole, those methods can generally be divided into two categories: One is based on the eye geometrical characteristics and the other based on pattern recognition. Some typical methods in the first category includes: Hough-Transform for iris detection[1], and eye area computing[2] etc. The method of Hough-Transform for iris detection is through searching iris in a well-detected eye picture by Hough-Transform to determine eye states. Once the iris is found in the picture, the eye state is considered as open, or is closed. This method depends too much upon the

initial value, and it is easy to get into local maximum which would cause incorrect iris locating. What’s worse, it only has good effect on open-eye pictures. While, eye area computing is a method which computes the real-time percentages of the biggest eye area. This algorithm is speedy but the effect is not good enough and easy to change with the illumination of the pictures. However, the second category which based on pattern recognition includes algorithms such as Support Vector Machines, Fisher method[3] neural networks[4], etc. They are generally convert the problem of identifying eye states into a question of classifying the opened or closed states, that is, consider and recognize the patterns of eye states. The converting vector can be used as a benchmark for opening or shutting judgment. And the Fisher method is through projection to change a high-dimensional picture to one-dimensional picture. The algorithm of neural networks, however, is through an advanced trained network to judge the eye state with inputting some extracted eye features. The performance of the second category is generally better than that of the first. But there are difficulties to apply in our real-time system due to the complexity of training and the diverse varieties of the images.

It has been found that the combinative detection algorithm can improve the precision effectively. Yin Li et al.[5] has ever described a combined method based on five different algorithms, but it was complicated and time-consuming which make it unacceptable in real-time driver fatigue detection system. In this paper, a newly combined algorithm which combined three methods is proposed. The three taken methods are: template matching recognition, upper eyelid curvature calculation, and projection for closed eye detection. With combination, the algorithm utilizes strengths of the three methods respectively. Performance has been evaluated relative to a database of 300 eye images with 95.67% of correctness of eye states detection. The remainder of this paper is organized as follows. Section 2 introduces the three eye states detection methods in detail. The combined algorithm is presented in section 3. In section 4, experiments and analysis are presented and conclusions are drawn in section 5.

2 An introduction of three eye states detection algorithms

217

2.1 Template matching recognition

To identify the opened and shut states, template matching recognition is a simple but relatively accurate method. In the gray-scale space, through the calculation of the correlative coefficient between the test image and two states templates images, the eye state can be determined. If the coefficient between the image and the open template is larger than that between the closed one, the state is judged open, and vice versa.

In the calculation of the correlative coefficient, open or closed eye template needs to be made. First, through scaling 100 open eye images, each image should have the same resolution. After that, make gray for each one and add all of the gray images, a sum can be calculated. Third, after averaging and cutting off the surrounding blank, the open template can be made. Also, there is the same process for making of closed template.

Fig.1 Open and closed eye template

In the matching process, the calculation of coefficient R between two images (suppose A and B) can be expressed as equation (1)

1/2 1/22 2

1 1 1 1 1R

ij ij

n m n m n

ij ijj i j i j

a b a b …… .(1)

where11 12 1 11 12 1

1 2 1 2

... ...... ... ... ... , ... ... ... ...

... ...

n n

mn mnm m m m

a a a b b bA B

a a a b b b

After each calculation, the template will be sliding a pixel in the test image. And according to the size of the test image and the templates, a series of correlative coefficients is obtained. The biggest one will be picked out as the final coefficient. Repeating these steps both for the open and closed template, two coefficients can finally be calculated. By comparison, the state of the test image is set as the bigger one.

2.2 Upper eyelid curvature calculation

It is easy to find that the upper eyelid seems like a downward parabola when the eye state is open while it is almost upward when eye state is closed. Utilizing this feature, the curvature of an upper eyelid can be calculated[6].

In the calculation, Sobel operator will be used for extracting the boundary of an eye image. And to avoid the interference of the boundary pixels, the middle part of the curve would be extracted to calculate the curvature (Fig.2).

Original Image Sobel Image Middle part of eyelid Fig.2 Processing of curvature calculation

Due to the huge complexity of the direct curvature calculation, an approximate calculation on condition that no precision will be affected is proposed as follows:

Fig.3 Approximate Calculation

2.3 Projection for closed eye detection

In a closed eye binary image, there often exists two considerable close lines. Both lines represent two maximums, and this feature can be used to judge the closed eye. Of course, when using this method, according to your test data set, threshold values of the maximum (it is set 5 pixels in our experiment) and the distance between the two lines (also set 5 pixels in our experiment) need to be set in advance for judgment. (See Fig.4)

0 5 10

5

10

15

Fig.4 Processing of projection algorithm

3 The combined eye states detection Algorithm The driver fatigue detection system makes tremendous demand of the precision and the speed of an algorithm. Nevertheless, the former three methods introduced in Section 2 can not fully satisfy the request. The method of template matching recognition which is more accurate than the other two still does not reach a satisfied correct rate. And the upper eyelid curvature is more suitable for open eye detection than closed eye detection; while, the projection method is just the other way. To ensure the accuracy as well as the speed, it’s reasonable to pick some images out, but not whole, for rejudgement. Based on this thinking, a combined eye states detection algorithm is proposed to meet these demands. This algorithm including three steps which can be expressed in detail as follows:

Step One: In the process of eye state identification, the method of template matching recognition is utilized first. With the calculated correlative coefficient, an absolute value of difference can be worked out. If the coefficient between the test image and the open eye template is A1, and that between the test image and the closed eye template is A2,then an absolute value of difference is |A1-A2|. The first

218

threshold vale K1 should be set in advance in order to evaluate the absolute value |A1-A2|. If |A1-A2| is larger than K1, the final result can be outputted or the image will go step two.

Step two: the upper eyelid curvature is calculated, and the second threshold vale K2 should be set in advance. Suppose the curvature of the test image is B1, and if B1 is larger than K2, then the result of open is outputted or the image will go step three.

Step three: projection for closed eye detection is used in this step. And the final result will be outputted according to the judgment of this method.

The flowchart is depicted in Fig.5

Fig.5 Flow chart of combined algorithm

4 Experiments and Analysis The goal of this section is to experimentally and scientifically demonstrate the validity of the combined algorithm. For comparison, the related three methods mentioned in section 2 have also been experimented.

4.1 Test Data Set

Although there are many public databases for face detection and face recognition, there is no public database for eye state detection. CAS-PEAL[7] face database is built by the ICT-ISVISION Joint Research & Development Laboratory under the sponsor of National Hi-Tech Program and ISVISION Technologies Co., Ltd. The database includes 99,450 face images with variations in illuminations, expressions, poses and accessory. Only small amount of images has been authorized to us, but it can full satisfy our need for experiments. In our former research, we extract 300

images of left eyes in the database including variation in illumination and poses. In the 300 images, open and closed eyes were taken up 150 respectively. All the eye images are gray scaled with a resolution of 30*15, while the resolution is 20*10 for the template models. Fig.6 shows images gotten from the database.

Fig.6 Open and closed eye images

4.2 Experiment Results

In plenty of experiments, when K1 was set as 0.02, and K2was 0.062, the highest correct rate can be got. The hardware platform is Intel Pentium Dual 1.8G with 1G Memory and Matlab7.0 is used as software platform.

The following tables show the results gotten from the four methods.

Tab.1 Results of template matching recognition

Image\Templates A1-A2

0.9440 0.9160 0.028

0.9755 0.9916 -0.0161

Tab.2 Results of upper eyelid curvature Image Curvature

0.2921

-0.2076

Tab.3 Results of projection for closed eye detection Image Projection for closed eye detection

Open

Close

Tab.4 Overall results

Algorithms CorrectRate

Elapsed Time (Seconds/Frame)

Template Matching 85.67% 0.0113

Upper Eyelid curvature 76.33% 0.0047

Projection for Closed Eye Detection 84.33% 0.0047

Combined Method 95.67% 0.0131

219

In the method of Upper eyelid curvature, the correct rate is less than the others. This is because there are some test images with glasses, or some eyes open too small that caused errors. The proposed method gets the correct rate of 95.67%, which is much better than the former three. In the elapsed time, the template matching is a little longer, because it takes much in the calculation of correlative coefficient. The overall time elapsed is 0.0131s/f which can fully satisfy in the application of real-time driver fatigue detection system if 30 frames per second when shooting a video.

The results demonstrate our thought. The combined algorithm puts the template matching at its first place which will ensure the accuracy of the whole algorithm, while the other two methods, in fact, which would not spent too much time, act as the complements to enhance the correctness rate further.

5 Conclusions and future work In this paper we present a novel combined eye states identification algorithm which composes three eye states detection methods. It includes template matching recognition, upper eyelid curvature calculation, and projection for closed eye detection. The proposed method has been tested and results show that this combined algorithm yields a much more robust, reliable and accurate fatigue detection than using a single one. In the future, we will add the tracking algorithm in our system and develop a better algorithm that can detect driver fatigue more accurate and time-saving.

References [1] Toennies, K.Behrens, F.Aurnhammer, M. “Feasibility

of hough-transform-based Iris localization for real-time-application”, Proceedings International Conference on Pattern Recognition, Vol.16,pp.1053-1056, (2002).

[2] Miyakawa, T. Takano, H. Nakamura, K. “Development of non-contact real-time blink detection system for doze alarm”, SICE 2004 Annual Conference, Vol.2, pp. 1626- 1631, (2004).

[3] Huabiao Qin, Yongpin Gao, Honglin Gan.Precise Eye “Location in Driver Fatigue State Surveillance System”, Vehicular Electronics and Safety,2007 ICVES. IEEE International Conferenc, pp.1~6, (2007).

[4] NG.Narole, Dr.P.R.Bajaj. “A Nero-Genetic System Design for Monitoring Driver’s Fatigue: A”, IJCSNS International Journal of Computer Science and Network Security, Vol.9, pp.87-91, (2009).

[5] Ying Li, Jianhuang Lai. “Combined Method of Eye States Detection”, Journal of Image and Graphics,Vol.8(A),No.10 pp.1140~1145, ( 2003).

[6] Lei Yunqi,Yuan Meiling. “Recognition of Eye states in Real Time Video”, Computer Engineering and

Technology, 2009. ICCET, Vol. 1, 22-24 Jan. 2009 pp.554 -559, (2009).

[7] Wen Gao, Bo Cao, Shiguang Shan, Xilin Chen, Delong Zhou, Xiaohua Zhang, Debin Zhao. “The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations”, IEEE Trans. on System Man, and Cybernetics (Part A) , Vol.38,pp.149-161,(2008).

220