Intelligent System Based Car Collision Prevention

School of Computing & Electrical Engineering

Intelligent system based car collision prevention

A project report s

requirements for the award of the degree of

ELECTRICAL ENGINEERING

Bahir Dar University

Institute of technology



By

Nebiyou Tarekegn

Solomon Genene

Tesfaye Asmera

Advisor

Mr. Tesfamichael Agidie (M.Sc)

A project report submitted in partial fulfillment of the

requirements for the award of the degree of

BACHELOR OF SCIENCE In

ELECTRICAL ENGINEERING

June, 2010



ubmitted in partial fulfillment of the

June, 2010

Bonafide certificate

Certified that this project report entitled “INTELLEGENT SYSTEM

BASED CAR COLLISION PREVENTION” is a Bonafide work carried out by

Mr.TESFAYE ASMERA, Mr. SOLOMON GENENE, and Mr. NEBIYOU

TAREKEGN who carried out the Project under my supervision. Certified further, that

to the best of my knowledge the work reported herein does not form part of any other

project report or dissertation on the basis of which a degree or award was conferred on

an earlier occasion on this or any other candidate.

Name Mr. Solomon Lule (M.Sc)

Head of the department signature of HOD

Name Mr.Tesfamichael Agidie (M.Sc)

Advisor signature of advisor

i

Abstract

Intelligent system based car collision prevention is the system that uses

different sensors to pass intelligent decision that mitigates or avoids collision.

Collision avoidance system mainly applies image and video processing techniques to

pass the intelligent decision that can warn the driver under different environment. The

system uses feature based technique that applies Hough transform method to detect

road lanes. The Hough transform is a feature extraction technique for identifying the

locations and orientations of lane features in digital image. The intelligent system uses

safe distance warning as collision avoidance method. It uses two safe distances to warn

the driver before the collision occurs. These safe distances are warned using visual and

audible system. The visual safe distance warning employs real time object detection

and tracking. Object is detected using background subtraction method while tracking is

based on the centroid of the detected object. The second safe distance is determined by

using Hondas algorithm. We use the infrared laser radar and different sensor to pass

decision in audible form.

ii

Acknowledgement

We would like to take this opportunity to express our deepest gratitude to our

Advisor ATO Tesfamichael Agidie, for guiding us in the technical aspect as well as

imparting his knowledge to us, especially for his helpful and offered invaluable

assistance, support and guidance throughout this work. Special thanks dedicated to our

friends who have been supporting, guiding and advising us in some part of the project,

Appreciation is also acknowledged to those who have contributed directly or indirectly

in the completion of this project. Lastly, our deepest gratitude to beloved GOD, because

we have came through this project only with its help.

iii

Table of contents Abstract ..................................................................................................................... i

Acknowledgement ..................................................................................................... ii

Table of contents ....................................................................................................... iii

List of figures ............................................................................................................ v

List of symbols, acronym and abbreviation ................................................................ vi

1. Introduction ........................................................................................................... 1

1.1 Back ground of the project ................................................................................. 1

1.2 Objective of the project ...................................................................................... 3

1.2.1 General objective ......................................................................................... 3

1.2.2 Specific objective ......................................................................................... 3

1.3 Methodology……………………………………………………………………...3

1.3.1 Literature review………………………………………………………...........3

1.3.2 MATLAB computing environment…………………………………………...3

1.4 Project scope……………………………………………………………………....4

1.5 Motivation………………………………………………………………………...4

1.6 Project goal………………………………………………………………….…….4

1.7 Document organization………………………………............................................5

2 Literature review and basics of image and video processing……………………...….6

2.1 Literature review……………………………………………...............................6

2.1.1 Introduction…………………………………………………………….…….6

2.1.2 Lane detection and departure warning method………....................................6

2.1.2.1 Lane detection method………………………………………………......7

2.1.2.2 Lane departure warning system method……………………………..…..8

2.1.3 Object detection and tracking methods……………………………………....9

2.2 Basics of image and video processing…………………………………..……….11

2.2.1 Noise removal……………………………………………...………...............11

2.2.2 Image enhancement and intensity adjustment…………………………….…12

2.2.3 Image segmentation………………………………………………….............13

2.2.4 Morphological operation…………………………………………………….13

2.2.5 Edge detection……………………………………………………………….14

2.2.6 Hough transform……………………………………………………………..14

3 System description…………………………………………………………..…….....16

iv

3.1System design and algorithm development…………………………..………….16

3.1.1 System design……………………………………………………...………..16

3.1.1.1 Hard ware specification…………………………………..……….….....16

3.1.1.2 Principles of operation……………………………………………..........18

3.2 Lane detection and departure warning system………………………………….24

3.2.1 Introduction………………………………………………………………....24

3.2.2 System configuration………………………………………………………..24

3.2.3 Approach to lane detection and departure warning…………………..….…25

3.2.4 Algorithm development……………………………………………………..26

3.2.4.1 Algorithm description……………………………………………...…....28

3.2.5 Algorithm for lane detection and departure warning system………..……...31

3.3 Safe distance warning…………………………………………………….…......33

3.3.1 Introduction …………………………………………………………….......33

3.3.2 Safe distance determination…………………………………………….......33

3.3.3 Algorithm development……………………………………………………..34

3.3.4 Object detection and tracking for visual safe distance warning……….........36

3.3.4.1 Introduction……………………………………………………………..36

3.3.4.2 System configuration…………………………………………………....36

3.3.4.3 Approach to object detection and tracking………………………..….....37

3.3.4.4 Algorithm for object detection and tracking………………………........39

4 Results and Discussion………………………………………………………….…..41

4.1 Implementation and results……………………………………………...............41

4.2 Problems encountered…………………………………………………………...48

5 Conclusion …………………………………………………………………….........49

Appendixes…………………………………………………………………..……...50

References………………………………………………………………………......53

v

List of figures

Fig 2.1Cartesian coordinate and polar coordinates …………………….......................15

Fig 3.1 Pin diagram of TLC 548 ADC………………………………………………..17

Fig 3.2 Functional block diagram of TLC 548 ADC…………………………………18

Fig 3.3 Interfacing circuit between infrared laser radar and CPU………………….…21

Fig 3.4 Video camera, infrared laser radar and buzzer allignment on the car………...22

Fig 3.5 The position of display unit, processor and buzzer within the car when viewed

from the top left……………………………………………………………………….22

Fig 3.6 General block diagram of the system…………………………………………23

Fig 3.7 Block diagram of lane detection and departure warning system……………..24

Fig 3.8 Flow chart of lane detection and departure warning………………………….27

Fig 3.9 Block diagram of safe distance warning system……………………………...34

Fig 3.10 Block diagram of object detection and tracking………………………….….37

Fig 3.11 Flow chart for object detection and tracking………………………….……..40

Fig 4.1 Images from video frame sequence……………………………………….…..41

Fig 4.2 Images from consecutive video sequence………………………………….....42

Fig 4.3 Oout put of back ground subtraction for images from fig4.1……………...…42

Fig 4.4 Output of back ground subtraction for images from fig4.2…………………..42

Fig4.5 Output of object detection and tracking from video sequence…………….......43

Fig 4.6 a) Output of object detection and tracking from video sequence………….….43

b) Lorry detected on the road lane…………………………………….….…...43

Fig 4.7 Input image of road lane……………………………………………………....44

Fig 4.8 Accumulator array of Hough transform…………………………………..…..45

Fig 4.9 Image result with fake and road lane detected……………………………..…45

Fig 4.10 Road lane and fake lane detected by Hough transform…………………..….46

Fig 4.11 Binary image found after noise removal………………………………..…...46

Fig 4.12 Output image after boundary extraction…………………………………......47

Fig 4.13 Lanes detected using Hough transform followed by boundary extraction…..47

vi

List of symbols, acronyms and abbreviations

ADC analog-to-digital converter

CCD charge coupled camera

CHEVP canny/Hough estimation of vanishing points

CLK clock

CMOS Complementary Metal-Oxide Semiconductor

CPU- central processing unit

CS chip select

DSP digital signal processing

D out data out put

DTR data transmitter and receiver

FSB full system back up

GND ground

GHZ giga hertz

IC integrated circuit

I/O input /output

LIDAR light detection and ranging

LPT local printer terminal

MMSE minimum mean square error

MSB most significant bit

nF nano farad

OS operating system

PC personal computer

PT pan-tilt

PTZ pan-tilt-zoom

RAM random access memory

REF reference

RGB red-green blue

SATA serial advanced technology attachment

SDRAM synchronous dynamic random-access memory

TH threshold

TTL transistor-transistor logic

vii

USB universal serial bus

WT wavelet transform

VIN voltage input

Ω Omega

eg Example

GHZ Giga hertz

GB Gigabyte

KHZ Kilo hertz

mW milli watt

rpm revolution per minute

µF micro farad

Vrel relative velocity

1

Chapter 1

1 Introduction

1.1 Background of the project

Transportation is one of the most important economic activities of any countries.

Among the various forms of transportation, road transportation is one of the most

popular means of transportation. As the number of vehicles on the road increases, so do

the dangers and economic losses from automobile accidents. Transportation has an

element of danger attached to it in the form of vehicle crashes. These crashes not only

cause death and injury but also bring along an immeasurable amount of agony to people

involved. There are different causes of collision and vehicle crashes. These are:

Road condition

Driver performance

Weather condition

From these causes of collision driver performance can affect transportation

greatly. It can control other collision sources. The collisions that occur due to driver

performance routed its base on the following errors.

Recognition error

Decision error

Performance error

In the past, safety systems have focused on reducing driver injury in case of an

emergency. Thus the introduction of seat belts, air bags, and more recently products

like on Star, which can automatically contact emergency services and help locate the

scene of an accident.

But, these days the development of vision based sensors facilitated the

application of computer vision technology in the traffic flow control. The application of

this technology will be able to warn drivers of dangerous situations in time to take

preventative action. With better sensors and data communication techniques, a driver

can be more aware of his or her environment. This allows the driver to react

appropriately to situations such as lane departure and collision occurrence.

A collision avoidance system is a system of sensors that is mounted on a car to

warn the driver if any dangers may lie ahead on the road. Some of the dangers that

2

these sensors can pick up on include how close the car is to other cars surrounding it,

how much its speed needs to be reduced while going around a curve, and how close the

car is to going off the road. The system uses sensors that send and receive signals from

things like other cars and obstacles in the road.

Lane departure warning system is one part of collision avoidance system that is

used to control the lane maneuvering of the car. It is the system that is used to warn the

driver when the car is moving out of lane.

The introduction of charge coupled video camera (CCD camera) enabled modern

cars to use vision intelligence in determining and following road lane. In this project

lane detection and departure warning system is developed to warn the driver if the car

is moving out of lane. It uses camera mounted on the vehicle to capture video image of

road information in front of the car. A video image is made of basically a series of still

images at a very small interval time between each capture. Then the image processing

application will retrieve useful information for the issuance of warning. It involves the

applications such as feature extraction and Hough transform. Lane departure warning

system proved to be very useful on high ways or clearly marked lanes. However, lane

detection and tracking is a challenge problem if roads are with incomplete lane

markings or roads lane markings are hidden due to parked cars or shadow.

Another part of this project is the object detection and tracking for collision

avoidance. It has importance in detecting obstacles and there by tracking them. It uses

video image from video camera to detect and track the upcoming or different objects in

front of the car. This project considers for simple object detection using feature

extraction and image segmentation.

In general intelligent system based car collision prevention can be understood as

the system that can be mounted on the car or fabricated together with a component of

car in order to detect obstacles ,objects and humans, warn driver and avoid the damage

to any resource in the vehicles path and vehicles itself.

3

1.2 Objective of the project

1.2.1General objective

The general objective of this project is to facilitate traffic flow control and

increase transportation safety.

1.2.2 Specific objective

This project has the following specific objectives

Developing algorithm for lane detection and departure warning system

Developing algorithm for object detection

Implementing the developed algorithm using image processing software

Implementing real time object detection

1.3 Methodology

So as to achieve the above stated objectives various methods and tools have been

used. The following list will describe the methods and the tools used in this work:

1.3.1 Literature Review

The literature review part has focused on reviewing the basis of video and image

processing along with that of the related works.

1.3.2 MATLAB computing environment

For the purpose of lane detection and real time object detection and tracking an

algorithm development for an implementation of the system on the video images

captured by camera requires suitable software. For the sake of this MATLAB

computing environment is the robust as it is high-performance language for technical

computing. It integrates computation, visualization, and programming in an easy-to-use

environment where problems and solutions are expressed in familiar mathematical

notation.

Typical uses include:

Math and computation

Algorithm development

Data acquisition

Modeling, simulation, and prototyping

MATLAB is suitable because it contains image and video processing toolbox

which provides basic framework for the development of the system. The toolbox is a

4

collection of functions that extend the capability of the MATLAB numeric computing

environment. The toolbox supports a wide range of image processing operations.

1.4 Project scope

Intelligent system based car collision prevention is the wide area that requires the

contribution from different area of engineering. It consists of lane keeping system, lane

departure warning system, antilock braking system, auto brake and start system,

obstacle detection, and safe distance warning system.

The scope of this project is mainly to focus on the development of a software module

for the lane detection and departure warning as well as object detection and tracking

using the image and video processing.

1.5 Motivation

The motivation behind this project is that nowadays, there are various car

collision accidents that have damaged many useful human and material resources. To

combat these accidents many systems have been proposed, many traffic rules have been

applied but none of them resulted in avoiding the accidents. Because they are passive in

nature, they don’t pass decisions rather they give help only after collision have

occurred. Such systems are passive systems and often collision occurs when the car

goes out off lane, when obstacle suddenly enters within the lane etc.

We have thought that if vision technology is applied the accidents can be reduced to a

great extent. Using vision technology the system can pass intelligent decision with the

help of different sensors.

1.6 Project goals

The following goals has been attained in this project

The system can detect road lane markings and pass lane departure warning

The system can detect any object in front of the car

The system can track any detected object

The system can pass safe distance warning.

5

1.7Document organization

This document has four main parts. The first part discusses about back ground of

the project, Project objectives, motivation and goals. The second part of the project

describes previous works on lane detection and object detection. It discusses about

image and video processing that are employed in this work. The third part of the project

is the main body of the project. It mainly focuses on the algorithm development for

road lane detection and departure warning as well as for safe distance warning. It also

cites system configuration for safe distance warning and lane departure warning.

Project implementation and results are discussed deeply under this part of the project.

The last part of the document deals with conclusion and future recommendations.

Appendixes and reference is described under this part.

6

Chapter Two

Literature review and basics of image and video processing

2.1 Literature review

2.1.1Introduction

With the application of intelligent system collision can be avoided. Intelligent

system mainly involves vision technology in detection, warning and braking of the car.

Such a system uses a number of sensors and processors to mitigate collision efficiently

and effectively. The most effective and usable sensor in vision technology is the video

sensor that is used to capture images for detection purposes. Over the past years the

application of vision system is not that much used in avoiding collision. But now the

detection of object and lane markings became active research areas in collision

avoidance system. Many algorithms have been proposed for the lane departure warning

system and object detection and tracking. Some of these algorithms are based on

modeling technique while others are based on feature based techniques. The application

of modeling technique is robust to different noises. But the modeling technique requires

complex computation and different assumption. The feature based technique uses some

feature of interest for object detection and tracking as well as lane marker detection and

departure warning. However, it is easily affected by noises and different occlusions.

2.1.2 Lane detection and departure warning methods

The main feature of any collision avoidance system is firstly to keep the car

within its own lane. Keeping the car within its own lane can avoid many collisions that

will occur due the car moving out of its lane. To develop such a system different

algorithms are proposed. The lane detection and departure warning methods are

classified into:

Lane detection method

Lane departure warning method

7

2.1.2.1 Lane detection method

Basically, there are two groups of approaches available in dealing with lane

detection problem. They are feature-based technique and model based technique. The

model-based technique uses a few parameters to represent the lane mapping or

structures. But this approach needs some complex computing and estimation. While,

the feature based technique localizes the lines in the perspective view images or

inversely projected view by combining the low level features such as painted lines or

line segment that are detected by simple image segmentation. A summary of various

method used for lane detection will be as follows

a. Lane detection using Cat mull-Rom spline

This method uses the modeling of road lane using the formulation of the lane

detection problem in the form of determining the set of lane model control points [11].

It uses three sets of control points. This method first determines the ground model of

the road. It then estimates what will the lane or road looks like on the image plane.

Then it detects the vanishing point of road lane and assumes it as first point. Then it

searches for control point in Edge image for the left lane model by estimation. It uses

this point to get the point on the right lane model. These points on the left and right lane

form the second group of control points. The third group of control point is at the

intersection of lane with image plane boundary. It uses interpolation to connect these

points and detect road boundary.

b. Lane detection using Hough transforms

This method uses feature based technique with normal perspective view of a

single CCD camera projection to detect road lanes. It uses low level image processing

and Hough transform. It applies the Hough transform algorithm to extract region that

looks like road markings [6].

c. Lane detection based on linear parabolic lane model

This method is one of the model based technique that successfully detect road

lane using linear parabolic model. A linear function is used to fit the near vision field,

and a quadratic function fits the far field [4]. The proposed algorithm is robust in

different illumination conditions and can efficiently handle the problem of detecting

curved road lane. It identifies the near and far field using certain threshold value. The

8

algorithm indicates the lane boundary model can be described using three coefficients.

It uses weighted least square method to compute their values. It fits this model to the

lane by using lane boundary region of interest.

d. Lane detection using B snake[12]

This method uses the Perspective effect of parallel lines that is constructed with

dual external forces for generic lane boundary or marking. It enables to describe a

wider range of lane structures Compared with other lane models and is able to describe

a wider range of lane structures since B-Spline can form any arbitrary shape by a set of

control point. In addition, it is robust against shadows, noises, occasional missing and

false markings due to the use of the parallel knowledge of roads on the ground plane.

Furthermore a robust algorithm, called Canny/Hough Estimation of Vanishing Points

(CHEVP) is presented for providing a good initial position for the B-Snake. Also, a

minimum error method by Minimum Mean Square Error (MMSE) is proposed to

determine the control points of the B-Snake model by the overall image forces on two

sides of lane.

2.1.2.2 Lane departure warning system methods

In driver assistance system lane departure warning system has wide importance.

Various methods have been proposed and developed for lane departure warning system.

Most of them are based on model technique and others use sensors to pass lane

departure warning.

a. Lane departure warning system based on linear parabolic lane

model

Here the system assumes as if the car is moving on the center of left and right

road lane. It then determines both left and right road lane orientation (θl and θr)[4]. It

uses the absolute value of the summation of left and right road orientation and

compares against certain threshold TH and passes lane departure warning. It is based on

a robust lane detection algorithm that is used to compute both lane orientations θl and

θr. The symmetry measure is computed at every frame, and a lane departure alarm is

triggered when symmetry is greater than TH.

9

2.1.3 Object detection and tracking methods

Basically there are many obstacles that can cause car crash on the road. These

obstacles may be moving or stationary object. But most of the time its moving objects

that can cause a collision. To avoid such destruction the car should have the system that

can detect and track any object in front of it. There are many object detection

algorithms. Some of them are:

a. Background Subtraction Techniques

Background subtraction is a commonly used class of techniques for segmenting

out objects in a scene for applications obstacle collision avoidance. Background

extraction techniques emphasizes on three important attributes: foreground detection;

background maintenance; and post processing. The concept of background subtraction

techniques revolves around comparison of an observed image with an estimate of the

image if it contained no objects of interest [10]. The areas of the image plane where

there is a significant difference between the observed and estimated images indicate the

location of the objects of interest. The name “background subtraction" comes from the

simple technique of subtracting the observed image from the estimated image and

thresholding the result to generate the objects of interest.

b. Moving object detection in wavelet compressed video

This method compares the wavelet transform (WT) of the current image with the

WTs of the past image frames to detect motion and moving regions in the current

image without performing an inverse WT operation. Moving regions and objects can be

detected by comparing the WTs of the current image with the WT of the background

scene which can be estimated from the WTs of the past image frames [9]. If there is a

significant difference between the two WTs then this means that there is motion in the

video. If there is no motion then the WTs of the current image and the background

image ideally should be equal to each other or very close to each other due to

quantization process during compression.

This method is computationally effective, as it does all the processing in WT. It

does not inverse the WT to obtain the actual pixels of the current image or the

estimated background.

10

c. A Contour-Based Moving Object Detection and Tracking

This method is used to detect and track moving objects which includes non-rigid

objects. This is based on using lines computed by a gradient-based optical flow and an

edge detector. Edges are extracted by using optical flow and the edge detectors are

restored as lines. Background lines of the previous frame are subtracted. The edges are

masked by a thresholding velocity magnitude for eliminating background edges with

little motion. Unmasked edges are used for restoring lines and extracting contours of

objects in subsequent process. Contours of objects are obtained by using snakes to

clustered-lines. Contours extraction is obtained by performing line restoration, line-

based background subtraction, clustering and active contours [8]. This method is robust

because edge based feature is applied. Edge-based features are insensitive to

illumination changes.

d. Feature Based Object Tracking Using PTZ Camera

Object tracking is an important task within the field of computer vision. The

proliferation of high-powered computers and the increasing need for automated video

analysis has generated a great deal of interest in object tracking algorithms. This part

presented a feature based object tracker which uses a pan-tilt (PT) camera to keep track

of the target. The task is to keep the target at the center of the grabbed image. As the

target starts moving in the real world, its position in the grabbed image is reported in

subsequent frames through a feature based tracking algorithm. The image position error

is processed by a proportional-integral controller and the camera is re-positioned

accordingly to place the target in the pre-specified image region [13].

11

2.2 Basics of image and video processing

Image and video processing has wide applications in vision based systems. It is

used to extract the needed feature for intelligence application. Every video image

captured by video camera must be processed to retrieve some information such as road

feature, obstacle character to enhance the intelligence of the system. The important part

of image and video processing that have wide application in this project are:

Noise removal and neighborhood processing

Image enhancement and adjustment

Image segmentation

Morphological operations

Region based processing

Edge detection

Image feature extraction

Image transforms

2.2.1 Noise removal

There are various ways in which the noise can be introduced to video image

captured for processing. These are noise from atmospheric conditions, camera and

material type used for image capturing. To make the developed vision system robust

the corrupted image properties should be restored. This can be accomplished by using:

Linear filtering

Non-linear filtering

The neighborhood processing involves

Defining center point (x, y)

Performing operation on pixels in a predefined neighborhood

Assigning output of operation to the center point

In linear noise filtering the center point is assigned the sum of values of

neighborhood pixels multiplied by filter coefficients. But it has less influence on real

time noises. This can be done by

a = imfilter (b, c,’replicate’) where a=filtered image

b= input image

c=filter mask

12

Non-linear filtering is based on non linear operations involving the pixels of

neighborhood. There are many non linear filters, but median filter can remove small

pixel noise. Noise is reduced by calculating the median of neighborhood elements and

storing the value to central element. Generally median filter is a 2-D filter that can be

computed as follows

a=medfilt2 (b,[m, n]) where a= filtered image

b=input image

[m, n] defines neighborhood of size m x n over which

median is computed.

2.2.2 Image enhancement and intensity adjustment

The aim of image enhancement is to improve the interpretability or perception of

information in images for human viewers, or to provide better input for other automated

image processing techniques [7].

Once the noise is removed from images some of its abstract features will be

restored. To work with such images and extract its features the image must be

enhanced. This can be done by:

Scaling image

Histogram equalization

Histogram equalization is the method that increases the global contrast of many

images. Through this adjustment the intensities can be better determined on the

histogram. This allows for areas of lower local contrast to gain higher contrast.

Histogram equalization, which involves transforming the intensity values so that the

histogram of the output image approximately matches a specified histogram. This can

be done by using histeq (histogram equalization) function that produces an output

image having intensity values evenly distributed throughout the range.

Image scaling is the process of resizing a digital image. Scaling is a non-trivial

process that involves a trade-off between efficiency, smoothness and sharpness. As the

size of an image is increased, so the pixels which comprise the image become

increasingly visible, making the image appears soft. Conversely, reducing an image

will tend to enhance its smoothness and apparent sharpness. Generally image scaling

improves the brightness of the image by simply multiplying every pixel values by

constant number.

13

2.2.3 Image segmentation

Image segmentation is the process of dividing digital image into multiple

regions. It shows the objects and boundaries in an image. Each pixel in the region has

some similar characteristics like color, intensity, etc. The important image

segmentation used in this project is the background subtraction method. Background

subtraction is the process of separating out foreground objects from the background in a

sequence of video frames. Many methods exist for background subtraction, each with

different strengths and weaknesses in terms of performance and computational

requirements. Frame difference is arguably the simplest form of background

subtraction. The current frame is simply subtracted from the previous frame, and if the

difference in pixel values for a given pixel is greater than a threshold T, the pixel is

considered part of the foreground.

|frame i–frame (i-1)|>T………………………………………………2.1

2.2.4 Morphological operation

Morphology is image processing operations that process images based on shapes.

Morphological operations apply a structuring element to an input image, creating an

output image of the same size. In a morphological operation, the value of each pixel in

the output image is based on a comparison of the corresponding pixel in the input

image with its neighbors.

In vision based system morphological operations on video images has great

importance. It is used for labeling and measuring objects in the binary image. Generally

there are 3-types of images

• RGB images- true color images

• Gray scale intensity images

• Binary images- logical arrays of 0s and 1s

Each of these images can be represented by different data classes. They are

double, uint8, uint16. Double data class is used for carrying operations on images and

uint8 data class is used for storing image as it takes only limited space of memory. For

the sake of different uses inter conversion of image data classes is necessary. Most

morphological operation is carried out on gray scale or binary images. The

morphological operation such as

• Removing loosely connected components

14

• Labeling of object boundaries in the images has great importance in this project.

Using the function BW=bwareaopen (a, p) it’s possible to remove all loosely connected

objects that have fewer than p pixels from image a. This is simply morphologically

open binary image. By the help of different tool box function that exist in MATLAB

we have determined characteristics that play important roles in this project.

2.2.5 Edge detection

Edge detection is the process of finding sharp contrasts in intensities in image.

Edge detection is used to identify the edges in an image. Edge detection function looks

for places in the image where the intensity changes rapidly, using one of these two

criteria:

Places where the first derivative of the intensity is larger in magnitude

than some threshold

Places where the second derivative of the intensity has a zero crossing

There are different types of edge detection. Some of these are Sobel, canny,

prewwit and Gaussian edge detection methods. Among these edge detectors canny

edge detection is the robust one as it with stand noises and extracts most important

edges from images. It uses Gaussian masks to scan all over the image.

2.2.6 Hough transform

The Hough transform is a feature extraction technique for identifying the

locations and orientations of certain types of features in digital image. It describes

features in some parametric form. The purpose of this technique is to find imperfect

instances of objects within a certain class of shapes by a voting procedure. A classical

Hough transform is commonly used for detection of regular curves such as straight

lines, circles and ellipses. Line segments can be described in number of ways. But, a

convenient equation for describing lines uses parametric equations.

Xcosθ +ysin θ =r…………………………………………………………2.2

Where r is length of normal from origin to the line

θ is orientation of r with respect to x axis

For any point (x, y) on the line r and θ are constant.

15

Y

r

θ

X

Fig 2.1 Cartesian coordinate and polar coordinates

In the image space the coordinate of edge points (x, y) are known while r and θ

are unknown variables. The values of r and θ can be found from edge points by using

equation. The plot of possible (r, θ) yields sinusoids in the Hough parameter space.

Collinear points in the Cartesian image space yield curves that intersect at common (r,

θ) point. The input to Hough transform should be the result of edge detection. Hough

transform can be applied by quantizing Hough parameter space into finite intervals

accumulator cells [6]. The accumulator space is plotted regularly with r as ordinate and

θ as abscissa. As Hough transform algorithm runs each (x, y) edge points are

transformed into (r, θ) curve.

Curves generated by collinear points in the image intersect peaks (r, θ) in Hough

transform space. Hence accumulator cells get incremented. This shows the existence of

straight line in the image space. This detected line segments can be extracted on the

basis of Hough peaks or local maxima in the accumulator space. The extraction can be

carried out based on thresholding to implement voting procedure.

16

Chapter 3

System description

3.1 System design and Algorithm development

3.1.1 System design

This part of the project describes the specification, interfacing of the hard ware

and general block diagram of the system used in this project. The project uses different

input materials to capture signals in front of the car. It processes the digital value of

these signals using PC processor to pass different forms of warning. The input devices

are infrared laser radar, video camera, switch, CPU processor, display and buzzer. The

rating and processing speed of CPU processor greatly affects response time of the

system. The CPU processor extracts lane and object information from video captured

by video camera and infrared laser radar by using appropriate software.

3.1.1.1Hard ware specification

Intelligent system based car collision prevention is the collision avoidance

system that uses different sensors and vision technology. Vision technology uses image

processors along with CPU compatible processors to efficiently and effectively detect

lane and objects. DSP signal processor is used to extract information from sensors to

avoid collision and before collision warning. But due to unavailability of image

processors and digital signal processors we are determined to use only CPU compatible

processors to carry out collision mitigation. So, we have to use personal computers with

high memory. The personal computer used has the following specification:

Memory SDRAM above 1GB (1/0) DDR2 SDRAM

Frequency 3.4 GHZ

Operating system windows XP, windows vista

Intel Pentium Dual-Core Processor E5200 (2MB L2 cache, 2.5

GHz, 800 MHz FSB)

Storage 160GB SATA hard drive, 7200RPM

Infrared laser radar is used to determine range and relative speed of object. It has

low loss and effective than radio wave radar. It has transmitter which transmits light

and receiver which receives reflected light. The infrared laser radar has the following

specification:

17

Wave length 850-950nm

Maximum viewing distance 120m

Pulse width 100ns, f=1KHZ

The interface design is needed for connecting infrared laser radar to CPU as it is

not connected directly using USB port or serial port. Infrared laser radar displays the

range and relative speed of obstacle in front of the car in digital form. But it cannot be

interfaced directly to CPU. By tapping the line that inputs analog electrical signal to the

analog and digital converter in the laser radar we can get equivalent electrical signal.

Using this electrical signal and analog to digital converter it’s possible to produce

digital signal that the computer processes. The output of analog to digital converter can

be accessed easily using PC serial port. Since the input signal from radar has high

frequency analog to digital converter with less than 16- bit is preferable. The analog to

digital converter preferred for this operation is the TLC 548 is CMOS analog-to-digital

converter (ADC) integrated circuits built around an 8-bit switched-capacitor

successive-approximation ADC. These devices are designed for serial interface with a

microprocessor or peripheral through a 3-state data output and an analog input. The

maximum I/O CLOCK input frequency of the TLC548 is 2.048 MHz

Pin diagram and pin description of TLC 548 analog to digital converter

Fig3.1 pin diagram of TLC 548 ADC

TLC548 is an 8-bit analog to digital converter with serial output. It is used in

microprocessor Peripheral or Standalone operation. It has On-Chip Software-

Controllable Sample-and-Hold Function. It generally works with 4-MHz Typical

Internal System Clock and requires power supply of Wide Supply Range of 3 V to 6 V.

it robust to other ADC as it consumes low power. It consumes maximum of 15 mW

power.

18

Pin description of TLC 548

Pin 1and pin 3 is used for providing reference voltage that can be used

in differential output voltage.

Pin 2 is used as analog input. It accepts the electrical signal from

infrared laser radar.

Pin 4 is used for grounding.

Pin 5 is power supply

Pin 6 is I/O clock used for data control and operation. The I/O CLOCK

together with the internal system clock allow high-speed data transfer

Pin 7 is the data output terminal.

Pin 8 is the chip select signal used for data control.

Function block diagram of TLC 548 ADC

Fig 3.2 functional block diagram of TLC 548 ADC

TLC 548 ADC has special feature including versatile control logic, an on-chip

sample-and-hold circuit that can operate automatically or under microprocessor control,

and a high-speed converter with differential high-impedance reference voltage inputs

that ease ratio metric conversion, scaling, and circuit isolation from logic and supply

noises.

3.1.1.2Principles of operation

The TLC548 is a complete data acquisition system on a single chip. Each

contains an internal system clock, sample-and-hold function, 8-bit A/D converter, data

register, and control logic circuitry. For flexibility and access speed, there are two

control inputs: I/O CLOCK and chip select (CS). These control inputs and a TTL-

compatible 3-state output facilitates serial communications with a microprocessor or

19

computer. A conversion can be completed in 17 µs or less, while complete input-

conversion-output cycles can be repeated in 22 µs for the TLC548.

The internal system clock and I/O CLOCK are used independently and do not

require any special speed or phase relationships between them. This independence

simplifies the hardware and software control tasks for the device. Due to this

independence and the internal generation of the system clock, the control hardware and

software need only be concerned with reading the previous conversion result and

starting the conversion by using the I/O clock. When CS bar is high, DATA OUT is in

a high-impedance condition and I/O CLOCK is disabled. The control sequence has

been designed to minimize the time and effort required to initiate conversion and obtain

the conversion result. A normal control sequence is:

1. CS bar is brought low. To minimize errors caused by noise at CS bar , the internal

circuitry waits for two rising edges and then a falling edge of the internal system clock

after a CS bar falling edge before the transition is recognized. However, upon a CS bar

rising edge, DATA OUT goes to a high-impedance state. The most significant bit

(MSB) of the previous conversion result initially appears on DATA OUT when CS bar

goes low.

2. The falling edges of the first four I/O CLOCK cycles shift out the second, third,

fourth, and fifth most significant bits of the previous conversion result. The on-chip

sample-and-hold function begins sampling the analog input after the fourth high-to-low

transition of I/O CLOCK. The sampling operation basically involves the charging of

internal capacitors to the level of the analog input voltage.

3. Three more I/O CLOCK cycles are then applied to the I/O CLOCK terminal and the

sixth, seventh, and eighth conversion bits are shifted out on the falling edges of these

clock cycles.

4. The final clock cycle is applied to I/O CLOCK. The on-chip sample-and-hold

function begins the hold operation upon the high-to-low transition of this clock cycle.

The hold function continues for the next four internal system clock cycles, after which

the holding function terminates and the conversion is performed during the next 32

system clock cycles, giving a total of 36 cycles. After the eighth I/O CLOCK cycle, CS

bar must go high or the I/O clock must remain low for at least 36 internal system clock

cycles to allow for the completion of the hold and conversion functions.

20

Here is how the ADC can be connected to serial port. The output of the TLC548

is not directly suitable for standard serial data reception, so the circuit below uses serial

port handshaking lines in a nonstandard way which enables the communication

between computer and converter chip to be implemented with as few components as

possible. The circuit takes all the power it needs from PC serial port. The computer's

RS-232 serial port is asynchronous, so it requires hand shaking control lines because

The A/D converter's serial port is synchronous. But, its handshaking requirements are

minimal and it only requires one wire for clock and one or two wires for data. This can

be accomplished using the RS-232 port's handshake lines. The voltage conversion of

signals from RS-232 port to IC of ADC input pins is done using a combination of series

resistors and the internal input protection diodes in the IC of ADC. The output of this

ADC was able to directly output signals which seemed to be detected by RS-232 port.

Using 25 pin serial port connecter the designed interfacing circuit looks like the

following. The circuit has two parts; the TLC548 ADC IC chip and hand shake lines

formed by diodes, resistors and capacitors. All diodes are IN4148 or similar. All

resistors connected to IC are 30 kΩ and their function is to protect the IC inputs with

the diodes against overvoltage. The 1 kΩ resistor, one diode, 5.1V zener-diode, 10µF

and 100nF capacitors make the power supply which takes circuit power from serial port

µData Terminal Ready –pin. A 25 pin female D connecter is used to connect the ADC

to PC serial port. The circuit diagram of interfacing between PC serial port and ADC is

shown in fig below

Fig 3.3 interfacing circuit between infrared laser radar and CPU

Video camera can be interfaced to computer using USB port. It can be accessed

by MATLAB if compatible driver software is installed for it.

be used as the display unit. Buzzer can be connected to host PC using USB port or it

can be wireless. Video camera should be placed at the position of rear view mirror in

front of the car. It should in normal projective view for lane

warning. Its viewing direction should be in horizontal direction for object detection and

tracking. The position of display unit is near the position of speedometer. This position

is suitable position for the driver potentially vie

placed at the position perpendicular to the sit of the driver. The infrared laser radar is

placed at the in front air intake of the engine in order to view any small and big

obstacles. The system is controlled by the d

inhibit the warning. The driver uses this when he or she thinks to overtake another car

and at car parking.

21

Fig 3.3 interfacing circuit between infrared laser radar and CPU


by MATLAB if compatible driver software is installed for it. The monitor screen can


. Video camera should be placed at the position of rear view mirror in

front of the car. It should in normal projective view for lane detection and departure



is suitable position for the driver potentially view the display unit. The buzzer can be



obstacles. The system is controlled by the driver. The driver can use limit switches to



The monitor screen can


. Video camera should be placed at the position of rear view mirror in

detection and departure



w the display unit. The buzzer can be



river. The driver can use limit switches to


22

Fig 3.4video camera, infrared laser radar and buzzer allignment on the car

Fig 3.5 the position of display unit, processor and buzzer within the car when viewed

from the top left.

23

Fig 3.6 general block diagram of the system

24

3.2 Lane detection and departure warning system

3.2.1 Introduction

Lane detection and departure warning system is the system that is developed to

detect road lane markings and warn the driver if the car is moving out of its lane. To

warn the driver the system should detect lane markings. In this project we have applied

feature based lane detection technique to detect and track road lane markings. The

developed system is robust if the lane markings are visible. The departure warning will

be issued using the deviation from center of the video. The system is implemented

using feature based technique with normal perspective view of a single CCD camera to

capture video images with specific frame rates. The system applies image segmentation

to separate unwanted objects and Hough transform algorithm to detect the presence of

lane marker. By applying the feature properties of lane marker it’s possible to delete

fake lines. The system has some draw back when the lane markings are missing and

highly affected by noises, but it effective on high ways with lane markings.

3.2.2 System configuration

This part of the project describes the system block diagram for the lane detection

and departure warning system. The system is vision based system that analysis video

and passes decisions. It composed of input devices to capture road video, output

devices to promote decisions. The input device is CCD video camera and the output

devices are the display device and buzzer. The input video is processed by processors

to give certain output.

Fig 3.7 block diagram of lane detection and departure warning system

Video

camera CPU processor

Buzzer

Display device

25

Video camera is used to capture road videos with specific frame rates. Now-a-

days most of the cameras are available with USB interface. By installing the driver for

the camera, the computer detects the device whenever you connect it. But for CCD

camera connected with grabber card and interfaced with computer, Windows OS

automatically detects the device. Different cameras can be used to capture video images

but we have preferred CCD camera because:

It has good resolution

The probability of noise introduction to image is low

It provides good quality image

In this project the video images are captured at specified rate of 30 frames per

second with video resolution of 640 X 480.Then display device are used to display

visual output of the system that can be used for lane departure warning. The lane

departure warning can be passed using buzzer in the form of audio.

3.2.3Approach to lane detection and departure warning

Video images of lane are captured using camera and identification of certain

features of lanes from other objects in the image is performed. Using gray scaling

followed by thresholding we can get a binary image containing different objects and

lines. But the road lanes have certain features that can be used to identify it from

different objects and fake lines that are in the captured road image. These features of

the lane are that:

Lane markings have similar width throughout the video

The distance that separates left lane and right lane is constant

The lane markings are thin and long line in the video image

Lane markings are continuous line or road lane is consisted of a number of

candidate lane continuous lane markings

The lane markings have unique shape while objects and fake lines have not

possessed it.

The lanes are parallel to each other

The lane which have many elements of lines in parallel if it is not continuous

By detecting the midline between left and right lane it’s possible to pass lane departure

warning. This is deeply explained in the algorithm development part.

26

3.2.4 Algorithm development

This part explains the algorithms developed to detect and track lanes. It assumes

road lane as long linear line or a combination of a number small lane marking. It also

includes the algorithm of lane departure warning to warn the driver if the car is

departed out of the lane. The system is developed using feature based technique with

taking the identified feature into an account. The algorithm is developed by using

MATLAB video and image processing tool box. MATLAB provides support to access

serial port (also called as COM port), parallel port (also called as printer port or LPT

port) of a PC or USB port. Depending on this we have assumed to connect video

camera to capture video and pass it PC for video and image processing part. The

algorithm developed for lane detection and departure warning is explained as follows.

27

No

No

Yes

No

Yes

Fig3.8 flow chart of lane detection and departure warning

Video input

Gray scaling

Frame grabbing Video frame

Filtering Histogram

equalization Image scaling

Edge detection

Hough transform

Hough peaks Hough lines Is it lane

marking?

Delete

Compute mid line Compare mid lines

Is

comparison

is greater

than TH Display

Buzzer

28

3.2.4.1 Algorithm description

i. Video input

It’s the material that is used to capture video images of road lane for processing

and lane extraction. This material could be different cameras that can be interfaced with

processor using USB port. The camera resolution to capture the images should be

640X480. It captures images at 30 frames per second.

ii. Frame grabbing

Video is nothing but it’s the sequence of images taken successively at certain

frame per second. Frame grabbing is used to separate video to sequence of images. In

this project the video camera captures 30 images of road lanes within one second. This

depends on the computation time needed. During this selection memory and other

resources should be taken into consideration. Then the images will be ready for

processing.

iii. Gray scaling

The captured images are RGB (true color images). RGB images are not suitable

for processing as they are composed of red, green and green colors. To help in

processing the images in each frame should be converted to gray scale or intensity

image. A gray scale digital image is an image in which the value of each pixel is a

single sample. Each sample is stored using 8 for 256 intensities. It has value ranging

from black to white pixel.

iv. Filtering

Noises may be introduced to images during capturing. Filters are used in order to

remove these noises from image. This will help in further processing by improving

image quality. This can be done using median filter.

v. Histogram equalization

Due to various reason the intensity of light at each pixel in a single band of the

electromagnetic spectrum (e.g. visible light) there difference in pixel intensity will

occur in the captured image. But this may cause some loss of properties of image. So

applying histogram equalization distributes intensity values over the image and the

pixel intensity in the image will became similar. This has certain use when shadows

29

and foggy or rainy weather condition exists. It can be implemented using histeq

(histogram equalization) function.

vi. Image scaling

It is the act of brightening the captured image. This has special wide influence on

lane detection during rainy and foggy conditions. Image can be scaled by multiplying it

by specific number. This means that each pixel value in the image will be multiplied by

this specific number.

vii. Edge detection

In this project the first step in the lane feature extraction is the edge detection of

images. Edge detection can be implemented by using canny edge detector. Canny edge

detection uses two thresholds values and scans all over the image using Gaussian

masks. It out puts the binary image with detected edges. It can be computed by using

edge function.

viii. Hough transform

After detecting the edge of object in the image we have applied Hough transform

to detect lines in the image [math works]. Hough transform detect lines in the image

using parametric equation of line.

rho = x*cos(theta) + y*sin(theta)……………………………………….3.1

The variable rho is the distance from the origin to the line along a vector

perpendicular to the line. Theta is the angle between the x-axis and this vector. The

Hough function generates a parameter space matrix whose rows and columns

correspond to these rho and theta values respectively. An array [rho] [theta] is used to

count how many pixels belong to the line through Hough Transform. The value of rho

should be greater than 10, and the value of theta should be between 30 and 150 to avoid

the detection of fake lane markings.

ix. Hough peaks

Hough peaks function works with Hough function to detect potential lines in the

image. Hough transform yields a parametric space with rho as row and theta as column.

Hough peaks finds the peak value in this parametric space. Peak value in the Hough

parametric space corresponds to potential line in the image.

30

x. Hough lines

After identifying peaks in the Hough parametric space, it’s evident that the line

exists at there. The line will be plotted using Hough line function. That means Hough

line is used to convert the polar coordinates to Cartesian coordinate, because parametric

space is composed on polar coordinates rho and theta.

xi. Determine lane markings

The output of Hough lines is that every line on the road lane will be detected.

The detected line may be the desired road lane or it may be fake lane marking. So the

lane marking should be searched depending on the lane feature described under our

approach to lane detection and departure warning part. This involves

Sort the candidate lines by their position from left to right

Choose the lane candidate which has biggest count number as the lane

marking.

Around each line cluster, choose the lane candidate which is longest as

the lane marking in real scene

Delete the fake lane marking candidates

xii. Determine mid line

Determining midline of the detected left and right lane is used in determining

whether the car is moving out of lane. The mid line is computed and stored for each

frame in the video. Then the current mid line is compared against the midline of the

previous frame. If the deviation exceeds threshold value, the lane departure warning

will be issued.

31

3.2.5 Algorithm for lane detection and departure warning

system

Here is the algorithm used for lane detection and departure warning. It can be

implemented using MATLAB programming language. MATLAB has built in function

to carry out some parts of this algorithm.

1. Create video input object

2. Set properties of video input object

3. While ( acquired frames < infinity)

i. Access these frames

ii. for (present position=initial position: final position)

• separate color values in RGB images

• compute luminance value Y=0.299R+0.587G+0.114B

• assign value to present position

initial position is 1st row, 1

st column of an image resolution

final position is last row, last column of an image resolution

iii. for (present position=initial position: final position)

• scan all surrounding elements

• sort all the values

• calculate median of it

assign the value of median to present position

initial position is 2nd

row, 2nd

column of an image resolution

final position is the row and column before last row and column

iv. for (present position=initial position: final position)

• create two arrays H and T of length 256 to zero values

• scan every pixels and increment H if pixel x has intensity p

H[p]=H[p]+1

• form cumulative histogram Hc and store it in H

• determine array T from stored value T[p]=(G-1/MN)*H[p]

G=gray level, M=row, N= column

• change gray level of image to q=T[p]

initial position is 1st row and 1

st column of image

final position is last row and last column of image

32

v. for (present position=initial position: final position)

• multiply each pixel values by constant number k

• store output at the present position

vi. for(present position=initial position: final position)

• convolve image with Gaussian filter

• convolve smoothed with derivative of Gaussian

• find local maxima

• carry out hysteresis thresholding

vii. for (present position=initial position: final position)

• quantize the parameter space p

[amin….amax][bmin…bmax]=accumulator array

• for each edge points (x, y)

for ( a= amin; a<amax: a++)

b=-xa+y;

(p[a][b])++

• find local maxima in accumulator array

• map each maxima to image space

x=pcosβ

y=psinβ

viii. for (present position=initial position: final position)

• determine statics of road lane

• for each candidate

compare candidate against the determined statics

Delete candidates which fail to meet the determined static

• impose over the image

• calculate end points of candidate

• determine midline of left and right candidate

• compare midline of current frame to previous frame

• issue warning

33

3.3 Safe distance warning

3.3.1 Introduction

Almost continuously, someone in the world dies from a traffic accident; many

more suffer injuries. During this time collision occurs when the driver loses his or her

attention of in front obstacle. When the speed of obstacle increases or decreases it may

enter into the safe distance of the car that makes it vulnerable for collision. Sometimes

collision may occur when pedestrians are suddenly crossing the road. Moreover, the

economic losses caused by traffic accidents are reaching as high as Everest hill. The

frontal car collision can be avoided if safe distance is maintained between a car and any

object in front of it. A considerable number of accidents can be avoided by recognizing

a hazard in sufficient time and making appropriate driving maneuvers. Such actions can

be achieved by warning signals to the driver within sufficient time. Suitable sensors are

required to identify hazardous situation and notify the driver within sufficient amount

of time. In this project laser radar sensors are used to determine the range and speed of

upcoming obstacle. Safe distance warning is based on time gap between the car and the

obstacle. There are safe distance warnings at two points before collision occurs. The

first warning is the visual warning, while the second warning is warning in audio.

3.3.2 Safe distance determination

Different safe distance determination algorithm has been proposed by different

car manufacturers depending on different criteria and conditions. The safe distance

warning system uses different sensors and processors to detect the presence of objects

within the safe distance of moving car. Infrared laser radar, video camera and wheel

speed sensors are used to capture road information and the properties of own car. These

are different condition that must be considered before passing safe distance warning.

Time gap to collision

Relative speed between the car and obstacles

Range between the car and the obstacle

34

By considering the above criteria the system configuration for safe distance

determination and safe distance warning looks like the following.

Fig 3.9 block diagram of safe distance warning system

Infrared laser radar operates in the same way as any other radars work. But the

main difference is that it uses infrared radiation rather than radio waves. It’s highly

efficient in detecting any moving and non moving obstacles. Its uses light detection and

ranging (LIDAR) principle. LIDAR-based systems offer advantages such as small size,

simple assembly, and relatively low cost. The laser-based systems measure the distance

of obstacle traveling in front of car by using triangulation or time-of-flight

measurement where the laser pulses are reflected by the preceding obstacles. The

sensor uses a high-power laser diode to transmit infrared light pulses with a wavelength

in the range of 850 nm to 950 nm. A high-speed PIN photodiode or avalanche photo

diode receives the light reflected by the preceding vehicle and determines the distance

and the speed relative to the obstacle in front. The distance to one or more objects can

be determined by the delay time of the reflection.

R= Cxt/2 ………………………………………………………………….3.2

Where C is speed of light, t is the time delay

The speed of obstacle is determined by taking several samples of ranges in one second

and taking average value of it.

3.3.3 Algorithm development

There are various safe distance warning algorithm that proposed previously. In

this project there are two safe distance warning methodologies. The first safe distance is

determined by the maximum viewing distance of video camera. The selected CCD

camera has the maximum viewing distance up to 80m. When an obstacle enters to this

distance the video camera starts to capture an object and the first safe distance warning

is passed in visual form for the driver. So the driver will get an attention of in front

obstacle early. The second safe distance is the critical safe distance. We used

Infrared laser

radar ADC Host PC

Buzze

r

35

HONDA’s algorithm to determine this critical safe distance. Critical safe distance is the

minimum distance that the driver uses to avoid collision.

D=2.2 Vrel+6.2…………………………………………………….. …….. 3.3

Where Vrel is relative velocity determined by infrared laser radar. If an obstacle enters

to this critical distance the second warning is passed by using audible warning. Here is

the algorithm used for the second warning:

1. Create input object

2. Set properties of input object

3. Read serial port

4. Determine safe distance

5. Compare safe distance to range

6. Pass audio warning

36

3.3.4 Object detection and tracking for visual safe distance

warning

3.3.4.1 Introduction

Different types of collision may be avoided if the car can detect and track any

obstacle in front of it. Object detection is the process of detecting objects in the video

captured by CCD camera using digital image processing techniques. Detecting an

object is not enough to avoid collision; it should be tracked until it moves out of the

region of interest. Object tracking is the process of locating moving objects in time

using a camera. An algorithm analyses the video frames and outputs the location of

moving targets within the video frame. Different video capturing cameras can be used

for this application, but CCD camera is robust to different conditions that arises noise.

The object detection is implemented using back ground subtraction and an

object is tracked using blob analysis. For the accurate object detection model based

technique is most accurate as certain feature of objects can be lost due noises, but

model based technique requires complex computation and large memory when using

MATLAB. The noises that can cause feature degradation can be removed using noise

removal methods and an object detected and tracked using back ground subtraction and

blob analysis. Tracking is based on the centroid of the detected object.

3.3.4.2 System configuration

This part of the project describes the block diagram of object detection and

tracking. To detect and track an obstacle in front of the car there should be a material

that tells whether an object is present or not. This can be done by the camera mounted

on the car. The camera calibration can affect the maximum viewing distance of video

camera and how the object is to be detected. The camera viewing direction should

horizontal and it should be mounted on the middle center of the car. The system is

composed of different input devices and output devices as well as processor. The input

devices are used to capture any existing obstacle. This is mainly done by camera. The

output device used is the display unit that displays the detected object. PC processor

can be used to process the video images being MATLAB is installed on it.

37

Fig3.10 block diagram of object detection and tracking.

3.3.4.3 Approach to object detection and tracking.

Object detection and tracking is the act of detecting and locating obstacles

captured by video camera. With camera installed on car it’s possible to capture any in

front obstacles up to 80m. Using digital image processing techniques the captured

object is differentiated from back ground objects. Object detection and tracking has

four main parts depending on their techniques. They are

Noise removal

Segmentation

Feature extraction

Blob analysis

a. Noise removal

It’s the first part of object detection and tracking which removes noises from

video frames. In real time condition there are various noises that can affect the quality

of image. The occlusion due to shadows affects the quality of images to a great extent.

Noises that are introduced to images can be removed by using median filtering.

b. Segmentation

After noise is removed from video frames image segmentation is used to separate

back ground from foreground objects. This is used to remove losses due to the presence

of shadows. But shadows can affect the efficiency of object detection. The image

segmentation is performed by using back ground subtraction. There are many types of

back ground subtraction that can be used to remove back ground from fore ground

object. The back ground subtraction is frame differencing followed by thresholding.

The system compares consecutive video frames and excludes pixels that exist in both

frames. After comparison there is some effect back ground pixels, which should be

removed by thresholding.

Camera Processor Display unit

38

c. Feature extraction

After thresholding foreground objects and some effects from shadow exists.

These effects are removed by using sobel edge detector. Sobel filtering is an effective

technique used to extract the boundaries in a given image. The Sobel operator

calculates the gradient of the image intensity at each point, giving the direction of the

largest possible increase from light to dark and the rate of change in that direction. For

every pixel in the image sobel operator determine intensity variation along x and y

direction. Every object has specific feature like color or shape. In this project object is

extracted using its color feature. The output of sobel displays intensity edge of objects.

This intensity is nothing but the objects color feature. The pixel values from the first hit

of the intensity values from top, bottom, left and right are stored. By using this

dimension values a rectangular bounding box is plotted within the limits of the values

produced.

d. Blob analysis

After object edge is identified by bounding box, the centroid of detected object is

determined. This helps in tracking of detected objects. Blob tracking is used for object

tracking. Tracking process will take the object pixel coordinates as input from object

detection process. Here centroid positions are plotted in each frame. After centroid

calculation it is easy to plot the location of the moving objects with frame to frame.

39

3.3.4.4 Algorithm of object detection and tracking

1. Create video input object

2. Set properties of video input object

3. While ( acquired frames < infinity)

i. Access these frames

ii. for (present position=initial position: final position)

• separate color values in RGB images

• compute luminance value Y=0.299R+0.587G+0.114B

• assign value to present position

initial position is 1st row, 1

st column of an image resolution

final position is last row, last column of an image resolution

iii. for (present position=initial position: final position)

• scan all surrounding elements

• sort all the values

• calculate median of it

assign the value of median to present position

initial position is 2nd

row, 2nd

column of an image resolution

final position is the row and column before last row and column

iv. for (present position=initial position: final position)

• select first frame(N) as back ground

• subtract first frame from second frame

• compare the output to threshold

• change back ground to second frame(N+1)

assign the output of threshold to current position

initial position is second frame

final position is the last frame

v. for (present position=initial position: final position)

• convolve frame N with sobel filter

• determine bounding box

• determine centroid

40

Here is the flow chart for object detection and tracking

Fig 3.11 flow chart for object detection and tracking

41

Chapter four

Results and Discussion

4.1 Implementation and results

So far we have been designing the system and developing an algorithm for its

implementation. In this project lane detection is done on video frames capture by frame

grabber. The visual safe distance warning uses the object detection and tracking to give

the initial visual warning for the driver. The prototype of Object detection and tracking

is implemented using web camera and on the sequence of video frames. Lane detection

system uses statics of the lane to identify road lane from other fake lines. The use of

Hough transform is greatly applied in the detection of road lane. The obstacle detection

is implemented on the sequence of video frames using back ground subtraction method.

The output of the system is described as following. The core of object detection and

tracking in this project is the back ground subtraction method. The output of frame

grabber is changed to gray level image followed by back ground subtraction method.

Back ground subtraction is used to differentiate an object from back ground

using frame difference followed by thresholding. It compares two consecutive frames

and cancels the common pixels. After common pixels are cancelled, it’s only intensity

of foreground object that remains.

Fig 4.1 images from video frame sequence.

42

Fig 4.2 images from consecutive video sequence.

Fig4.3 out put of back ground subtraction for images of fig4.1

Fig 4.4 output of back ground subtraction for images from fig4.2

From the above figures back ground subtraction suppresses back ground from

foreground objects. These two men are moving from frame to frame, so certain feature

that represents them is not suppressed as they are moving. Back ground subtraction

identifies objects not only by their move’s from frame to frame but also by their color

feature. White objects are easily identifiable. But with camera moving at high speed

and at the presence of high shadows certain feature of objects may not be completely

identified. This was the major problem in this project. But this problem can be

suppressed by sobel filter after back ground subtraction. Using the results from back

ground subtraction foreground objects are identified from back ground. Here, after back

43

ground subtraction what remaining is only the intensity of foreground object. Using

the intensity of fore ground object it’s possible to determine the bounding box that

determines the maximum tip of the intensity. After the bounding box is determined the

object is detected by calculating the centroid of bounding box. Object can be tracked by

using this centroid. The output of object detection and tracking using intensity or color

variation is shown below

Fig4.5 output of object detection and tracking from video sequence

Fig 4.6 a) output of object detection and tracking from video sequence

a) b) c)

d) e)

fig 4.6 b) the lorry detected on the road lane.

44

The obstacle detection and tracking using bounding box has importance for the

drivers to clearly identify the obstacle. The problem with this intensity based object

detection exists when shadows and occlusion exists. Sometimes when an obstacles

reaches close to each other, their bounding box my merge toward each other and they

may be detected as a single object with centroid pointing at the mid distance between

them.

The lane detection sub system applies region based processing to identify the

region that looks like road lanes. We have used image statics to identify lanes from

other fake lanes. The system uses region based processing and region based statics to

identify feature of lanes on the image from video sequence. To efficiently detect lane

markings we have applied Hough transform method. It uses the output of canny edge

detection as an input and creates accumulator array to compare between lines that exists

in the image. Road lanes are usually the longest line and thin line that exists in the

image. They have the maximum peaks in the accumulator array created by Hough

transform

Fig 4.7 input image of road lane

Using canny edge detection and region based enhancement, the input image changes to

the form suitable for region based processing. The system uses different noise removal

methodology along with image enhancement to extract specific features within the road

lanes. The output of image enhancement is feed into Hough transform algorithm. The

accumulator array created by Hough transform for the above image looks

45

Fig 4.8 accumulator array from hough transform

In this image there is the point of highest intensity, which shows the presence of

road lane. So Hough peaks shows the presence of lane at that point. The detected lane

looks like:

Fig 4.9 image result with fake and road lane detected

From this image it’s clear that every region that looks like lane is detected.

Generally road lane has highest peak in the accumulator array. By considering only

highest peak i.e relatively higher peak it’s possible to suppress some fake lanes. Finally

the road lane detected by hough transform looks as shown in fig below.

46

Fig 4.10 road lane and fake lane detected by hough transform

Now we have road lane detected by hough transform. But still there exists some

fake lanes due to real time effect of video camera and lighting condition. So we have

applied feature based differentiation to identify the two lanes on which the car is

moving. This is done by using low level image processing described below

Fig 4.11 binary image found after noise removal

The above figure shows the conversion of output image from Hough transform to

binary image. This conversion has importance minimizing computation in feature

extraction by dividing the image into two parts. This image has only two pixels which

are 0 and 1. The region that looks like lane has highest intensity of pixel. There is also a

region that has some value of pixels but less than pixel value of lane. These are the

region of fake lanes. The region of interest is the region that contains only visible road

lanes. Using boundary extraction, those points which are not in the region of interests

are removed. The output of this boundary extraction is:

47

Fig 4.12 output image after boundary extraction

Due to shadows are not removed by only boundary extraction the user defined

function is used to delete the effect of the shadow and other occlusion. The user defined

function uses the specific characteristics of road lane to delete these fake lanes. These

special properties of road lane are identified under chapter three.

Fig 4.13 lanes detected using Hough transform followed by region based statics.

48

4.2 Problems encountered

During the work of this project we have passed through problems by struggling.

Video processing and image processing application are carried out on DSP processor or

image processor with high image processing speed. But due to scarcity of this

equipment, sometimes it’s possible to use personal computers with high memory.

Image processing generally requires a large memory of operation. The unavailability of

personal computer with suitable operating speed and RAM has cut our aim to apply

model based technique to detection and warning system developed. The main problems

with such a computer are the out of memory error. This error normally occurs when an

image slot memory over laps with virtual memory of the PC. Generally the problems

encountered in this project are the unavailability of required devices for implementation

of the project.

49

Chapter 5

Conclusion

In this project feature based Hough transform has been employed to be detect

road lanes and to warn driver if the car is going to move out of lane. Simulation of road

lane detection has been done using MATLAB and the results are discussed. Fake lanes

detected due to different noises and occlusion is removed boundary statics determined

from properties of road lane. The detection of road lane helps in determining lane

departure and there by pass warning. So the system can pass intelligent decision to

warn the driver when the car moves out of its lane. The second part of the intelligent

system based car collision prevention is the safe distance warning. There are two safe

distances at which the driver should be warned. The first warning is passed in visual

form. The second part is passed in audible form. The developed system uses infrared

laser radar and video sensor to detect the obstacle in front of the car. An algorithm is

developed for safe distance warning if an obstacle enters to the safe distance of the car.

The developed system uses background subtraction for object detection and centroid

calculation for object tracking in visual warning system.

The interface between infrared laser radar and PC serial port is designed and it

circuit diagram has been drawn. This interface has wider importance for the critical safe

distance warning. Finally the lane detection is implemented on various video frames.

The visual warning which is passed by using object detection and tracking is

implemented using webcam on the real time objects.

Recommendations

Intelligent system based car collision prevention is the wide research areas. It

cannot meet totally under short period of time as collision avoidance system requires

integration of different sensors and a bundle of high memory capacity processors. Each

sensor needs a different interfacing .But this can be simplified by using ECU which has

sensor interfacing unit. To make the system more robust and stand alone C++ portable

program can be used. In this project simulation of lane detection and tracking as well as

visual warning using object detection and tracking is proposed and done. As this

project has wider importance and complex computation we recommend critical safe

distance braking and intersection collision avoidance.

50

Appendixes

A.MATLAB code for lane detection i=imread('lane.jpg');

b=rgb2gray(i);

e=medfilt2(b, [3 3]);

f=histeq(e);

c=b*1.2;

l=200;

s=c>l;

d=bwareaopen(s,750);

imshow(d);

hold on;

[B L]=bwboundaries(d,'noholes')

for k=1:length(B)

boundary=Bk;

hk=plot(boundary(:,1),boundary(:,2),'g','linewidth',4);

end

stats=regionprops(L,'majoraxislength','minoraxis','orientation');

lanes=findlanes(B,h,stats);

hold off;

imshow(i);

hold on;

for k=1:length(lanes)

plot(lanesk(:,2),lanesk(:,1),'g','linewidth',4);

end

hold off

B. User defined function for deleting fake lines

function lanes=findlanes(B,h,stats)

l=0;

global metric;

for k = 1:length(B)

metric = stats(k).MajorAxisLength/stats(k).MinorAxisLength;

testlane(k);

end

function testlane(k)

if metric > 5 && all(Bk(:,2)>100)

if metric > 5 && all(Bk(:,1)>150) && stats(k).MinorAxisLength<20 &&

abs(stats(k).Orientation)>30 ...

&& abs(stats(k).Orientation)<90 && all(Bk(:,2)<350)

l=l+1;

lanes(l,:)=Bk;

else

delete(h(k))

end

end

end

C.MATLAB code for object detection from video sequence

function d = trackfff(video)

51

if ischar(video)

avi = aviread(video);

pixels = double(cat(4,avi(1:2:20).cdata))/255;

clear avi

else

pixels = double(cat(4,video1:2:20))/255;

clear video

end

nFrames = size(pixels,4);

for f = 1:nFrames

pixel(:,:,f) = (rgb2gray(pixels(:,:,:,f)));

end

rows=480;

cols=320;

nrames=f;

for l = 2:nrames

d(:,:,l)=(abs(pixel(:,:,l)-pixel(:,:,l-1)));

k=d(:,:,l);

bw(:,:,l) = im2bw(k, .2);

bw1=bwlabel(bw(:,:,l));

imshow(pixel(:,:,l))

hold on

cou=1;

for h=1:rows

for w=1:cols

if(bw(h,w,l)>0.5)

toplen = h;

if (cou == 1)

tpln=toplen;

end

cou=cou+1;

break

end

end

end

disp(toplen);

coun=1;

for w=1:cols

for h=1:rows

if(bw(h,w,l)>0.5)

leftsi = w;

if (coun == 1)

lftln=leftsi;

coun=coun+1;

end

break

end

end

end

52

disp(leftsi);

disp(lftln);

widh=leftsi-lftln;

heig=toplen-tpln;

widt=widh/2;

disp(widt);

heit=heig/2;

with=lftln+widt;

heth=tpln+heit;

wth(l)=with;

hth(l)=heth;

disp(heit);

disp(widh);

disp(heig);

rectangle('Position',[lftln tpln widh heig],'EdgeColor','r','linewidth',4);

disp(with);

disp(heth);

plot(with,heth, 'r*');

drawnow;

saveas(gcf,strcat('result',num2str(l)),'jpg');

hold off

end;

D.MATLAB code for object detection using webcam vid = videoinput('winvideo',1);

set(vid,'TriggerRepeat',Inf);

vid.FrameGrabInterval = 5;

vid_src = getselectedsource(vid);

set(vid_src,'Tag','objectdetection');

figure;

start(vid)

pause(2);

while(vid.FramesAcquired<=100) % Stop after 1000 frames

data = getdata(vid,2);

diff_im = imabsdiff(data(:,:,:,1),data(:,:,:,2));

diff = rgb2gray(diff_im);

diff_bw = im2bw(diff,0.2);

bw2 = imfill(diff_bw,'holes');

s = regionprops(bw2, 'centroid');

cd = s.Centroid;

centroids = cat(1, s.Centroid);

imshow(data(:,:,:,2));

hold(imgca,'on');

plot(imgca,centroids(:,1),centroids(:,2),'g*');

hold on;

rectangle('Position',[cd 60 33],'LineWidth',2,'EdgeColor','b');

hold(imgca,'off');

end

stop(vid)

53

References

1. Rajitha tv (2008) , ‘Object tracking from video sequence’ .award of degree of

master of technology in computer science, Cochin university of science and

technology

2. Arnab Roy, sanket shinde and kyoung-don kong an, ‘approach for efficient real

time moving object detection’. department of computer science, Watson school

of engineering and applied science, state university of new York at Binghamton

3. Yifei wang, naim dahnoun, and alin achim(2009), ‘a novel lane feature

extraction algorithm based on digital interpolation’17th

European signal

processing conference(EUSIPCO 2009) Glasgow, Scotland

4. Claudio rosito jung and Christian Roberto kelber(2004), ‘lane departure

warning system based on a linear parabolic lane model’ IEEE intelligent

vehicle sysmposium university of parma, parma, Italy

5. Chew chin lee(2007), ‘moving object detection at night’, award of degree of

master of engineering university technology of Malaysia

6. Muhammad azwan nasirudin and mohd rizal arshald, ‘a feature based lane

detection system using hough transform method’, usm robotics research

group(URRG), school of electrical and electronic engineering Malaysia

7. Rafael C.Gonzalez, Richard E. Woods, Steven L.Eddins (2004), ‘Digital Image

Processing Using MATLAB’, New jersey :Pearson Prentice Hall

8. Masayuki Yokoyama and Tomaso Poggio(2005), ‘A Contour-Based Moving

Object Detection and Tracking’

9. B.Ugur Toreyina et. Al(2006), ‘Moving object detection in wavelet compressed

video’

10. Alan M. McIvor(2000), ‘ Background Subtraction Techniques’

11. Yue wang, dinggang and eam khwang teoh, ‘lane detection using catmull-rom

spline’, school of electrical engineering nanyang technological university ,

IEEE International Conference on Intelligent Vehicles 1998

12. Yue wang, dinggang and eam khwang teoh , ‘lane detection using b-snake’,

school of electrical engineering nanyang technological university

13. Abhishek rawat and tulip kumar toppo, ‘feature based object tracking using

PTZ camera’

54

Documents

Intelligent System Based Car Collision Prevention