1-s2.0-S0926580510001147-main

8/12/2019 1-s2.0-S0926580510001147-main

1/9

Detection of large-scale concrete columns for automated bridge inspection

Z. Zhu , S. German, I. Brilakis

School of Civil and Environmental Engineering, Georgia Institute of Technology, Atlanta, GA. 30332, USA

a b s t r a c ta r t i c l e i n f o

Article history:

Accepted 29 July 2010

Keywords:

Concrete columns

Automatic identication systems

Images

Information technology

Bridge inspection

There are over 600,000 bridges in the US, and not all of them can be inspected and maintained within the

specied time frame. This is because manually inspecting bridges is a time-consuming and costly task, and

some state Departments of Transportation (DOT) cannot afford the essential costs and manpower. In this

paper, a novel method that can detect large-scale bridge concrete columns is proposed for the purpose ofeventually creating an automated bridge condition assessment system. The method employs image stitching

techniques (feature detection and matching, image afne transformation and blending) to combine images

containing different segments of one column into a single image. Following that, bridge columns are detected

by locating their boundaries and classifying the material within each boundary in the stitched image.

Preliminary test results of 114 concrete bridge columns stitched from 373 close-up, partial images of the

columns indicate that the method can correctly detect 89.7% of these elements, and thus, the viability of the

application of this research.

Published by Elsevier B.V.

1. Introduction

According to data from the Bureau of Transportation Statistics up

to 2008, there arecurrently more than 600,000 highway bridges in theUSA, around 25% of which are rated as either structurally decient,

functionally obsolete or both[1]. At current spending levels (roughly

$10.5 billion per year),it is estimated that the annual gap of 1.9 billion

dollars is required to reduce this percentage to near zero by 2024 [2].

The problem could be alleviated with the introduction of automated

inspection to replace current manual practices, which costs local

Departments of Transportation (DOT) millions of dollars every year

[3]. Each of these bridges requires inspection at regular intervals

(usually not exceeding two years with few exceptions) to determine

their physical and functional conditions and ensure that they still

satisfy present service requirements [4]. Routine inspections are

widely adopted and are carried out manually by certied bridge

inspectors, following the National Bridge Inspection Standards by the

Federal Highway Administration (FHWA) [5] and the Manual on

Bridge Evaluation by the American Association of State Highway and

Transportation Ofcials (AASHTO)[4]. Inspection results are mainly

based on the inspectors' observations and visual assessment and

represent the condition state information on bridge elements (e.g.

columns, abutments, decks/slabs, and girders). This information is

input into a bridge management system, such as PONTIS[6], to help

transportation agencies make system-wide prioritization decisions in

allocating limited construction maintenance and rehabilitating

resources.

Several limitations of manual inspections have been identied in

previous research studies. First, the results from manual inspectionsare subjective and not always reliable [7,8]. Second, manually

collecting bridge inspection data is costly and time-consuming

[9,10]. Third, a number of safety risks are associated with inspections

since inspectors often work at high heights or in heavy trafc zones

[11]. Also, the requirement of experienced inspectors in bridge

inspection poses a challenge for the construction industry, which is

now facing the pressing shortage of experienced and highly trained

inspection personnel[12].

In order to overcome these limitations, it is necessary to create

automated bridge inspection toolkits that can considerably reduce the

eld work required for inspectors and automatically produce bridge

inspection reports. Toward this objective, several methods have been

created. For example, Jauregui et al. utilized QuickTime Virtual Reality

and panoramic image-creation tools for recording observations and

measurements during bridge eld inspections [9]. Jaselskis et al.

investigated the use of laser scanning to acquire and process bridge

as-built data[13]. Moreover, Abdel-Qader et al. focused on detecting

the presence of bridge surface cracks [14], and they found that the fast

Haar transform had more accurate crack detection results than three

other edge detection techniques (Canny, Sobel and Fourier Trans-

form)[15]. Although these methods help inspectors nd defects on

bridges, the detected defects cannot be used directly to rate bridge

conditions. This is because the measurements of these defects are of

little value unless they can be spatially correlated with the members

on which they lay. For example, a diagonal (shear) crack versus a

horizontal (exural) crack indicates a completely different type of

Automation in Construction 19 (2010) 10471055

Corresponding author. Tel.: +1 404 385 1276.

E-mail addresses: [email protected](Z. Zhu),[email protected](S. German),

[email protected](I. Brilakis).

0926-5805/$ see front matter. Published by Elsevier B.V.

doi:10.1016/j.autcon.2010.07.016

Contents lists available at ScienceDirect

Automation in Construction

j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / a u t c o n
http://dx.doi.org/10.1016/j.autcon.2010.07.016http://dx.doi.org/10.1016/j.autcon.2010.07.016http://dx.doi.org/10.1016/j.autcon.2010.07.016mailto:[email protected]:[email protected]:[email protected]://dx.doi.org/10.1016/j.autcon.2010.07.016http://www.sciencedirect.com/science/journal/09265805http://www.sciencedirect.com/science/journal/09265805http://dx.doi.org/10.1016/j.autcon.2010.07.016mailto:[email protected]:[email protected]:[email protected]://dx.doi.org/10.1016/j.autcon.2010.07.016

8/12/2019 1-s2.0-S0926580510001147-main

2/9

damage on a column. Therefore, bridge elements also have to be

detected, so that the defects can then be spatially correlated with the

corresponding elements to produce relative measurements for

automated bridge inspection. The detection of bridge elements is

always the rst step for automated bridge assessment. This will be

followed by defects detection (e.g. crack detection) and defects

impact assessment.

Existing work in structural element detection cannot capture both

the bridge columns and the detailed defect information on theirsurfaces from a single image, which is necessary in order to accurately

correlate the defect and member spatial properties. This is because

bridge elements are usually large-scale. For the purpose of this study,

a large-scale structural element is dened as the element that cannot

be completely t into an image when the smallest defects of interest

occupy at least 200 pixels on the element surfaces [16]. In order to

detect these elements and retrieve their dimensions, existing

methods require structural elements to be entirely visible in the

eld of view (FOV). This forces users to fully zoom-out with their

cameras or stand far away when capturing large-scale columns,

considering that the FOV of a typical compact camera is limited to

50(H)35(V), when the human's FOV can reach 200(H)135(V)

[17]. On the other hand, capturing defect data often requires close up

images as to allow for detailed information retrieval concerning the

extent and impact of each defect. As a result, when columns are fully

visible, the defect image data on these elements is invisible ( Fig. 1).

This paper presents an automated method for the detection of

large-scale concrete bridge columns captured in multiple close-range

images taken for the purpose of defects detection and assessment. To

the authors' knowledge, there are no existing computer vision

techniques that can be used directly for concrete column recognition

other than those presented in the paper. As a result, current existing

defect detection methods can only be used to detect surface defects

after structural element surfaces are manually extracted from image

background[18].

The proposed method starts with the SIFT detector to nd the

features that are invariant to afne transformation in each image. The

SIFT-based descriptors exhibit high matching accuracy, which makes

them out-perform other local descriptors [19,20]. Although thecomputational complexity is associated with the SIFT detector, it is

not a critical issue in this study, since no real-time requirements are

imposed for routine bridge inspection. The detected features are then

matched to calculateimage transformationmatrices, which areused to

combine the imagescontaining different segments of one columninto

one single image. Following that, the concrete column surfaces in the

image are identied.An edge mapis produced from thestitched image

using the Canny edge detector, and the long vertical lines in the map

are retrieved using the Hough Transform. The bounding rectangle for

each pair of long vertical lines is created. If the resulting rectangle

resembles the shape of a column (i.e. the rectangle's width is smaller

than its length) and the color/texture in the rectangle are matched to

concrete based on the classication result of an articial neural

network, the region between the pair of lines is assumed to be a

concrete column surface, belonging to a concrete bridge column.

The method was implemented using Microsoft Visual C++. Adatabase of close-range concrete bridge images (373 total) was used

to test the method. The test results were compared with manual

detection results to nd the detection precision and recall of the

method. Precision measures how many detected columns are

correctly identied, while recall measures how many actual columns

are correctly identied. Based on the test results, the average

precision ratio and recall of the method are 89.7% and 84.3%,

respectively, which validate that most large-scale concrete bridge

columns can be correctly detected using the method proposed in this

paper.

2. Background

2.1. Manual bridge routine inspection

Highway bridge inspections are carried out manually by certied

inspectors following the established standards and manuals (i.e.

National Bridge Inspection Standards and AASHTO Manual on Bridge

Evaluation). Before going onto a bridge, the inspectors prepare

sketches and note templates for references throughout the inspection

[21]. During inspection, they record the actual bridge conditions by

observing existing defects that lay on primary bridge components,

such as decks, exterior beams and piers. The defects to be inspected

include different types of cracks (e.g. exural, shear, vertical and bond

cracks), loss of cover and spalling, and corrosion and eforescence.

This information is used to rate bridgeconditions. So far, there aretwo

rating systems that can be adopted. The FHWA Recording and Coding

Guide [22] denes one system which uses the National Bridge

Inventory (NBI) zero to nine scale for rating a bridge decksuperstructure and substructure. The second system, the PONTIS

rating system, uses a set of three to ve condition states to describe

the conditionof approximately 160 bridge elements, such as columns,

girders, slabs and trusses [23]. Both rating systems are built on

qualitative denitions. For example, in the NBI rating system, a bridge

is rated at Fair Condition (5) when all of its primary components are

sound but may have minor cracking, spalling, or scour [22].

Fig. 1.The loss of detailed defect information when tting a large-scale column in the view.

1048 Z. Zhu et al. / Automation in Construction 19 (2010) 10471055

8/12/2019 1-s2.0-S0926580510001147-main

3/9

Such manual inspections have several inherent limitations. First,

they are inefcient. Reviewing previous reports, preparing inspection

plans, and collecting and analyzing eld data requires a large amount

of inspector hours [9,20], which has proven costly over time. In

Tennessee, in 2006 and 2007, the annual cost of bridgeinspection was

$7.6 million; considering the Tennessee Department of Transporta-

tion lost $22million in Federal bridgefundsin thelast twoyears, some

bridge inspections and maintenance projects have to be pushed back

[24]. In addition, the results from manual inspections are subjectiveand highly variable. An on-site survey conducted by FHWA Non-

Destructive Evaluation Center (NDEVC) indicated that manual

inspections were completed with signicant variability[7,8]. Finally,

the current guidelines need considerable updating with regard to

health monitoring and non-destructive testing techniques, as well as

in regards to theproper integration of that data into the ratingprocess

[25].

2.2. Recent research efforts towards automated bridge routine inspection

Researchers have proposed many methods regarding the aim of

automated bridge inspection, most of which focus on as-built bridge

data collection and defect detection. Laser scanning and photogram-

metry are two remote sensing techniques recently used to collect

bridge as-built data. Laser scanning can create accurate and detailed

3D range images. Most laser scanners used in construction adopt the

time-of-ight principle. Other scanners adopting triangulation or

structured-light are seldom applied for scanning large structures due

to their limited scanning range of meters and/or safety constraints

[26,27]. In the time-of-ight principle, a pulse of laser light is emitted

to probe the object in the scene, and a time-of-ight laser range nder

at the heart of this type of scanner, nds the distance of a surface by

timing the round-trip time of the pulse of light. So far, this technique

has been used in several bridge survey applications [13,28]. Aside

from laser scanning, photogrammetry has also been used for this

purpose, due to the affordable and portable equipment involved[29].

It relies solely on the present light energy to retrieve bridge geometry

data, which is implicitly contained in images captured from different

perspectives. Examples of using photogrammetry to document bridgeas-built conditions can be found in the work of Jauregui et al. [9,30].

As for defect detection, many methods have been created to locate

cracks in images. They use image processing techniques, such as

wavelet transforms and segmentation, to extract crack points from

the image background. The effectiveness of these methods has been

veried in real concrete structures. For example, Abdel-Qader et al.

proposed a principal component analysis (PCA) based algorithm for

detecting unsupervised bridge cracks [14]. Also, they compared the

effectiveness of four edge detection techniques (Canny edge detector,

Sobel edge detector, Fourier transform and fast Haar transform) in

detecting cracks on concrete bridges[15]. According to the detection

results for 50 images, they found that the fast Haar transform was the

most reliable in crack detection. Sinha and Fieguth took advantage of

the linear property of crack features and introduced two crackdetectors for identifying crack pieces [31]. Yu et al. used a graphical

search to retrieve an integral crack based on manually provided

endpoints[32].

2.3. Structural element detection

Automating the collection of bridge as-built data and the detection

of surface defects is not entirely sufcient to provide a comprehensive

bridge inspection. In order to produce bridge ratings, the impact of

existing defects on bridge elements needs to be further evaluated,

which rst requires the detected defects to be spatially correlated

with the member on which they lay in order that then, the

interpretation of the specic defects according to the specic member

can take place. For this reason, bridge elements need to be detected

(Fig. 2). Although element detection is only one step in solving the

whole defects interpretation problem, it is always the rst step, and

critical to evaluate existing defects on bridge elements later. The

purpose of this concentrates on the automated detection of bridge

elements while keeping as much of thedetaileddefects information as

possible.As for the subsequent steps precise crack detection and the

structural analysis of existing cracks at the level of structural

engineering domains they are beyond the discussion of this paper.

In general, object detection is de

ned as locating a desired objectin a scene[33]. It can be performed in range data (3D) or visual data

(2D). The methods of detecting objects in range data rely on the

object's shape features, such as surfaces or contours, since range data

are insensitive to illuminations [34]. The shape information can be

described globally or locally. The global shape descriptors [35,36]

capture all shape characteristics of a desired object in range data.

Therefore, they are more discriminative, but less robust, than the local

shape descriptors[37,38]in cluttered scenes, where the range data of

the desired object is easily occluded [39]. Both types of methods

(global descriptors and local descriptors) are computationally

complex, especially when the number of search objects and the

number of scanned points is increased[40]. In order to overcome this

limitation, Bosche and Haas suggested representing a 3D CAD model

using range points so that the matching can be performed at the point

level. However, this method requires the model to be registered rst

in the laser scanner's spherical coordinate frame [41].

Objectdetection methods in 2D visualdata canbe further classied

as: 1) scale/afne-invariant feature-based, 2) color/texture-based and

3) shape-based. 2D Scale/afne-invariant feature-based methods use

image scale/afne-invariant features extracted by feature detectors,

such as SIFT[42], to locate an object in the image. Such methods have

shownto be powerful indetecting a specic object. Recently,the useof

3D local shape descriptors was investigated for object class detection.

For example, Ruiz-Correa et al. created numeric signature to form a

constellation of components to quantify 3D object surface geometry,

and then relied on a set of symbolic signatures to encode the spatial

conguration of these component for object category detection [43].

However, the concept of these 3D local shape descriptors cannot be

applied in object category detection in 2D visual data, especially instructural element detection. This is because structural elements

always have a uniform material on their surfaces, which makes it

difcult to retrieve enough 2D scale-invariant features to characterize

the geometry of whole 2D element surfaces. Moreover, structural

Fig. 2.Automated bridge inspection.

1049Z. Zhu et al. / Automation in Construction 19 (2010) 10471055
http://localhost/var/www/apps/conversion/tmp/scratch_5/image%20of%20Fig.%E0%B2%80

8/12/2019 1-s2.0-S0926580510001147-main

4/9

elements, although geometrically simple, are characterized by large

topographical variations (e.g. aspect ratio) and therefore no simple

scale/afne transformation can be used to characterize them[44].

Color/texture based methods rely on the objects' interior color and

texturevalues to perform detection.Neto et al.observed that thecolor and

texture values for most materials (e.g. concrete and steel) in an image do

not change signicantly[45].Based on this observation, material regions

in an image can be identied by checking their color and texture values

[16,46]. Moreover, two dimensions of the detected regions (MCD andPMCD) for concrete and steel can be used to determine whether the type

of structural element of the region is a beam or a column[47]. However,

when one element (e.g. a concrete column) is connected to another

structural element with the same material (e.g. a concrete beam) this

method regards them as one single element instead of two separate

elements and produces erroneous results[44].

Edge information is another indicator for structural element

detection. Edges are dened as the areas where image intensities

change signicantly (e.g. discontinuities in object surface orienta-

tion). Shape-based methods make use of this information to identify

object boundaries by analyzing the distribution of edge points using

the Hough transform [48], covariance matrices [49] or principle

component analysis[50]. Such methods can detect thin and stick-like

objects whose color and texture are easily corrupted by image

background[51], but the sole reliance on edge information renders

these methods inadequate in complex scenes[52].

The authors have created a building column detection method using

both boundary and material information in an image/video [44]. The

method considered that each building concrete column has a pair of long

vertical lines and the textureand color patterns on its surface are uniform.

Based on these two visual characteristics, the method located building

concrete columns in images or videos by identifying their boundaries

using edge detection and Hough transform techniques and recognizing

concrete material using the concept ofmaterial signatures[46].

The proposed method, like other column detection methods,

requires concrete columns to be entirely visible in the FOV of a

camera. This limitation has no signicant adverse effects in most

construction applications. Take automated project progress monitor-

ing for an example. A user can always adjust a camera (changingposition, shooting direction, etc.) to an appropriate view, so that

concrete columns in the view can be separated each other, and the

concrete construction project progress can be automated by auto-

matically counting the number of concrete columns that have been

built at the site. However, in automated inspection, the method is not

always applicable, since it is difcult to capture the detailed defect

information on column surfaces in one image, when a user has to

adjust the FOV of a camera to t an entire column. It is common that

the user takes multiple pictures to capture the defects that are located

at different areas of one column and then, combine all defect

information contained in these pictures for the sake of rating the

column.

2.4. Image stitching

Image stitching is the combination of multiple images into one high-

resolution image based on their overlapping areas. It is extensively

investigated in the eld of computer vision, and so far there are several

commercial applications available, such as PhotoStitch[53]and REALVIZ

Stitcher[54]. In general, the process of stitching any two images can be

divided into two steps: 1) image alignment, the analysis of image

orientation information (e.g. translation and rotation), and 2) image

blending, the smooth combination of the color values of two images in

their overlapping areas[55].

Image alignment is performed using direct-based methods or feature-

based methods. Direct-based methods are also called pixel-based

methods [56]. These methods search for a transformation matrix that

can minimize the sum of absolute differences between the overlapping

pixels of two images. For example, Baker and Matthews generalized the

LucasKanade algorithm to implement the work of image alignment [57].

Similar work can also be found in the work of Agarwala et al. [58]and

Shum and Szeliski [59]. All of these methods provide accurate orientation

information since they use all available image data; however, a close

initialization is always required[60]. Also, they have a limited range of

convergence; thus,they work in matchingsequentialframes of a video but

fail too often to be useful in photo matching [59]. Feature-based methods

overcome these limitations. They determine image orientations byautomatically detecting and matching the same features that appear in

multiple images. Features can be points, line segments, or regions.

Matching the features in images is solved through the use of multiple

randomized kd trees, a spill tree or a hierarchical k-means tree[61].

Feature-basedmethodsare remarkably robust, since they caneven detect

and match the features in images that differ in scale [56].

The process of blending two images is nothing more than

calculating their weighted average. The weights can be simply xed

at 0.5 for both images or determined based on their distance

maps[62]. The combination of pixel values using a weighted average

always produces inaccuracies, such as double exposure, visible seams,

color distortion and contrast loss [63]. In order to address these

problems, image blending is performed in multiple resolutions or

bands. For example, Grundland et al. represented images with

Laplacian pyramids and then the blending was performed at each

level of the pyramids independently [63]. Allen et al. convolved an

image with a xed Gaussian kernel multiple times to generate

increasingly smoothed versions[64]. The differences between these

versions represent the image at different frequency levels. The

blending of two images was performed at the frequency level of

each image rst, and the ultimate result was collected from the

summation of the blending results at each level.

Brown and Lowe presented a novel system that can fully automate

these two steps[60]. In their work, the feature detector [42]is rst

used to extract and match the features that are invariant to image

scale and afne distortion in each image. Once these features are

matched, the locations ( x1 ;y1 T

; x2 ;y2 T

:::::: xn ;yn T) of these fea-

tures are then retrieved. For any two images, the afne transform

matrix between the two images can be calculated using Equation 1where x11 ;y

11

T; x12 ;y

12

T::: x1n ;y

1n

Tare feature locations in the rst

image, x21 ;y21

T; x22 ;y

22

T::: x2n ;y

2n

Tare the matched feature locations

in the second image and matrix A12 = a11 ; a12 ; a13

a21 ; a22 ; a23

is the afne

transform matrix that can transfer the locations of any image pixel in

the rst image to the second one. Thus, these two images can be

combined into one image with the afne transform matrix, and the

combination can be performed iteratively until one composite image

is created from all individual images. Their approach is robust for

automatically stitching most bridge column images based on the

following test results. For this reason, the method proposed in this

paper directly adopts their approach for image stitching. The main

contribution of this paper is to create a framework that combines the

method of image stitching with the method of normal concretecolumn detection for detecting large-scale columns captured in

multiple images.

x11 ;y11 ; 1

x12 ;y

22 ; 1

::::::::::::

x1n ;y1n ; 1

0BBBBB@

1CCCCCA

a11 ; a21a12 ; a22a13 ; a23

0@

1A =

x21 ;y21

x22 ;y

22

:::; :::

x2n ;y2n

0BBBBB@

1CCCCCA

1

3. Proposed method

A novel method of detecting a large-scale bridge column captured

in multiple close range images is proposed in order to overcome the


8/12/2019 1-s2.0-S0926580510001147-main

5/9

aforementioned limitations characteristic of existing automated

bridge inspection procedures (Fig. 3). First, the pictures depicting

different segments of a single bridge column are stitched together.

Then, using the Canny operator and the Hough Transform, the column

boundaries (i.e. a pair of long vertical lines) are located. At this point,

the visual features of the image region within the column boundaries

are input into an articial neural network which performs the

concrete material recognition and results in the complete detection

of the concrete column.For the purpose of the method discussed in this paper, the rst

step, image stitching, is performed based on the work of Brown and

Lowe[60].Prior to performing any image stitching, the image noise,

which is always produced duringthe process of capturing an image, is

reduced to minimize its effect on all subsequent image processing

operations. A 5 pixel by 5 pixel median lter is selected for this

purpose, due to its ability to signicantly improve the image signal to

noise ratio [65]. Then, by matching theresulting detected features,the

image transformation matrices can be calculated. These matrices are

used to form one single image from the multiple images containing

different segments of the column. Specically, the RANSAC algorithm

[66] is used to nd feature matching outliers and inliers, while the

bundle adjustment algorithm [67] is adopted to estimate the

transformation matrix elements for all images. Image stitching errors

such as visible image edges and radial distortions are removed

through a multi-band blending algorithm.

Once a stitched image is retrieved, the column needs to be located

within this image. The work of concrete column detection consists of

identifying long near-vertical lines for each column surface as well as

determining whether or not the material contained between two lines

consists of concrete. In order to detect the column boundaries, the

Canny operator is used to produce an edge map, a binary image

composed of edge points (marked as white) and non-edge points

(marked as black), distinguishing the edge of the column from its

surroundings in the rest of the image. The Hough transform is then

used to analyze the distribution of edge points and retrieve long

vertical line information in the edge map. Each retrieved vertical line

is compared to its neighboring vertical lines. If two vertical lines have

similar size, they are regarded as a pair. This comparison is performed

iteratively, till no further pair can be found. In order to eliminate line

pairs not corresponding to actual columns, the following two visual

characteristics of columns are used to narrow the retrieved line pairs.

First, the aspect ratio (length/width) of the rectangle formed by each

long vertical line pair should be at least smaller than one. Second,concrete column surface is supposed to have the uniform texture and

color patterns. Specically, a bounding rectangle for each pair of

vertical lines is rst constructed based on their Cartesian coordinates

(x,y), and the aspect ratio (length/width) of each bounding rectangle

is calculated in order to determineif theboundingrectangle is in fact a

column. Considering the large-scale bridge columns investigated in

this research, the ratio can be assumed to be on the order of six or

seven.

Following the column boundary identication, it is necessary to

determine the properties of the material within the boundary. Thus,

theimageregion containedin theidentied boundary is retrieved and

its visual features (e.g. normalized red, normalized green and

normalized blue) are calculated. The columns visible in the stitched

images can be detected using the authors' previous work in detecting

concrete columns [39]; however, in order to further reduce human

intervention in dening appropriate threshold values for concrete

material recognition, an articial neuralnetwork is used to classify the

type of material within the boundary. The network design of the

network is composed of three layers (input, hidden and output). In

the input layer, the network receives the visual features and transfers

them to the hidden layer. For each neuron node in the hidden layer,

individual material classication boundary lines in the feature space

are created. Then, these material classication boundary lines are

combined together using an ANDoperation in theoutput layer to form

a concrete material classication model to determine whether the

region is composed of concrete or not. The classier mainly has two

important parameters. One is the number of neuron-nodes in the

hidden layer. Technically, the larger the number of neuron nodes in

the hidden layer, the larger the number of classication boundariescan be found, which results in a more precise classication

approximation. Another important parameter is related to the

classier training. The network is trained using supervised learning

(back-propagation). The output values of the network are manually

specied as 1 for concrete samples and1 for non-concrete samples.

During the training, the network automatically adjusts the neurons'

weights to minimize identication errors. Although it is possible to

repeat the training process millions of time to retrieve minimum

identication errors, long training will result in an overtraining

problem, which makes the network only memorize the peculiar

patterns of the training samples but fail to generalize them. The exact

training repetition number is determined experimentally from the

authors' previous work [68]. Specically, 160 concrete and non-

concrete samples are retrieved from real construction site images andthey are divided into two sets: one for the classier training and the

other for the classier testing. The training is repeated from 100 to

400,000 times, and the corresponding training and testing accuracy

rates (i.e. the number of corrected classied samples over the number

of samples) are recorded. It is found that the accuracy for training

samples gradually increases with the increase of repetition training

times, however, the accuracy for test samples increases after reaching

a certain limit (9000) and then drops signicantly due to the over-

tting problem[68].

When the material contained in the region is identied as

concrete, one column surface is assumed to be detected. Thus, the

detection of the concrete column is performed with the combination

of the identication of the column's boundaries and the classication

of the concrete material.Fig. 3. Method overview.


8/12/2019 1-s2.0-S0926580510001147-main

6/9

8/12/2019 1-s2.0-S0926580510001147-main

7/9

detected concrete column surfaces (TP+ FP), and the number of

actual concrete column surfaces (TP+ FN) in each image. This

information is used to calculate the detection precision and recall.

Table 1 illustrates the average detection precision (89.7%) and

recall (84.3%) for 114 concrete bridge column images stitched from

373 original images. The precision and recall for each single stitched

example are not used because they do not truly reect the overall

performance of the method. Large-scale concrete column images are

usually taken close to the columns in order to capture the detailed

defect information. Only one column (one or two column surfaces) isvisible in each example of the stitched images. As a result, the

prevision andrecall ratios vary signicantly case by case. Forexample,

suppose there aretwo visible columnsurfacesin onestitchedimage. If

both surfaces can be detected, the precision and recall for the image

are 100% and 100%. However, if one column surface is not correctly

detected, the precision will drop to 50%. Similarly, if one column

surface fails to be detected at all, the recall will drop to 50%.

5. Conclusion and future research

The limited FOV of a camera makes it not always possible to have a

large-scale bridge element t entirely in a view, and at the same time

capture all its surface defects of interest in detail. This limitation can

be overcome with the recent development of image stitchingtechniques in the eld of computer vision. The paper presents a

novel method of detecting large-scale concrete bridge columns from

digital images. The method starts with image stitching techniques to

combine multiple images containing different FOVs for one bridge

column into one stitched image. Then, the Canny operator and the

Hough Transform are used to recognize the pair of long vertical lines

of a concrete column in the stitched image. The visual features of the

region of the line pair are then presented to an articial neural

network to indicate whether the region contains concrete or not, so

that a large-scale concrete column surface can be detected.

The work presented in this paper provides a quantitative, quick,

and thus less costly means to assist in current bridge inspection

practices. Real concrete bridge images were used to validate the work

presented in this paper. The test results indicate that a large-scale

Fig. 5.Examples of image stitching and column detection: (a) the succeeded detection; (b) the failed detection.

Table 1

Precision and recall for 114 concrete bridge column images.

Summarization

Total TP 166

Total FP 19

Total FN 31

Average precision (TP/(TP +FP)) 89.7%

Average recall (TP/(TP + FN)) 84.3%


8/12/2019 1-s2.0-S0926580510001147-main

8/9

bridge concrete column captured in multiple images can be detected

in the stitched image with high detection precision (89.7%) and recall

(84.3%). Future work will focus on two aspects: the validity of the

alteration to the proposed method which addresses curve edges, and

the detection and spatial correlation of the defects on the column

surfaces with the surfaces themselves.

Acknowledgement

Theauthors would like to thank theNational Science Foundationfor its

generous support of this project through Grants #0933931 and 0904109.

References

[1] Bureau of Transportation Statistics (BTS), Condition of U.S. highway bridges:19902008, Sept. 10, 2009 bhttp://www.bts.govN.

[2] R. Kirk, W. Mallett, Highway bridges: conditions and the federal/state role, 2007CRS report for congress, b http://www.fas.org/sgp/crs/homesec/RL34127.pdfN(April, 2010).

[3] USA Today, State-by-state update on bridge conditions, 2008 bhttp://www.usatoday.com/news/nation/2008-07-25-bridge-chart_N.htmN(April, 2010).

[4] American Association of State and Highway Transportation Ofcials (AASHTO),Manual for condition evaluation of bridges, 2nd Ed, 2001 Washington, D.C.

[5] Federal Highway Administration (FHWA), National bridge inspection standards,Federal Register 69 (239) (2004) 7441974439.

[6] PONTIS, Pontis Product Support Website, 2009 bhttp://www.pontis.bakerpro-jects.com/N(Sept. 9, 2009).

[7] B. Phares, G. Washer, D. Rolander, B. Graybeal, M. Moore, Routine highway bridgeinspection condition documentation accuracy and reliability, Journal of BridgeEngineering 9 (4) (2004) 403413.

[8] M. Moore, B. Phares, B. Graybeal, D. Rolander, G.A. Washer, Reliability of visualinspection for highway bridges, Volume I: Final report and, Volume II:Appendices, U.S. Department Of Transportation, Washington, D.C, 2001 FHWA-RD-01-020(021),.

[9] D. Jauregui, K. White, Implementation of virtual reality in routine bridgeinspection, Journal of Transportation Research Record 1827 (2003) 2935.

[10] T. Koglin, Movable bridgeengineering,John Wiley& Sons,Inc978-0-471-41960-0, 2007.[11] NJDOT, Bridge Inspection Work Zone, Setup Guide, 2009 bhttp://www.state.nj.us/

transportation/eng/structeval/pdf/InspectionWorkZoneSetup.pdfN(Sept. 19, 2009).[12] 12, Transportation Pooled Fund Program (TPFP, TPF Studies: Support of the

Transportation Curriculum Coordination Council, Federal Highway Agencies,2009 last visit: http://www.pooledfund.org/projectdetails.asp?id=435&sta-tus=4, (Sept. 3, 2009).

[13] E.J. Jaselskis, Z. Gao, R. Walters, Improving transportation projects using laserscanning, Journal of Construction Engineering and Management, ASCE 131 (3)(2005) 377384.

[14] I. Abdel-Qader, S. Pashaie-Rad, O. Abudayyeh, S. Yehia, PCA-based algorithm forunsupervised bridge crack detection, Advances in Engineering Software 37 (12)(2006) 771778.

[15] I. Abdel-Qader, O. Abudayyeh, M. Kelly, Analysis of edge-detection techniques forcrack identication in bridges, Journal of Computing in Civil Engineering 17 (4)(2003) 255263.

[16] I. Brilakis, L. Soibelman, Y. Shinagawa, Construction site image retrieval basedon material cluster recognition, Journal of Advanced Engineering Informatics,Volume 20, Issue 4, Elsevier Science, 2006, pp. 443 452, October 2006.

[17] B. Freeman, F. Durand, Panoramas, 2006 Class lecture notes, bhttp://www.groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/15_pano_6.pdfN(Sept. 22, 2009).

[18] Z. Zhu, I. Brilakis, Detecting air pockets for architectural concrete qualityassessment using visual sensing special issue in sensors in construction andinfrastructure management, Journal of Information Technology in Construction13 (2008) 86102.

[19] K.Kolajczyk,C. Schmid,A performanceevaluation oflocaldescriptors,IEEETransactionson Pattern Analysis and Machine Intelligence 27 (2005) 16151630.

[20] J. Bauer, N. Sunderhauf, P. Protzel, Comparing several implementations of tworecently published feature detectors, Proc. of the International Conference onIntelligent and Autonomous Systems, Toulouse, France, 2007.

[21] [21] Sunkpho, J. (2001). A Framework for Developing Field Inspection SupportSystems. Ph.D Disseration, Department of Civil and Environmental Engineering,Carnegie Mellon University.

[22] Federal Highway Administration (FWHA), Recording and coding guide for thestructure inventory and appraisal of the nation's bridges, U.S. Department ofTransportation, Washington, D.C., 1995

[23] P. Thompson, R. Shepard, AASHTO Commonly Recognized Bridge Elements,Successful Applications and Lessons Learned National Workshop on Com-monly Recognized Measures for Maintenance, 2000 bhttp://www.pdth.com/images/coreelem.pdfN(Sept. 21, 2009).

[24] E. Wasserman, Are Tennessee Bridges Safe?, 2007 bhttp://www.tdot.state.tn.us/documents/Wasserman_editorial.pdfN(Sep. 21, 2009).

[25] Federal Highway Administration (FHWA), Questions and Answers on theNational Bridge Inspection Standards 23 CFR 650 subpart C, U.S. Department

of Transportation, 2009 bhttp://www.fhwa.dot.gov/Bridge/nbis/N(Sept. 29,2009).

[26] T. Ditto, Acquisition of three-dimensional surfaces using holographic primaryobjectives, 2007 last visit: http://www.drillamerica.com/PDF/3D.pdf.

[27] G. Sansoni, M. Trebeschi, F. Docchio, State-of-the-art and applications of 3Dimaging sensors in industry, cultural heritage, medicine, and criminal investiga-tion, Sensors 9 (2009) 568601.

[28] P. Tang, B. Akinci, Extracting surveying goals from point clouds to supportconstruction and infrastructure inspection, Construction Research Congress 2009,Seattle, USA, 2009.

[29] Z. Zhu, I. Brilakis, Comparison of civil infrastructure optical-based spatial data

acquisition techniques, Journal of Computing in Civil Engineering 23 (3) (2009)170177.[30] D. Jauregui, Y. Tian, R. Jiang, Photogrammetry Applications in Routine Bridge

Inspection and Historic Bridge Documentation, 2006 Report NM04STR-01,bh t t p : / / w w w . nm s h t d . s t a t e . n m . us / u p l o a d / i m a g e s / Re s e a r c h / NM04STR01PhotogrammetryApplicationsHistoricBridges_2006.pdf N (Sept.18, 2009).

[31] S. Sinha, P. Fieguth,Automated detectionof cracks in buried concretepipe images,Automation In Construction 15 (1) (2006) 5872.

[32] S.-N. Yu, J.-H. Jang, C.-S. Han, Auto inspection system using a mobile robot fordetecting concrete cracks in a tunnel, Automation in Construction 16 (3) (2007)255261.

[33] F. Ge, T. Liu, S. Wang, J. Stahl, Template-based object detection through partialshape matching and boundary verication, International Journal of SignalProcessing 4 (1) (2008) 148157.

[34] S. Stiene, K. Lingemann, A. Nuchter, J. Hertzberg, Contour-based object detection inrange images, Data Processing, Visualization, and Transmission, 2006, pp. 168175.

[35] D. Wang, J. Zhang, H. Wong, Y. Li, 3D model retrieval based on multi-shellextended Gaussian image, Lecture Notes in Computer Science: Advances in Visual

Information Systems, 2007, pp. 426437.[36] M. Kazhdan, T. Funkhouser, S. Rusinkiewicz, Rotation invariant spherical

harmonic representation of 3D Shape descriptors, Symposium on GeometryProcessing, 2003, pp. 156164.

[37] Y. Shan, H. Sawhney, B. Matei, R. Kumar, Shapeme histogram projection andmatching for partial object recognition, IEEE Transactions on Pattern Analysis andMachine Intelligence 28 (4) (2006) 568577.

[38] D. Huber, A. Kapuria, R. Donamukkala, M. Hebert, Parts-based 3D objectrecognition, Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol.2, 2004, pp. 8289.

[39] A. Patterson IV, P. Mordohai, K. Daniilidis, Object detection from large-scale 3Ddatasets using bottom-up and top-down descriptors, European Conference onComputer Vision, 2008, pp. 553566.

[40] F. Bosche, C. Haas, Towards automated retrieval of 3D designed data in 3D senseddata,Proceedings of the2007 ASCEInternational Workshopon Computing in CivilEngineering. Pittsburg, PA, 2007.

[41] F. Bosche, C. Haas, Automated retrieval of 3D CAD model objects in constructionrange images, Automation in Construction 17 (2008) 499512.

[42] D. Lowe, Distinctive image features from scale-invariant keypoints, InternationalJournal of Computer Vision 60 (2) (2004) 91110.

[43] S. Ruiz-Correa, L.G. Shapiro, M. Meila, G. Berson, M.L. Cunningham, R.W. Sze,Symbolic signatures for deformable shapes, IEEE Transactions on Pattern Analysisand Machine Intelligence 28 (1) (2006) 7590.

[44] Zhu, Z. and Brilakis, I. (2009) "Concrete Column Recognition in Images andVideos", Journal of Computing in Civil Engineering, American Society of CivilEngineers (in review).

[45] J. Neto, D. Arditi, M. Evens, Using colors to detect structural components indigital pictures, Computer Aided Civil and Infrastructure Engineering 17(2002) 6176.

[46] I. Brilakis, L. Soibelman, Y. Shinagawa, Material-based construction site imageretrieval, Journal of Computing in Civil Engineering, Volume 19, Issue 4, AmericanSociety of Civil Engineers, October 2005, pp. 341355.

[47] I. Brilakis, L. Soibelman, Shape-based retrieval of construction site photographs,Journal of Computing in Civil Engineering, Volume 22, Issue 1, American SocietyofCivil Engineers, January/February 2008, pp. 1420.

[48] L.A. Fernandes, M.M. Oliveira, Real-time line detection through an improvedhough transform voting scheme, Pattern Recognition 41 (1) (2008) 299314.

[49] D. Guru, B. Shekar, P. Nagabhushan, A simple and robust line detectionalgorithm based on small eigenvalue analysis, Pattern Recognition Letters 25(2004) 113.

[50] Y. Lee, H. Koo, C. Jeong, A straight line detection using principle componentanalysis, Pattern Recognition Letter 27 (2006) 17441754.

[51] P. David, D. DeMenthone, Object recognition in high clutter images using linefeatures, Proceedings of the 10th IEEE International Conference on ComputerVision, Vol. 2, Issue 17-21, 2005, pp. 15811588.

[52] C. Nikolaos, E. Anagnostopoulos, I. Anagnostopoulos, V. Loumos, E. Kayafas, Alicense plate-recognition algorithm for intelligent transportation systemapplications, IEEE Transactions on Intelligent Transportation Systems 7 (3)(2006) 377392.

[53] Canon PhotoStitch Sofware, bhttp://www.kenrockwell.com/canon/photostitch.htmN(Nov. 02, 2009).

[54] Realviz, bhttp://www.realviz.comN(Nov. 02, 2009).[55] Y. Chuang, Image Stitching,Nov. 01, 2009 LectureNotes, bhttp://www.csie.ntu.edu.tw/

~cyy/courses/vfx/05spring/lectures/handouts/lec06_stitching_4up.pdfN.[56] R. Szeliski, Image Alignment and Stitching: A Tutorial, 2005 Technical Report,

MSR-TR-2004-92.

http://www.bts.gov/http://www.fas.org/sgp/crs/homesec/RL34127.pdfhttp://www.usatoday.com/news/nation/2008-07-25-bridge-chart_N.htmhttp://www.usatoday.com/news/nation/2008-07-25-bridge-chart_N.htmhttp://www.pontis.bakerprojects.com/http://www.pontis.bakerprojects.com/http://www.state.nj.us/transportation/eng/structeval/pdf/InspectionWorkZoneSetup.pdfhttp://www.state.nj.us/transportation/eng/structeval/pdf/InspectionWorkZoneSetup.pdfhttp://www.pooledfund.org/projectdetails.asp?id=435&status=4http://www.pooledfund.org/projectdetails.asp?id=435&status=4http://www.groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/15_pano_6.pdfhttp://www.groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/15_pano_6.pdfhttp://www.pdth.com/images/coreelem.pdfhttp://www.pdth.com/images/coreelem.pdfhttp://www.tdot.state.tn.us/documents/Wasserman_editorial.pdfhttp://www.tdot.state.tn.us/documents/Wasserman_editorial.pdfhttp://www.fhwa.dot.gov/Bridge/nbis/http://www.drillamerica.com/PDF/3D.pdfhttp://www.nmshtd.state.nm.us/upload/images/Research/NM04STR01PhotogrammetryApplicationsHistoricBridges_2006.pdfhttp://www.nmshtd.state.nm.us/upload/images/Research/NM04STR01PhotogrammetryApplicationsHistoricBridges_2006.pdfhttp://www.kenrockwell.com/canon/photostitch.htmhttp://www.kenrockwell.com/canon/photostitch.htmhttp://www.realviz.com/http://www.csie.ntu.edu.tw/~cyy/courses/vfx/05spring/lectures/handouts/lec06_stitching_4up.pdfhttp://www.csie.ntu.edu.tw/~cyy/courses/vfx/05spring/lectures/handouts/lec06_stitching_4up.pdfhttp://www.csie.ntu.edu.tw/~cyy/courses/vfx/05spring/lectures/handouts/lec06_stitching_4up.pdfhttp://www.csie.ntu.edu.tw/~cyy/courses/vfx/05spring/lectures/handouts/lec06_stitching_4up.pdfhttp://www.realviz.com/http://www.kenrockwell.com/canon/photostitch.htmhttp://www.kenrockwell.com/canon/photostitch.htmhttp://www.nmshtd.state.nm.us/upload/images/Research/NM04STR01PhotogrammetryApplicationsHistoricBridges_2006.pdfhttp://www.nmshtd.state.nm.us/upload/images/Research/NM04STR01PhotogrammetryApplicationsHistoricBridges_2006.pdfhttp://www.drillamerica.com/PDF/3D.pdfhttp://www.fhwa.dot.gov/Bridge/nbis/http://www.tdot.state.tn.us/documents/Wasserman_editorial.pdfhttp://www.tdot.state.tn.us/documents/Wasserman_editorial.pdfhttp://www.pdth.com/images/coreelem.pdfhttp://www.pdth.com/images/coreelem.pdfhttp://www.groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/15_pano_6.pdfhttp://www.groups.csail.mit.edu/graphics/classes/CompPhoto06/html/lecturenotes/15_pano_6.pdfhttp://www.pooledfund.org/projectdetails.asp?id=435&status=4http://www.pooledfund.org/projectdetails.asp?id=435&status=4http://www.state.nj.us/transportation/eng/structeval/pdf/InspectionWorkZoneSetup.pdfhttp://www.state.nj.us/transportation/eng/structeval/pdf/InspectionWorkZoneSetup.pdfhttp://www.pontis.bakerprojects.com/http://www.pontis.bakerprojects.com/http://www.usatoday.com/news/nation/2008-07-25-bridge-chart_N.htmhttp://www.usatoday.com/news/nation/2008-07-25-bridge-chart_N.htmhttp://www.fas.org/sgp/crs/homesec/RL34127.pdfhttp://www.bts.gov/

8/12/2019 1-s2.0-S0926580510001147-main

9/9

[57] S.Baker, I. Matthews, Lucas-Kanade 20 years on:a unifyingframework:Part 1: thequantity approximated, the warp update rule, and the gradient descentapproximation, International Journal of Computer Vision 56 (3) (2004) 221255.

[58] A. Agarwala, et al., Interactive digital photomontage, ACM Transactions onGraphics 23 (3) (2004) 292300.

[59] H.-Y. Shum, R. Szeliski, Construction of panoramic mosaics with global and localalignment, International Journal of Computer Vision 36 (2) (2000) 101130.

[60] M. Brown, D. Lowe, Automaticpanoramic image stitchingusing invariant features,International Journal of Computer Vision 74 (1) (2007) 5973.

[61] M. Golparvar-Fard, F. Pea-Mora, S. Savarese, Application of D4AR a 4-dimensional augmented reality model for automating construction progress

monitoring data collection, processing and communication, ITcon Vol. 14, SpecialIssue Next Generation Construction IT: Technology Foresight, Future Studies,Roadmapping, and Scenario Planning, 2009, pp. 129153.

[62] A. Efros, Image Blending, 2008 Lecture Notes, bhttp://www.graphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdfN(Nov. 05, 2009).

[63] M. Grundland, R. Vohra, G. Williams, N. Dodgson, Nonlinear multi-resolutionimage blending, MG&V 15 (3) (2006) 381390 (Feb. 2006).

[64] C. Allene, J. Pons, R. Keriven, Seamless image-based texture atlases using multi-band blending, Pattern Recognition, 2008. ICPR 2008. 19th InternationalConference on, 2008.

[65] M. Juneja, R. Mohana, An improved adaptive medialtering method for impulsenoise detection, International Journal of Recent Trends in Engineering 1 (1)(2009) 274278.

[66] M. Fischler, R. Bolles, Random sample consensus: a paradigm for model ttingwith application to image analysis and automated cartography, Communicationsof the ACM 24 (1981) 381395.

[67] W. Triggs, P. McLauchlan, R. Hartley, A. Fitzgibbon, Bundle adjustment: a modernsynthesis, Vision Algorithms: Theory and Practice, number 1883 in LNCS, 1999,pp. 298373.

[68] Zhu, Z. and Brilakis, I. (accepted for publication) "Intelligent Learning andDetection of Concrete Regions in Construction SiteImages"Proceedingsof the 6th

International Conference on Innovation in Architecture, Engineering andConstruction (AEC), June 911, 2010, University Park, PA.[69] G. Bradski, A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV

Library, O'Reilly Media, Inc0596516134, 2008.[70] FANN, Fast Articial Neural Network Library, 2009 b http://www.leenissen.dk/

fann/N(April 10, 2009).[71] T. Wickens, Elementary Signal Detection Theory, Oxford University Press, New

York0195092503, 2002.

http://www.graphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdfhttp://www.graphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdfhttp://www.leenissen.dk/fann/http://www.leenissen.dk/fann/http://www.leenissen.dk/fann/http://www.leenissen.dk/fann/http://www.graphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdfhttp://www.graphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf

Documents

1-s2.0-S0926580510001147-main