An Image Mosaicking Method for Tunnel Lining
Inspection
Cuong Nguyen Kim Mien Truong University of Civil Engineering, Phu Yen, Vietnam
Email: [email protected]
Kei Kawamura Graduate School of Science Technology Innovation of Yamaguchi University, Ube, Yamaguchi, Japan
Email: [email protected]
Masando Shiozaki Sumitomo Mitsui Construction Co., Ltd., Tokyo, Japan
Email: [email protected]
Amir Tarighat Department of Civil Engineering, Shahid Rajaee Teacher Training University, Iran
Email: [email protected]
Abstract—The paper proposes an image mosaicking method
for tunnel lining inspection. The conventional methods only
used the cost value of the pixel being processed based on
similarity metric to estimate an image-matching location. To
improve the image-mosaicking efficiency, the curvatures of
the cost curve at candidate matching points are adapted.
Moreover, experimental results for an actual tunnel
demonstrate that the curvature measurement can select the
precise matching points accurately for assisting defect
inspection.
Index Terms—tunnel lining inspection, image mosaicking,
curvature metrics
I. INTRODUCTION
Many concrete infrastructure structures, including
buildings, bridges, and especially tunnels, have been in
use for over 50 years now deteriorating. Therefore, to
improve the efficiency of tunnel inspection work and
obtain more accurate results, the authors propose a
mobile tunnel inspection system (MOTIS) that is capable
of image acquisition with reliable resolution and high
running speed. Its advantages are simple structure,
inexpensive cost, and quality images. Subsequently, an
image stitching technique is applied to generate an
overall view of the tunnel wall at high resolution.
Moreover, many researchers have considered
automated crack detection on the tunnel lining surface
such as Yu et al. [1] and Zhang et al. [2]. However, they
did not detail about tunnel-image mosaicking generation
to assist visual inspection.
Manuscript received March 7, 2019; revised January 8, 2020.
Many concrete infrastructure structures, including
buildings, bridges, and especially tunnels, have been in
use for over 50 years now deteriorating. Therefore, to
improve the efficiency of tunnel inspection work and
obtain more accurate results, the authors propose a
mobile tunnel inspection system (MOTIS) that is capable
of image acquisition with reliable resolution and high
running speed. Its advantages are simple structure,
inexpensive cost, and quality images. Subsequently, an
image stitching technique is applied to generate an
overall view of the tunnel wall at high resolution.
II. EXISTING METHOD
In our previous papers [3], [4], and [5], the authors
reported a video acquisition system and a procedure of
the image stitching for the entire tunnel lining as follows:
A. Video Acquisition System
The video images of the entire tunnel lining surface are
continuously scanned by a video acquisition system
assembled on a car. Fig. 1 shows the system consisting of
six digital video cameras (from V1 to V6), three
illuminators attached to a steel framework shaped to work
within a half of the cross-section of the tunnel, and a car.
This device is able to slide from the side to the top of the
car so that the full tunnel lining surface can be captured
by several passes through the tunnel.
Fig. 2 shows an image stitching procedure for the
entire tunnel lining which involves three steps. Captured
image data are first retrieved from the six video cameras,
each of which contains a number of images depended on
the length of the tunnel. In Step 1, the images for each
camera are stitched into a panoramic image in the
longitudinal direction of tunnel for each camera. In Step 2,
these panoramic images obtained in Step 1 are stitched in
101
International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020
© 2020 Int. J. Struct. Civ. Eng. Res.doi: 10.18178/ijscer.9.1.101-105
the circumferential direction of tunnel for each region
corresponding to each pass. Subsequently, in Step 3, the
connected images of all regions in step 2 are stitched
together to make a layout panorama which consists of a
full view of the entire tunnel lining surface.
Video cameraSONY HDR-CX630V
Illuminator LED light Steel frameSliding from side to top
Inspection CarImage acquisition systemcamera angle of view between 5o- 6o
Figure 1. Image acquisition system.
Figure 2. Procedure of automatic image-mosaicking construction.
B. Case Study
The tunnel inspected in this experiment is an actual
single-core circular tube with the length of 230 m, the
width of 10.25m, and the height of 4.7m in Yamaguchi
Prefecture, Japan. The video digital cameras are SONY
HDR-CX630V, with LED light 600lx and camera angle
of view between 50- 6
0. The distance between the
cameras and the tunnel wall is 3m. The data set acquired
by each camera consists of 1,558 images with a
resolution of 1920 × 1080 pixels for each picture. It is
useful to use a photographic laser distance meter to
maintain the constant distance between the tunnel lining
and the inspection car to ensure the constant resolution.
The speed of the inspection car is maintained at 30 5
km/h corresponding to the desired image motion quantity
from 500 to 700 pixels. Further, a prototype software
written in C++ was developed to implement the full
image stitching algorithm.
III. PROPOSED METHOD
A. Image Matching Procedure
Fig. 3 shows an automatic image-matching process of
two consecutive images in the tunnel longitudinal
direction (X-axis). Here, the image 1 is the referenced
image, and the image 2 is the registered image of the
image matching process. A search area is set in advance.
Each movement step of the search point is with respect to
the location which enables the image 2 to be shifted on
image 1 to find the appreciate image-matching location
by measuring the similarity and sharpness in the
brightness of all pixel pairs in the overlapped region of
the two images. Furthermore, to accelerate the search and
measurement process, the search point and similarity
metric are skipped in the search area and overlapped
region with predefined values, respectively.
In this study, matching cost functions to measure the
most intuitive similarity of the pixel intensity values
within the overlapped region are used. Those are the most
popular functions such as the sum of square difference
(SSD), and the sum of absolute difference (SAD). More
complex measure about angle correlation such as the
normalized cross correlation (NCC) and Zero mean
normalized cross correlation (ZNCC) get score in the
interval [-1, 1] are used to measure intuitive similarity of
the color pixel-wise pair in terms of brightness in the
overlapped region, as in Eqs. (1) to (4):
2
1 2 ( , ){ , , }[( )]
( 1) ( 1)
M N
i jR G Bi m j n
I I
SSDN n M m
(1)
{ , , }1 2 ( , )| ( ) |
( 1) ( 1)
R G B
M N
i j
i m j n
I I
SADN n M m
(2)
1( , ) 2( , ){ , , }
2 2
1 ( , ) 2 ( , )
M N
i j i jR G Bi m j n
M N M N
i j i j
i m j n i m j n
I I
NCC
I I
(3)
1( , ) 1 2(i, j) 2
2 2
1(i, j) 1 2(i, ) 2
(I ).(I )
( ) . ( )
M N
i j
i m j n
M N M N
j
i m j n i m j n
I I
ZNCC
I I I I
(4)
In these equations, I1 and I2 are the intensity of pixels
at coordinates (i, j) of images 1 and 2 in the overlapped
region, respectively; SAD is the sum of difference
between the pixel values of images 1 and 2 for each color
channel (R, G, and B).
(Y-axis)
Running direction
(X-axis)
Image 1
Image 2
Search Point
(k, l )
-50
50
0810
Overlapped
region
Figure 3. Searching process of image-matching location
102
International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020
© 2020 Int. J. Struct. Civ. Eng. Res.
However, the standard SAD and SSD functions have
poor performance because image data are attributable to
radiometric distortion and noise. For that the authors
propose modified SAD and SSD functions as follows: The
numerators of Eqs. (1) and (2) are divided by the area of
the overlapping region (the total number of pixels) to
normalize the overlapping region in each measurement.
Furthermore, (m, n) and (M, N) are as the lower left and
upper right coordinates (pixel) of the overlapped region
of the image pairs, respectively. In these functions, the
higher similarity of the overlapped region of two images
yields the smaller score of SAD, SSD.
Conversely, NCC, and ZNCC are determined from the
correlation between the pixel intensity values in the
overlapped region. The larger value of them yields the
higher similarity. These measure is robust than distance
metrics because they are invariant to linear brightness and
contrast variations. However, they have more complex
calculations of division, multiplication and square root.
Therefore, their computation time is more than SAD, SSD.
Advantage over NCC, ZNCC is immune to intensity
distortion. But it is computationally more expensive than
NCC.
The traditional similarity metrics are affected by
artifacts such as non-uniformed brightness, periodic
structures, featureless, and noises. These factors result in
the image-matching error. Therefore, in some cases, the
global minimum cost value of the similarity metric
doesn’t yield well corresponding to the ground truth (G-T
image mosaicking). Otherwise, the local minimum cost is
coincident with the G-T image mosaicking shown in Fig.
4. The authors consider to the local minimum cost.
Global minimum cost
local minimum cost
Figure 4. Cost space of SAD metric for a (31-312) pair in camera 1.
B. Curvature Metric with Full Search Method
The conventional similarity metrics only exploited the
information of the matching point of the pixel being
processed without considering the information of
neighbour pixels. Therefore, the results of similarity
metrics frequently cause to many error ratio. The authors
propose a direct curvature metric method based on
density gradient of the neighbor pixels on full search area
as the following equations:
4
(k,l)
0 1 0 (k , l)
1 4 1 (k, l ) (k, l) (k, l )
0 1 0 (k , l)
C
CUR C C C
C
(5)
(k ,l ) (k ,l) (k ,l )
8(k,l) (k ,l) (k,l) (k , )
(k ,l ) (k ,l) (k ,l )
1 1 1
1 8 1
1 1 1
l
l
C C C
CUR C C C
C C C
(6)
4
(k-1,l)
4 4 4 4(k,l) (k,l-1) (k,l) (k,l+1)
4(k+1,l)
1=
5
CUR
CNP CUR CUR CUR
CUR
(7)
(k 1,l 1) (k 1,l) (k 1,l 1)
8(k,l) (k 1,l) (k,l) (k 1, )
(k 1,l 1) (k 1,l) (k 1,l )
1
9l
CUR CUR CUR
CNP CUR CUR CUR
CUR CUR CUR
(8)
where C(k,l) is a cost value of the central pixel using cost
functions ; () is the skipped pixel number or a size of
search window; CUR4
(k,l) and CUR8(k,l) are the curvature
metric value refer to the 4-connected and 8-connected
neighborhood pixels at the (k,l) coordinates, respectively;
CNP4(k,l) and CNP
8(k,l) are the average values of the
curvature metrics between the central pixel and neighbor
pixels refer to the 4-connected and 8-connected
neighborhood pixels at the (k,l) coordinates, respectively.
The larger CUR and CNP value mean higher matching
possibility.
In most of image matching cases, the curvature metric
yields the correct image matching point result shown in
Fig. 5. However, in some special cases, the direct CUR is
also affected by the outliers shown in the following Fig. 6.
Therefore, to eliminate the outliers, the search-strip
method is adopted [6].
Figure 5. The correct matching point using direct curvature metric in the
cost space.
103
International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020
© 2020 Int. J. Struct. Civ. Eng. Res.
TABLE I. ERROR RATE (%) OF THE SIMILARITY METRICS
Camera
SAD SSD NCC ZNCC
Error rate
Running
time Error rate
Running
time Error rate
Running
time Error rate
Running
time
V1 63.50 62 40.99 65 42.59 78 39.28 88
V2 45.38 60 32.39 63 29.63 77 27.45 139
V3 46.47 62 37.27 66 32.84 82 29.97 140
V4 33.18 64 20.78 68 20.85 83 18.99 138
V5 39.00 64 23.48 67 21.55 83 19.12 138
V6 37.66 64 41.23 63 41.94 82 41.71 139
Average 44.20 62.67 32.69 65.33 31.57 80.83 29.42 130.33
IV. PERFORMANCE EVALUATION
A Evaluation of Image Matching Accuracy Based on
the Parameters of the Video Acquisition System
The speed of the inspection car is set at about 30 km/h,
with possible variation in the range 25 to 35 km/h with
respect to the reliable range of image motion quantity of
[500 700] pixel. The error rate is computed from the
following equation (9):
1
1100%
0 if IMQ [500 700]
1
N
i
i
i
i
AME IMEN
IMEOtherwise
(9)
where N is a number of images in each camera, IMEi is
ith image matching error. IMQi is image motion quantity
at ith measurement; and AME is the accumulative error
rate for each camera. This equation indicates that an IMQ
value is out of range [500 700] pixels considered as an
IME.
Effected by outliers
Figure 6. The incorrect matching point using direct curvature metric in the cost space.
From results of the Table I, the average matching
errors of SAD, SSD, NCC, and ZNCC are respective
around 44.2%, 62.7%, 31.6% and 29.42%. Those
locations where the IMQ has below 500 pixels result from
lacking information variance of the tunnel wall.
Additionally, those locations where IMQ has over 700
pixels are out of search range.
Comparing among the four methods, the error ratios of
SSD, NCC, and ZNCC are almost same. The ZNCC
method is a little better than the others. However, its
running time is the longest of the others with 130.33
minutes, around time cost double.
The SAD method gets the best of time cost, but the
worst of accuracy image stitching in this case. The SSD
method results in higher accuracy than NCC and SAD
methods with average time cost is 65.33 minutes. This
time is faster than the one of NCC matching cost function
but slower than the running time of SAD matching cost
function. SAD metric is too sensitive with noise or outlier
than SSD metric. As shown in Fig. 7, the result of the
SAD metric chooses the location 1 to match. The location
1 is an example of the overlapped region of two images.
In contrast, the result of the SSD metric chooses the
location 2 to match.
Fig.3.1 Comparison of similarity measurement between SAD and SSD.
36 31 27
261 29 26 24
30 25 33
Location: 2
50 5 10
256 75 13 25
15 60 3
Location: 1
<
1296 961 729
7693 841 676 576
900 625 1089
2500 25 100
5625 169 625 12878
225 3600 9
>
SAD
SSD
Figure 7. Comparison of the similarity measurement between SAD and SSD.
104
International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020
© 2020 Int. J. Struct. Civ. Eng. Res.
The subtraction values of the blocks in the overlapped
region at the location 2 have a less difference than the
ones at the location 1.
The Table II shows comparison results of the proposed
curvature metrics for six cameras before refinement is
applied. As can be seen, all of CUR and CNP metrics
have error ratio very small. Comparing among the results
of CUR metrics, CUR4 is the highest accuracy with the
average accuracy of six cameras of 1.95%. Otherwise,
CNPstrip has the best results of the error ratio of 0.52%
compared to the others [6].
TABLE II. ERROR RATE OF THE CURVATURE METRICS
Cam-era
CUR4
CNP4
CUR8
CNP8
CUR strip
[6]
CNPstrip
[6]
V1 1.09 0.90 0.96 0.83 0.77 0.45
V2 0.45 0.39 0.83 0.39 0.77 0.26
V3 0.71 0.64 2.12 0.90 1.67 0.26
V4 0.90 0.90 1.16 0.96 1.99 0.83
V5 0.45 0.45 0.77 0.51 1.60 0.52
V6 8.10 7.52 19.78 4.50 5.80 0.77
Aver
age 1.95 1.80 4.27 1.35 2.10 0.52
V. DISCUSSION
The methods of similarity metrics only exploited the
cost value of the pixel being processed, such that its
image matching accuracy wasn’t high. However, the
curvature metric at the nearest candidates and the
candidate being processed yielded the precise matching-
point. The main advantage of the curvature metric using
search strip was to eliminate the artifacts of the neighbor
pixels surrounding the candidate matching point before
measuring the sharpness at the peaks of the cost curve.
Furthermore, a huge number of the raw images of
tunnel lining surface were matched automatically with the
high accuracy no using pre-processing step.
VI. CONCLUSIONS
The main contributions of the paper comprised of
introducing an imaging device capable of scanning
pictures of the entire tunnel surface.
Comparing the image motion quantity between the
result of similarity metrics and curvature metrics, the
results of curvature metrics are improved significantly.
Indicating the cost space of the similarity and
curvature metrics, the authors propose the curvature
metric to find the precise-matching points for image
mosaicking and assisting tunnel lining inspection.
CONFLICT OF INTEREST
The authors declare no conflict of interest.
AUTHOR CONTRIBUTIONS
Cuong Nguyen Kim conducted research and wrote
paper; Kei Kawamura contributed methodology;
Masando Shiozaki supplied image acquisition device;
Amir Tarighat checked and wrote paper; all authors had
approved the final version.
ACKNOWLEDGMENT
The research in this article is supported by the financial
support of JSPS KAKENHI Grant Number 15K0618.
REFERENCES
[1] S. Yu, J. H. Jang, C. S. Han, “Auto inspection system using a
mobile robot for detecting concrete cracks in the tunnel,” Automation in Construction 16, pp. 255-261, 2007.
[2] W. Zhang, Z. Zhang, D. Qi, Y. Liu, “Automatic crack detection
and classification method for subway tunnel safety monitoring,” Sensors, vol. 14, pp. 19307-19328, 2014.
[3] J. Byrne, F. D. Laefer, “Maximum feature detection in aerial unmanned aerial vehicle datasets,” Journal of Applied Remote
Sensing, vol. 11, 2017.
[4] K. Kawamura, M. Koga, J. Matsumoto, M. Shiozaki, and S. Sawamura, “A study on panoramic images generation of tunnel
wall,” Japan Society of Civil Engineers, vol. 71, no. 2, pp. 142-151, 2015.
[5] K. Kawamura, M. Koga, J. Matsumoto, M. Shiozaki, and S.
Sawamura, “Tunnel wall panoramic images generation using a local search algorithm,” Japan Society of Civil Engineers, vol. 72,
no. 2, pp. 73-82, 2016. [6] C. N. Kim, K. Kawamura, A. Tarighat, M. Shiozaki, “An image
matching method based on the curvature of cost curve for
producing tunnel lining panorama,” journal of JSCE, vol. 6, no. 1, pp. 78-90, 2018.
Copyright © 2020 by the authors. This is an open access article
distributed under the Creative Commons Attribution License (CC BY-
NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-
commercial and no modifications or adaptations are made.
Cuong Nguyen Kim is currently a PhD candidate in department of information science
and engineering at Yamaguchi University, Japan.
He has considered research on image processing, meta heuristics, and the assessment of concrete
structure conditions.
Kawamura Kei is an Associate Professor in department of information
science and engineering at Yamaguchi University, Japan. He received
his PhD degree in system engineering from Yamaguchi University, Japan, in 2000. His research areas are structural engineering, and
maintenance management engineering. For many years, he has developed methods for bridge management systems and the evaluation
of concrete structure conditions.
Amir Tarighat is an Associate Professor in department of civil
engineering at Shahid Rajaee Teacher Training University, Tehran, Iran.
His research interests consist of concrete technology, durability-based design, structural health monitoring and simulation/modeling of
cementitious materials structure from nano to macro scale.
Masando SHIOZAKI is currently a PhD candidate at Okayama
University and belonged to Sumitomo Mitsui Construction Company.
He has considered image processing, crack detection, inspection device of bridges and tunnels, and the assessment of concrete structure
conditions.
105
International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020
© 2020 Int. J. Struct. Civ. Eng. Res.