Multi-Sensor Imagery Processing

for Military Application

2013. 1. 18. (fri) 10:40~11:10

Sang-Hee Kim, Ph.D.

Project Manager in ISR center of ADD

Motivation Ⅰ

GEOINT Data Generation Ⅱ

Multi-sensor Imagery Matching Ⅳ

Conclusion Ⅴ

3-D Model Reconstruction Ⅲ

MOTIVATION

R&D

www.armyrecognition.com/News/2007/March/Milit...

Victory smiles upon

those who anticipate

the changes in the

character of war,

not upon those who

wait to adapt themselves a

fter the changes occur.

-- Italian Air Marshall Giulio Douhet

www.journal.dnd.ca/vo8/no4/robertso-eng.asp

Platforms Intelligence

(PGM) ≫ (ISR)

Platforms Intelligence

(PGM) ≈ (ISR)

SMART system

Smart Multi-INT Application for Relevant Targeting

– Multi-platform/multi-sensor imagery integration

– Image/Video Exploitation and Analysis

– Accurate 3-D Model Reconstruction and Management

– Real-time Sensor ATR

Minimization of off-target attack

– Performance enhancement of weapon system

– Accurate target intelligence and timely transmission

Complex targets

– Randomly shaped facility, natural occlusion or camouflage

표적의 복잡, 다양성

8/74

위협요소 (AAA/SAM 모델 및 화망)

GEOINT activities

Satellite-GRPS, Airborne-GRPS, UAV-GRPS – Ground Receiving and Processing System

– EO/IR(image and/or video), SAR and MTI

Image-based Point Positioning System – cf, Raindrop or CGS on DPPDB in USA

3-D Model Reconstruction and Matching – Geo-spatial data generation

– 2-D terrain feature and 3-D facility extraction

– Geometry-based and image-based matching

Multimedia Database for Threat Detection – Feature-based image analysis and retrieval

GEOINT Data Generation (DEM and Ortho-image)

Receiving Range

Antenna /Receiving Device

Satellites

SPOT-2

SPOT-4

Radarsat-1

2,000km

•Data Acquisition System

GRPS system

Image acquisition Stereo images3D modeling

Regular grid values Coordinate calculation Matching

Ortho-image (texture)Indirect rectificationDigital Elevation Model

left right

left right

matching

point

matching

point

Groundcoordinate

DEM

flat DEM

Image acquisition Stereo images3D modeling

Regular grid values Coordinate calculation Matching

Ortho-image (texture)Indirect rectificationDigital Elevation Model

left right

left right

matching

point

matching

point

Groundcoordinate

DEM

flat DEM

Rigorous Sensor Modeling (SPOT)

Equation

– GCP / Image

– EOP

Exterior Orientation Parameter

– Satellite Position: (XL, YL, ZL)

– Attitude: (L, L, L)

X

Y

Z

O

GCP(Xgcp,Ygcp,Zgcp)

(iR,jR)

(iL,jL)

EOP(XL,YL,ZL,L, L, L) EOP(XR,YR,ZR,R, R, R) X X X t X t X t

Y Y Y t Y t Y t

Z Z Z t Z t Z t

L 0 1 2

2

n

n

L 0 1 2

2

n

n

L 0 1 2

2

n

n

...

...

...

2nd polynomial

3 Dimensional Modeling

Strip mode

0 11 12 13

21 22 23

31 32 33

y

f

R

X X

Y Y

Z Z

R R R R

m m m

m m m

m m m

L

L

L

L

L L L L

, ( ) ( ) ( )

X X X t X t

Y Y Yt Y t

Z Z Z t Z t

t t

t t

t t

L

L

L

L

L

L

0 1 22

0 1 22

0 1 22

0 1 22

0 1 22

0 1 22

Collinearity (SPOT line L)

o Satellite orbit parameter

o equation

F fm X X m Y Y m Z Z

m X X m Y Y m Z Z

G y fm X X m Y Y m Z Z

m X X m Y Y m Z Z

i L i L i L

i L i L i L

ii L i L i L

i L i L i L

0 11 12 13

31 32 33

11 12 13

31 32 33

( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( )

o Taylor series

FF F F F

XX

F

YY

F

ZZ

F F F F

XX

F

YY

F

ZZ

F F F F

00

00

00

00

00

00

0

11

11

11

11

11

11

22

22

22

X

XF

YY

F

ZZ

22

22

22 0

GG G G G

XX

G

YY

G

ZZ

G G G G

XX

G

YY

G

ZZ

G G G G

00

00

00

00

00

00

0

11

11

11

11

11

11

22

22

22

X

XG

YY

G

ZZ

22

22

22 0

o Observation equation construction

GCP

orbit

image

C

C

C

I

A

I

B

V

V

V

2

1

2

1 0

0

o Simplified equation

V B C

p n m p n m n m n m p n m( , ) ( , ) ( , ) ( , )2 18 3 1 2 18 3 18 3 18 3 1 2 18 3 1

RPC(Rational Polynomial Coefficients)

Developed for IKONOS by Space Imaging Inc.

WGS84, Geographic coordinate _

_

_

_

_

_

Latitude LAT OFFP

LAT SCALE

Longitude LONG OFFL

LONG SCALE

Height HEIGHT OFFH

HEIGHT SCALE

( , , )( , , )

( , , )

( , , )( , , )

( , , )

L

L

S

S

Num P L HY f Latitude Longitude Height

Den P L H

Num P L HX g Latitude Longitude Height

Den P L H

2 2

1 2 3 4 5 6 7 8 9

2 3 2 2 2 3

10 11 12 13 14 15 16

2 2 2 3

17 18 19 20

2 2

1 2 3 4 5 6 7 8 9

2 3 2 2 2

10 11 12 13 14 15

L

L

Num a a L a P a H a LP a LH a PH a L a P

a H a PLH a L a LP a LH a L P a P

a PH a L H a P H a H

Den b b L b P b H b LP b LH b PH b L b P

b H b PLH b L b LP b LH b L

3

16

2 2 2 3

17 18 19 20

P b P

b PH b L H b P H b H

2 2

1 2 3 4 5 6 7 8 9

2 3 2 2 2 3

10 11 12 13 14 15 16

2 2 2 3

17 18 19 20

2 2

1 2 3 4 5 6 7 8 9

2 3 2 2 2

10 11 12 13 14 15

S

S

Num c c L c P c H c LP c LH c PH c L c P

c H c PLH c L c LP c LH c L P c P

c PH c L H c P H c H

Den d d L d P d H d LP d LH d PH d L d P

d H d PLH d L d LP d LH d L

3

16

2 2 2 3

17 18 19 20

P d P

d PH d L H d P H d H

_ _

_ _

Line Y LINE SCALE LINE OFF

Sample X SAMP SCALE SAMP OFF

RPC fitting

Gene Dial, Jacek Grodecki, Block adjustment with rational polynomial camera models,

ACSM-ASPRS 2002 Annual Conference Proceedings, 2002.

( ) ( ) ( )

( ) ( ) ( )

( , , )

( , , )

( , , ) ( , , ) _ _

( ,

ij

ij

j j j

i k k k L

j j j

i k k k S

Line p p Latitude Longitude Height

Sample r r Latitude Longitude Height

p Latitude Longitude Height f Latitude Longitude Height LINE SCALE LINE OFF

r Latitude Lo

0

0

, ) ( , , ) _ _

S L

S L

ngitude Height g Latitude Longitude Height SAMPLE SCALE SAMPLE OFF

p a a Sample a Line

r b b Sample b Line

RPC : accuracies vary with satellites

Need to correct using minimal GCPs

RPC correction-1

pi=(x, y, lon, lat, alt)

Forward

form

Pi’=(x’, y’, lon, lat, alt)

Transformation Matrix – ≤ 2 points : Translation

– 3 points : Affine Transform

– ≥ 4 points : Homography

RPC correction-2

[ ] 3x3 Left image

[ ] 3x3 Right image

3-D

Reconstruction

RPC + GCP + DEM

Ortho-rectified Image

P=(X,Y,Z)

Z

Ortho image

(X,Y)

Interpolated pixel

p=(x,y) p’=(x’,y’)

correction

RPC Forward Form

DEM

input

process

Position Accuracy Evaluation

CMAS r0

90%

Circular MAS Linear MAS

measurement

Statistical analysis

Test points

LMAS r0

90%

Below c

Multi-resolution/Multi-layer Terrain

Modeling and controlling

Fused mesh of multi-resolution DEMs

Geo-consistent processing

higher terrain layers and base layers (DEM and texture)

3D model

Geo-specific (high resolution)

Geo-typical (low resolution)

3-D Model Reconstruction

GPS navigation

TERCOM

navigation

Image-based

navigation

GPS

interference

Mission start

Registered Optical and DEM

Parameterized Model Library

Site Model Visualization

Model Indexing

Elevation Surface and Histogram

Model F

it: O

ptim

ization

Automated 3-D Model Reconstruction(1/2)

Image Sequence

Feature Extraction/ Matching

Relating Image

Projective Reconstruction

Auto-Calibration

Dense Matching

3D Model Building

Automated 3-D Model Reconstruction(2/2)

Manual Extraction(1/2)

Vertex: 495

Face: 395

Vertex: 1226

Face: 886

Manual Extraction(2/2)

3D-METER(Model Extraction and Terrain Registration)

Depth map generation

Terrain & Feature

Integration

Ortho-rectificatio

n

3D Model Database

DTM, 3D Model, Texture Image, Model Hierarch, Orthophoto Etc…

3D Modeling

3D Height estimation

3D Model Editing

Texture Mapping

Stereo Matching

Terrain Feature and Boundary

추출 및 2D 모델링 Feature & Rooftop Extraction/Modification

Metadata

GCP

Image Data

RPC analysis

Image Processing

Single Image Dual Image Stereo Image

New project, GCP correction

ROI, Resolution merged, Ortho-Image

(1) Extraction of 2-D terrain feature

SNAKE(Active Contour Model) algorithm

N

i

idigisic pdEpcEpbEpaEE1

)()()()(

Internal energy External energy

cE

sE

“Continuity”

“Smoothness”

“Edgeness”

a, b, c, d : weights to control the curved shape

gE

dE “Clustering”

clustering term

to deal with

concave or highly

curved region

s

Pi

Edge attraction

Smoothness Continuity

Simple feature extraction (Road)

Complex feature(river): SNAKE+ISODATA+CPS

original ISODATA result Boundary energy

proposed snake snake

Comparison of # of control points

input control points

Snake control points

Control points

after CPS

building 14 168 80

road 9 343 86

river 120 1145 192

Extraction of 3-D Facilities

Rooftop morphology

Satellite and Sun position

Move Rooftop-shadow edge

Shadow direction estimation

Rooftop-shadow edge selection

Line evaluation

Shadow Region

Yes

No

)tan(/

)tan(/

sat

sun

HVL

HSL

Height calculation (H)

Facade line azimuth

su

Satellite azimuth

North Sun azimuth

Facade Normal

tsa

sun

san scan

Building

t

tsasan

tsusun

sansan

sunsun

sansun

KSH

VLS

SLS

SSS

90

90

)cos(

)cos(

S : Estimated Shadow length

))}tan(/)(cos())tan(/)/{(cos()sec( sansansunsunscantK

Consideration of Terrain elevation

Rooftop

Image plane

Terrain

S

p R1

R'1

hs

H

ht

S'

t

t

t

t

t

h

zDyDh

zDxDh

p

zD

ht

tD

h

y

x

p

./.

./.

.

Search the terrain patch in azimuth direction of Satellite

3-D Model Extraction(Facility)

Coordinate generation by image registration

3D models extracted and merged on terrain

QuickBird Aerial Image WorldView

Case2

04

Reference Target Data generation (Geometry+image) on sensor parameter

Accuracy evaluation of 3D model

sample : 10 facilities

1 3

Model #1

3D-METER

X : 246494.63

Y : 4063550.50

Z : 9.55

11.37 19.37

9.21

Model #3

30.66

10.74

5.06

3DM

X : 247068.84

Y : 4063560.25

Z : 12.23

11.37 19.37

9.21

Model Index Data

Points Model size

X Y Z WW W H

1

Measured 246495.132 4063550.269 12.029 20.115 12.168 8.907

3D-METER 246494.63 4063550.5 9.55 19.37 11.37 9.21

Difference

(m) 0.502 -0.231 2.479 0.745 0.798 -0.303

3

Measured 247069.465 4063560.056 13.66 30.638 10.386 6.46

3D-METER 247068.84 4063560.25 12.23 30.66 10.74 5.06

Difference

(m) 0.625 -0.194 1.43 -0.022 -0.354 1.4

Relative Error btw models (Test Site #3)

1

40.253

2

3

4 5

6

7

8

9

10 11

12

36.045 40.195

52.461

Site 1 (unit: m) Site 2 (unit: m) Site 3 (unit: m)

Absolute error

- X : 5.889

- Y : 1.375

-Z : 0.425

Size error : 0.283

Absolute error

- X : 7.121

- Y : 0.470

- Z : 0.195

Size error : 0.357

Relative error : 0.271

Absolute error

- X : 7.378

- Y : 0.986

- Z : 0.733

Size error : 0.570

Relative error : 0.694

Data : World view2, SRTM Level2

Relative Accuracy

Within 2 pixels in error !!

Multi-sensor Imagery Matching

49

Registration based on Multi-sensor imagery

Target

Detect/

Track

3D Model

Super-resolution

Initial match

Multi-sensor image registration

CCD IR Precise match

Image registration

Registration

);,(maxarg mfreg PIIP

PS

: Pixels in a fixed image

: Pixels in a moving image

If: Fixed image

Im: Moving image

S: Similarity measure

P: Parameter set of transformation model

Similarity measure

);( YXI

)(XH

),( YXH

)(YH

)|( YXH )|( XYH

),()()(

)|()(

)|()();(

YXHYHXH

XYHYH

YXHXHYXI

),(

)()();(

YXH

YHXHYXI

N

+1

+1

+1

+1

Fixed image intensity

Movin

g im

age inte

nsi

ty

<Joint histogram> EO image

IR image

Challenging Problems

LOST

LOCK - Target

CM : Flare

LOCK - Target

LOST

(1) Counter Measure

(2) Image Database

Image Retrieval Processing

Image Indexing & Storing

Input Image

Segmentation &

Feature Extraction Image Indexing Metadata

Query Image

Segmentation &

Feature Extraction Query Processing

Location data

Feature data

Annotation …

Meta

data

Meta

data

Meta

data

Image Database

Browsing Results

. . . . . . . .

Similarity

Function

Relevance

Feedback …

Threat Detection

(3) True Ortho-rectified Image

Traditional Ortho image

- Rectified from a DTM (floor)

- Slanted buildings

- Occulted areas

True Orthoimage

- Rectified from a DSM

- Straight buildings

(roofs correctly located)

- No occulted areas

Shadow area

Occluded area

(4) Fusion on Multiple Sensor Images

enhancement

Multi-sensor

Image

(real time)

fusion 3D model

Satellite image,

Lidar data

(off line)

Feature matching

3D model

(texture + terrain)

Satellite image Lidar

CCD IR or SAR

Image Fusion

Target recognition 3-D model information

(5) 3D Positioning with New Single Image

Dataset :

New single image, Satellite Stereo image Database(DPPDB, IBPPS, etc)

Procedure

DPPDB : Digital Point Positioning Data Base

IBPPS : Image Based Point Positioning System

Start

Calculate RPC coeff. of the single image

Observe Image Coord. of the Objects in the single image

and Sat. Stereo Images

Input Single Image

Observe Image Coord. of GCPs

Input Sat. Stereo Images and RPCs

Calculate GRD Coord. of GCPs

3D Geopositioning with multi-images RPC model

Finish

Observe Image Coord. of GCPs

RPC Sensor Modelingthe Single Image

(101) (102)

(103) (104)

(105)

(106)

(107)

(108)

(109)

H

(0,0,0)

i

iHi

( , , )i i i iGCP H

Sat. Stereo(Left)

( , )L Ll s

Sat. Stereo(Right)

( , )R Rl s

RRPCsLRPCs

RPCs

Single Image

( , )i il s(101)

(102)

(103)

(104)

(105)

(106)

(107)

(104)

Dataset :

New single image, Satellite Stereo image Database(DPPDB, IBPPS, etc)

Procedure

Start

Calculate RPC coeff. of the single image

Observe Image Coord. of the Objects in the single image

and Sat. Stereo Images

Input Single Image

Observe Image Coord. of GCPs

Input Sat. Stereo Images and RPCs

Calculate GRD Coord. of GCPs

3D Geopositioning with multi-images RPC model

Finish

Observe Image Coord. of GCPs

RPC Sensor Modelingthe Single Image

(101) (102)

(103) (104)

(105)

(106)

(107)

(108)

(109)

Ground

l

s

( , )L Ll s

l

s

( , )R Rl s

U,V,W

RRPCsLRPCs

l

s Single Image

( , )l s

RPCs

Sat. Stereo(Left) Sat. Stereo(Right)

(108)

(108) (109)

CONCLUSION

My ultimate research goal : SMART

GEOINT data generation using GRPS system

Target model reconstruction : 3D-METER

– One-stop integrated modeling software giving simple, efficient, accurate

(application-oriented system)

Multi-sensor imagery matching

Some challenging problems to be worked out