Upload
hoangcong
View
231
Download
2
Embed Size (px)
Citation preview
next level solutions13 July 2001Harris Proprietary
An Autonomous 3An Autonomous 3--D Photogrammetric ApproachD Photogrammetric Approach
Steven G. Blask, John A. Van Workum{sblask, jvanwork}@harris.com
Harris Corporation GCSD
to Airborne Video Geoto Airborne Video Geo--RegistrationRegistration
next level solutions13 July 2001Harris Proprietary
Outline
• Overview of Harris Registration Approach
• Airborne Video Extensions - PVR System
• DARPA AVS-PVR Processing Results
• Discussion
next level solutions13 July 2001Harris Proprietary
Impact of ESD Errors
Geo-Reference Image Video Frame
next level solutions13 July 2001Harris Proprietary
MP1
+
MP2
+
MP3
+
MP4
+
MP5
+
MP6
+
MP7
+
MP8
+MP9
+
MP10MP10
+
MP11
+
MP12
+
MP13MP13
+
MP14
+
MP15
+
MP16
+
Photogrammetric Model Based Registration Overview
MP1
+
MP2
+
MP3
+
MP4
+
MP5
+
MP6
+
MP7
+
MP8
+MP9
+
MP10MP10
+
MP11
+
MP12
+
MP13MP13
+
MP14
+
MP15
+
MP16
+
AmbiguityResolution
ConvergedSolution
Sensor ModelAdjustment
DEM orSite Model
Initial Transformation Process• Image -> 3-D Surface -> 2D View• Tile Overlap Area in Scene Space• Build Common Neighborhood Windows
Normalized X-Correlation• In Enhanced Edge Space• Build Correlation Surface• Evaluate Correlation Peaks
Consistent Subset Analysis• Compute Mean Offset Vector• Sequentially Sort Peaks• Resolve Ambiguities & Outliers• Adjust Sensor Parameters• Repeat at Next Resolution Level
OutputAdjusted
Support Data
Support Data InitializesSensor Geometry Model
Registered Images
x, y, z, v, g.
x, y, z, v, g.
PeaksMatchPoints
AdjustedParameters
next level solutions13 July 2001Harris Proprietary
Initial Transformation Process
• The images are subsampled to create reduced resolution data sets• Software resampler creates patches at any required GSD on demand
Level5..
2
1
0
256
128
64
256
128
64
next level solutions13 July 2001Harris Proprietary
Initial Transformation Process
Sensor 1 Sensor 2
next level solutions13 July 2001Harris Proprietary
• Use a priori knowledge of each sensor imaging event and a Digital Elevation Model (DEM) to project imagery to the 3D terrestrial surface
Initial Transformation Process
Collected ImageCollected Image
Orthorectified Image
DEM
u = Pu (ϕ, λ, h)v = Pv (ϕ, λ, h)
GeometryModel forSensor 1
GeometryModel forSensor 2
(ϕ, λ, h) = F (line, sample,Xv, Yv, Zv, roll, pitch, heading,azimuth, elevation, twist,focal length)
next level solutions13 July 2001Harris Proprietary
Initial Transformation Process
• Orthorectification places the images in a common orientation with minimal distortion present (unmodelled buildings & trees still layover)
next level solutions13 July 2001Harris Proprietary
Normalized X-Correlation
Image 1Neighborhood
Pre-EmphasisFilter
Cross-Correlation
Image 2Neighborhood
3-D Cross-Correlation
Surface
Pre-EmphasisFilter
xij
3N = # of elements in reference patch
2N = # of elements in comparison patch
= elements in comparison patch
ijy = elements in reference patch
( )
������
�
�
������
�
�
���
����
�
−���
����
�
−
−=
����
����
��
= =
= =
= =
= =
= =
3
2
1 1
1 1
2
2
2
1 1
1 1
2
1 1
2
3
N
yy
N
xx
yxx
NNr
n
i
n
jijn
i
n
jij
n
i
n
jijn
i
n
jij
n
i
n
jijij
Correlation Patch
up to 4 candidates are chosen
ReferencePatch
ComparisonPatch
next level solutions13 July 2001Harris Proprietary
Matchpoint Display
Geo-Reference ImageryVideo Mission Image
next level solutions13 July 2001Harris Proprietary
+ + +
+ + + +
+ + + +
+ + + +
+
Estimate of Misalignment
• Multiple correlation peaks are computed for each grid point neighborhood
• A parametric hill finder is used to evaluate each peak
• The mean and standard deviation of registration error are calculated from the offset and average ellipse
• The best consistent subset of correlation peaks is chosen by sequential sorting
• Offset vectors imply global ground “correction” needed to improve registration, wild pt. editing eliminates outliers
next level solutions13 July 2001Harris Proprietary
Sensor Adjustment Process
• Sensor parameters are adjusted to minimize the error between ground projections of common match points
• Conjugate Gradient Search, Least Squares, and Kalman Filter adjustment algorithms
Image 2 geometry parameters
Minimal Perturbation Adjustment Procedure
Image 1 geometry parameters
Registration measurements
(correspondences)
x1, y1, z1, v1, v1.
x2, y2, z2, v2, v2.
next level solutions13 July 2001Harris Proprietary
Output & Derivative Products
• Improved telemetry used by Geolocation & Mosaic– Telemetry parameters initialize sensor model to define a 3D ray
through any pixel in the image, which may be intersected with the DEM to produce a geolocation or orthorectify a video frame.
– By improving telemetry, we improve geodetic accuracy of pixels.
N38.12.00, W077.24.00
N38.07.00, W077.18.45
N38.12.00, W077.18.45
N38.07.00, W077.24.00
next level solutions13 July 2001Harris Proprietary
Registration Solution
���
�
�
���
�
�
⋅++⋅+⋅= ���
����
�
∂∂
���
����
�
∂∂
���
����
�
∂∂
2
2
22
2
21
2
21n
ne
xf
xf
xf σσσ
�
���
�
�
���
�
�
⋅++⋅+⋅= ���
����
�
∂∂
���
����
�
∂∂
���
����
�
∂∂
2
2
22
2
21
2
21n
nxf
xf
xf σσσ
�
1x1x , 2x2x ,� nxnx parameterstherepresents parameterstherepresents
σσ σσ11 , 22 ,�
nσnσ ixofvariancestherepresents ixofvariancestherepresents
f parameterstheoffunction therepresents parameterstheoffunction therepresents
• Advantage of model-based approach: can perform rigorous error propagation to characterize geopositioning solutions and provide a posteriori error covariances for adjusted sensor model params
initial
registered
higher accuracy,tighter error bounds
RegisteredSensor 1 &Sensor 2
next level solutions13 July 2001Harris Proprietary
Airborne Video Extensions
Precision Video Registration SystemPrecision Video Registration System
next level solutions13 July 2001Harris Proprietary
PVR Architecture
Video andTelemetry
Video A/Dand
TelemetryDecoder
Video A/Dand
TelemetryDecoder
Reference& Frame
Repository
Reference& Frame
Repository
PrecisionGeolocationProcesses
PrecisionGeolocationProcesses
Telemetry &RegistrationProcesses
(RTVR)
Telemetry &RegistrationProcesses
(RTVR)
PrecisionMosaic
Processes
PrecisionMosaic
Processes
Video Video Frames
AdjustedTelemetry
RawTelemetry
Video Frames
C2 UserC2 User
IA UserIA User
Auto’dProc.
Auto’dProc.
PVRManager
PVRManager
� Controls PVR Processes� Handles Status Messages� Reports Metrics
Precision Broadcast
CAGS Services PVR ClientsPVR Processing
next level solutions13 July 2001Harris Proprietary
RTVR Architecture
Process Control
Telemetry Adjustments
Telemetry Database
RawTelemetry
Selected ESD
SelectedVideo Frames
Match PointSequential State
Estimator
MatchPointsLSE Worm
Controller
Observation CombinerPrecision
MatchPoints
MatchPoints
MatchPoints
PrecisionMatchPoints
PrecisionMatchPoints
MatchPoints
Observation Generators
next level solutions13 July 2001Harris Proprietary
Telemetry Queue/Database
Time
VideoFrames
30 Hz
Position
10 Hz
Attitude
30 Hz
Gimbals
25 Hz
Sensor
6.25 Hz
Raw Telemetry(from CAGS Metadata)
Telemetry Adjustment(from Match Pt. SSE)
HARD REGISTRATION
treq
tprev
tnextHARD REGISTRATION
SOFT REGISTRATION
Quality of Service:PrecisionExtrapolatedInterpolated
Quality of Service:PrecisionExtrapolatedInterpolatedPosition, Attitude,
Gimbals, Sensor DeltasGoal: 1 Hz
Oct’99 FEX: 0.22 Hz
next level solutions13 July 2001Harris Proprietary
Affine Consistent Subset
• Required to account for scale and rotation distortion
next level solutions13 July 2001Harris Proprietary
200 Unregistered Frames
Selected frames from (in order) 1 hour, sparsefeatures, class1, and class2 data sets, 29Mar99.
Raw telemetry errors in ascending order.
next level solutions13 July 2001Harris Proprietary
Original CSS Results
Selected frames from (in order) 1 hour, sparsefeatures, class1, and class2 data sets, 29Mar99.
Registration errors for original consistentsubset criterion in ascending order.
Outliers due to fuselage obscuration and low elevation angles
next level solutions13 July 2001Harris Proprietary
Affine CSS Results
Selected frames from (in order) 1 hour, sparsefeatures, class1, and class2 data sets, 29Mar99.
Registration errors for affine consistentsubset criterion in ascending order.
Outliers due to fuselage obscuration and extremely low low elevation angles (17-20 deg)
next level solutions13 July 2001Harris Proprietary
KF Adjustment Algorithm
• Sparse scene content of Airborne video requires accumulation of match points over space and time
• Kalman filter adjustment vs. N-frame co-registration– Adds one image at a time to solution– Only need to estimate parameters for one image– Smaller set of equations– No waiting for additional images
• State vector X models adjustments to telemetry; slowly varying bias suggests constant state model is suitable:
next level solutions13 July 2001Harris Proprietary
KF vs. Single Frame Results
Oct'99 VA D2 Clip
0
5
10
15
20
25
30
35
40
63 frames, EO vs DOQ DTED1
grou
nd e
rror
(m)
beforeautoregkftype
kf: median: 3.80 90th: 8.44 mean: 4.70 std dev: 2.50 <10m: 93.3%autoreg: median: 7.44 90th: 13.33 mean: 8.27 std dev: 3.62 <10m: 71.6%
before: median: 23.25 90th: 28.43 mean: 23.24 std dev: 4.08 <10m: 0.0%
next level solutions13 July 2001Harris Proprietary
KF vs. Single Frame Results
Oct'99 VA D2 Order Stats
0
5
10
15
20
25
30
35
63 frames, EO vs DOQ DTED1
grou
nd e
rror
(m)
beforeautoregkf
next level solutions13 July 2001Harris Proprietary
Scene Content & Prescreener
• Compute MPt normalized image space residuals:
• Apply thresholds– min. no. match points
• 9 for frame-to-mono ref.• 5 for frame-to-stereo ref.• 4 for frame-to-frame
– avg. norm. res. ≤≤≤≤ 1 pixel– max. norm. res. ≤≤≤≤ 2 pixels
��
���
�
−−
=
=
22
11
-1T
xyxy
Σρ
ε
εε
Correlation surfaceexamples
Scene contentexamples
StrongPeak
Ridges
WeakPeaks
next level solutions13 July 2001Harris Proprietary
Significant Scene Content Changes Low Salient Feature Density
Dynamic Video Worm
Successful Mission-to-Reference Single Frame Event
Prescreened Mission-to-Reference Single Frame Event,Successful Mission-to-Mission Match Points
Worm Anchor Mission-to-Reference Frame
Prescreened Mission-to-Mission, Unanchored End of Worm
Unanchored Worm Segment, Interpolated “Soft” Adjustment
next level solutions13 July 2001Harris Proprietary
DTED Accuracy
DTED Terrain Surface
Solution Point
Solution Covariance
Ray Covariance
Image Ray/Range Arc
Image 1
Video 1
Video 2
Geo-Ref
Further Accuracy Improvement
Geometry effects
next level solutions13 July 2001Harris Proprietary
PVR Georegistration Performance Using DOQ & DTED
Dynamic WormDynamic Worm(LSE, KF, & Prescreener)(LSE, KF, & Prescreener)
next level solutions13 July 2001Harris Proprietary
DOQ Timing
• Reference Data– USGS Digital Ortho Quarter-Quad (1m GSD)– NIMA Digital Terrain Elevation Data (100m posts)
• Timing Data– SGI Octane– Dual 225MHz R10,000 cpu’s– 512Mb RAM total– Controller, Generator, Worm Combiner thread
next level solutions13 July 2001Harris Proprietary
NY Intersection Circle Stare
next level solutions13 July 2001Harris Proprietary
Feb'00 NY Site, Clip A4
0
10
20
30
40
50
60
70
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61
61 frames, EO vs DOQ & DTED1
Gro
und
Erro
r (m
)
beforekftype
before: median: 33.1 90th: 42.4 mean: 32.5 std dev: 8.6 <10m: 0%kf: median: 2.7 90th: 4.4 mean: 3.0 std dev: 1.3 <10m: 100%
NY Intersection Circle Stare
next level solutions13 July 2001Harris Proprietary
Feb'00 NY Site, Clip A4 Timing Stats
0
2
4
6
8
10
12
14
16
18
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61
61 frames, EO vs DOQ & DTED 1
Late
ncy
(sec
)
reg timereg type
NY Intersection Circle Stare
mean: 9.5stdev: 2.6
next level solutions13 July 2001Harris Proprietary
VA 15-Oct Fast Straight Line
next level solutions13 July 2001Harris Proprietary
Oct'99 VA Site, Clip D2
0
5
10
15
20
25
30
35
40
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63
63 frames, EO vs DOQ & DTED3
Gro
und
Erro
r (m
)
beforekftype
before: median: 23.2 90th: 28.4 mean: 23.2 std dev: 4.1 <10m: 0%kf: median: 3.8 90th: 8.4 mean: 4.7 std dev: 2.5 <10m: 93%
VA 15-Oct Fast Straight Line
next level solutions13 July 2001Harris Proprietary
Oct'99 VA Site, Clip D2 Timing Stats
0
10
20
30
40
50
60
70
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63
63 frames, EO vs DOQ & DTED3
Late
ncy
(sec
)
reg timereg type
VA 15-Oct Fast Straight Line
next level solutions13 July 2001Harris Proprietary
NC Suburban Run
next level solutions13 July 2001Harris Proprietary
Mar'00 NC Site, Clip B4
0
5
10
15
20
25
30
35
40
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55
55 frames, EO vs DOQ & DTED1
Gro
und
Erro
r (m
)
beforekftype
before: median: 24.4 90th: 27.6 mean: 24.3 std dev: 3.1 <10m: 0%kf: median: 4.5 90th: 9.1 mean: 4.9 std dev: 2.3 <10m: 96%
NC Suburban Run
next level solutions13 July 2001Harris Proprietary
Mar'00 NC Site, Clip B4 Timing Stats
0
50
100
150
200
250
300
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55
55 frames, EO vs DOQ & DTED 1
Late
ncy
(sec
)
reg timereg type
NC Suburban Run
next level solutions13 July 2001Harris Proprietary
DOQ Validation Summary
Rollup - DOQ - Before Registration
0
10
20
30
40
50
60
70
A1 A2 A3 A4 B3 B4 D2 D3 D4
Gro
und
Erro
r (m
)
Rollup - DOQ - After Registration
0
5
10
15
20
25
30
35
A1 A2 A3 A4 B3 B4 D2 D3 D4 G
roun
d Er
ror (
m)
LEGEND:• maximum- 90th percentile- 3rd quartile- median- 1st quartile- 10th percentile• minimum
Note 2X scale changeBefore vs. After
next level solutions13 July 2001Harris Proprietary
DOQ Validation Summary
11 June 2001 Rollup - DOQ
05
1015202530354045
A1 A2 A3 A4 B3 B4 D2 D3 D4
Gro
und
Erro
r (m
)
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
Succ
ess
(% <
10m
)
Mean Before Mean After Std Dev Before Std Dev After Success Rate
Look Angle GSD # FramesHigh Fine 0High Medium 273High Coarse 6
Moderate Fine 0Moderate Medium 134Moderate Coarse 165
Low Fine 0Low Medium 12Low Coarse 60
DOQ Total: 650
High > 55 Fine .15-.3Mod 35-55 Med .3 - 1Low < 35 Coarse > 1
next level solutions13 July 2001Harris Proprietary
References
J. K. Bryan, D. M. Bell, A. J. Lee, N. H. Carendar, and F. H. Baker, “A New Image Registration Paradigm”, Electronic Imaging International Conference Proceedings, Boston, MA, Sep. 1993.
D. M. Bell, J. K. Bryan, and S. B. Black, “Mechanism for Registering Digital Images Obtained From Multiple Sensors Having Diverse Image Collection Geometries,” U.S. Pat. No. 5,550,937, Aug. 1996.
J. Hackett, D. Trask, and R. Cannata, “Automated Near-Real Time Registration and GeopositioningBased Upon Rigorous Photogrammetric Modeling,” ASPRS-RTI Conf. Proc., Tampa, FL, 1998.
A. J. Lee, D. M. Bell, and J. M. Needham, “Adjustment of Sensor Geometry Model Parameters Using Digital Imagery Co-registration Process to Reduce Errors in Digital Imagery Geolocation Data,” U.S. Pat. No. 5,995,681, Nov. 1999.
R. Cannata, M. Shah, S. Blask, and J. Van Workum, “Autonomous Video Registration Using Sensor Model Parameter Adjustments,” Proc. 29th Applied Imagery Pattern Recognition Workshop,Washington, D.C., Oct. 2000, pp 215-222.
J. Van Workum and S. Blask, “Adding Precision to Airborne Video with Model Based Registration,” Proc. 2nd Int’l Workshop on Digital and Computational Video, Tampa, FL, Feb. 2001, pp 44-51.
S. G. Blask and J.A. Van Workum, “An Autonomous 3D Photogrammetric Approach to Airborne Video Geo-Registration,” Invited Talk at IEEE Workshop on Video Registration held with The Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada, July 13, 2001, http://www.cs.ucf.edu/~vision/workshop/workshop.html.