ATR Theory and ATR Performance Analysis and Predictionjao/Talks/InvitedTalks/SPIE...ATR Theory and ATR Performance Analysis and Prediction Joseph A. O’Sullivan Electronic Systems

Supported by:Supported by: ARO Center for Imaging Science DAAH 04ARO Center for Imaging Science DAAH 04--9595--1049410494

ONR MURI N00014ONR MURI N00014--9898--11--0606--0606

Boeing FoundationBoeing Foundation

ATR Theory and ATR Performance ATR Theory and ATR Performance

Analysis and PredictionAnalysis and Prediction

Joseph A. OJoseph A. O’’SullivanSullivanElectronic Systems and Signals Research LaboratoryElectronic Systems and Signals Research Laboratory

Department of Electrical EngineeringDepartment of Electrical Engineeringjaojao@@eeee..wustlwustl..eduedu

Michael D. Michael D. DeVoreDeVore andand NataliaNatalia A.A. SchmidSchmid

Washington University in St. LouisWashington University in St. LouisSchool of Engineering and Applied ScienceSchool of Engineering and Applied Science

22

ATR Theory and PerformanceATR Theory and Performance OO’’Sullivan, SPIE SAR 2001Sullivan, SPIE SAR 2001

Invitation from Fred GarberInvitation from Fred Garber

“…“… givegive ‘‘invited paperinvited paper’’ addressing the subject addressing the subject

matter of the day.matter of the day.

The subjects of Thursday's session are:The subjects of Thursday's session are:

ATR Performance EvaluationATR Performance Evaluation,, Theoretical Approach to ATRTheoretical Approach to ATR,,

andand ATR Performance Prediction.ATR Performance Prediction.””

My vision of ATR theory and ATR performance analysis.My vision of ATR theory and ATR performance analysis.

33


ATR Theory and PerformanceATR Theory and Performance

•• ATR Systems of InterestATR Systems of Interest

•• Training and Testing ParadigmTraining and Testing Paradigm

•• Some System Design IssuesSome System Design Issues

•• Information Theory and ATR Information Theory and ATR

•• System Implementation IssuesSystem Implementation Issues

•• ConclusionsConclusions

aa=T72=T72

SARSAR

PlatformPlatform

rr

TargetTarget

ClassifierClassifier

OrientationOrientation

EstimatorEstimator

ââ=T72=T72

=45=45°°^

ModelModel

DatabaseDatabase

OutlineOutline

44


ATR Systems of InterestATR Systems of Interest

•• Imaging SensorImaging Sensor

•• Problem DefinitionProblem Definition

•• Algorithm for Algorithm for

–– ClassificationClassification

–– Parameter estimationParameter estimation

•• System Resource ConstraintsSystem Resource Constraints

–– Database sizeDatabase size

–– Processor speedProcessor speed

–– Communication speedsCommunication speeds

–– ArchitectureArchitecture

aa=T72=T72

SARSAR

PlatformPlatform

rr

TargetTarget



EstimatorEstimator

ââ=T72=T72

=45=45°°^

ModelModel

DatabaseDatabase

Parameters:Parameters:

•• PosePose

•• VelocityVelocity

•• ““FeaturesFeatures””

55


ATR System Design: Training Paradigm ATR System Design: Training Paradigm

ParameterParameter

ExtractionExtraction

ScoreScore

FunctionFunction InferenceInference

Scene and SensorScene and SensorPhysicsPhysics

Training DataTraining Data

ProcessingProcessing

ââ=T72=T720 50 100 150 200 250 300 350 400

7.4

7.6

7.8

8

8.2

8.4

8.6

8.8x 10

4

Azimuth (degrees)

Raw HRR Raw HRR

DataData

SAR ImageSAR Image ScoreScore

functionfunction

66


ATR System Design: Training Paradigm ATR System Design: Training Paradigm

•• Likelihood functions Likelihood functions

parameterized by functionsparameterized by functions

•• TrainingTraining

–– Function estimationFunction estimation

•• InferenceInference

–– Hypothesis testingHypothesis testing

–– Parameter estimationParameter estimation

FunctionFunction

EstimationEstimation

LL((rr||aa,, )) InferenceInference

Scene and SensorScene and SensorPhysicsPhysics

Training DataTraining Data

ProcessingProcessing

ââ=T72=T720 50 100 150 200 250 300 350 400

7.4

7.6

7.8

8

8.2

8.4

8.6

8.8x 10

4

Azimuth (degrees)

Raw HRR Raw HRR

DataData

SAR ImageSAR Image LogLog--likelihoodlikelihood

functionfunction

77


ATR Theory and PerformanceATR Theory and Performance

•• ATR Systems of InterestATR Systems of Interest

•• Training and Testing ParadigmTraining and Testing Paradigm

•• Some System Design IssuesSome System Design Issues

•• Information Theory and ATRInformation Theory and ATR

•• System Implementation IssuesSystem Implementation Issues

•• ConclusionsConclusions

aa=T72=T72

SARSAR

PlatformPlatform

rr

TargetTarget



EstimatorEstimator

ââ=T72=T72

=45=45°°^

ModelModel

DatabaseDatabase

88


ModelModel--Free versus ModelFree versus Model--Based ApproachesBased Approaches

•• ModelModel--Based ApproachesBased Approaches

–– ConditionalConditional likelihoodslikelihoods for datafor data

derived from understanding physicsderived from understanding physics

•• ModelModel--Free ApproachesFree Approaches

–– Processing architecture fixedProcessing architecture fixed——

no model for data assumedno model for data assumed

–– Examples:Examples:

»» Neural networksNeural networks

•• Intermediate ApproachesIntermediate Approaches

–– Use models when knownUse models when known

–– Use constrained architectures for restUse constrained architectures for rest

»» MSE on logMSE on log--magnitudesmagnitudes

»» MSE on quarter powerMSE on quarter power

»» Most featureMost feature--based classifiersbased classifiers

rrp(r|a,p(r|a,

rr ffp(f|a,p(f|a,featurefeature

99


Performance Analysis and PredictionPerformance Analysis and Prediction

•• Clear problem statementClear problem statement

–– Hypothesis testingHypothesis testing

–– Estimation problemEstimation problem

•• Known distributionsKnown distributions

–– Information boundsInformation bounds

»» ChernoffChernoff, Rate functions, Rate functions

»» Fisher Information, CRLBFisher Information, CRLB

–– LaplaceLaplace approximationsapproximations

––Monte Carlo techniquesMonte Carlo techniques

•• Unknown distributionsUnknown distributions

––MinimaxMinimax boundsbounds

•• Partially known distributionsPartially known distributions

Achievable PerformanceAchievable Performance

InformationInformation--TheoreticTheoretic

BoundsBounds

1010


Issues in Function EstimationIssues in Function Estimation

•• Statistical Tradeoffs:Statistical Tradeoffs:

–– Approximation errorApproximation error——

Estimation errorEstimation error

–– BiasBias——VarianceVariance

–– OvertrainingOvertraining

•• Learning Theory BasisLearning Theory Basis

•• Current InformationCurrent Information--

Theoretic View:Theoretic View:

–– Complexity regularizationComplexity regularization

–– MDL BasisMDL Basis

Moulin, Yu, Barron, Moulin, Yu, Barron, RissanenRissanen

-- LLR(f) + LLR(f) + Complexity(f)Complexity(f)

0 1 2 3 4 5 6 7-6

-4

-2

0

2

4

6

8

10

12

14

Lo

g s

quare

d e

rro

r

Complexity: Log Dimension

Approximation and Estimation Error

Log integrated squared errorLog integrated squared error

ISE=App. Error + Est. ErrorISE=App. Error + Est. Error

Individual errors exponential Individual errors exponential

in dimension in dimension

1111


Regularization for Function EstimationRegularization for Function Estimation

Tikhonov

Grenander’s Sieves

Prior Likelihoods

Constraint Sets

Penalty Functionals

Complexity Regularization

..

.

fS

F

F1

F2

f2

f1

12

..

1212


Robust Conditionally Robust Conditionally GaussianGaussian ModelModel

J. A. OJ. A. O’’Sullivan and S. Jacobs, IEEESullivan and S. Jacobs, IEEE--AES 2000AES 2000

Model each pixel as complex Model each pixel as complex GaussianGaussian plus uncorrelated noise:plus uncorrelated noise:

i

NaK

r

i

Ai

i

eNaK

ap 0

2

,

0

,,

1,r

R

aBayes r argmaxa

maxkp r k ,a

ˆHS r,a argmax

k

p r k ,a

GLRT Classification and MAP Estimation:GLRT Classification and MAP Estimation:

J. A. OJ. A. O’’Sullivan, M. D. Sullivan, M. D. DeVoreDeVore, V. , V. KediaKedia, and M. Miller, IEEE, and M. Miller, IEEE--AES to appear 2000AES to appear 2000

1313


ConditionallyConditionally GaussianGaussian ModelModel

Model each pixel Model each pixel ii as independent, zero mean, complex conditionally as independent, zero mean, complex conditionally GaussianGaussian

pR ,A,C

2 r ,a,c2 1

c2 i2 ,a

e

ri2

c2 i2 ,a

i

Where:Where: ii22(( ,,aa) = variance function over pose and class) = variance function over pose and class

cc22 = constant over all pixels to account for power fluctuation = constant over all pixels to account for power fluctuation

a, ˆ, c2

argmaxa, ,c2

lnp r a, ,c2

p r 2i

Ii (a, )

Recognition by maximizing the logRecognition by maximizing the log--likelihood ratiolikelihood ratio**

Where:Where: 22 = average clutter variance= average clutter variance

IIii = mask function= mask function

**SchmidSchmid & O& O’’SullivanSullivan ““ThresholdingThresholding Method for Reduction of DimensionalityMethod for Reduction of Dimensionality””

1414


Normalized Conditionally Normalized Conditionally GaussianGaussianResultsResults

1515


PerformancePerformance--Complexity LegendComplexity Legend

Forty combinations of number of piecewise constant

intervals and training window width

1616


ATR Performance and ComplexityATR Performance and ComplexityComparison in terms of:

• Performance achievable at a given complexity

• Complexity required to achieve a given performance

Information Theory Basis: Rate-Distortion Theory

Rate-Recognition Theory

1717


Image Segmentation Image Segmentation Target ExtractionTarget Extraction

InformationInformation--Theoretic ApproachTheoretic Approach•• Hypothesis Test:Hypothesis Test:

–– pixels on target vs. on clutterpixels on target vs. on clutter

•• PixelwisePixelwise measure of information for discriminationmeasure of information for discrimination

D(pD(pii||p||p00))

ConditionallyConditionally GaussianGaussian

•• Segmentation ComplexitySegmentation Complexity

–– LikelihoodsLikelihoods on snakes (contours)on snakes (contours)

–– Complexity of regionComplexity of region

2

2

2

2

ln1ii rr

Top 5Top 5 Top 50Top 50 Top 100Top 100 Top 300Top 300

1818


System Design Issues:System Design Issues:

DynamicallyDynamically ReconfigurableReconfigurable AlgorithmsAlgorithms

•• Information Theory ContributionsInformation Theory Contributions

–– SuccessivelySuccessively RefinableRefinable ModelsModels

»» EffrosEffros, Cover and , Cover and EquitzEquitz,, RimoldiRimoldi

»» J. Shapiro, Said and J. Shapiro, Said and PearlmanPearlman

»» R.R. DeVoreDeVore, A. Cohen, , A. Cohen,

»» I.I. DaubechiesDaubechies, D. , D. DonohoDonoho,, ……

–– SuccessivelySuccessively RefinableRefinable RecognitionRecognition

»» RateRate--distortiondistortion RateRate--recognitionrecognition

»» LogLog--time, logtime, log--spacespace RateRate

1919


SuccessivelySuccessively--RefinableRefinable Sensor ModelsSensor Models

Consider decreasing interval Consider decreasing interval

widthswidths

dd11=2=2 ,, dd22== ,, ……,, ddmm=2=2 /2/2mm--11

••••••

Search over Search over kk in level in level ii ordered by the most likely pose at level ordered by the most likely pose at level ii--11

˜d,i2

k ,a1

di2,a d

2 kNd

d2

2 kNd

d2

Divide azimuth into Divide azimuth into NNdd nonnon--

overlapping intervals of width overlapping intervals of width dd

2020


FourFour--Class ExampleClass Example

Classification error as a function of number of Classification error as a function of number of

bits passed between the database and processorbits passed between the database and processor

•• Eventually, search covers all Eventually, search covers all

possibilitiespossibilities

•• BreadthBreadth--first search quickly first search quickly

finds good combinations of (finds good combinations of ( ,,aa))

•• Method for modeling target Method for modeling target

reflectivity statistics from sample reflectivity statistics from sample

imagesimages

•• Target models used to estimate Target models used to estimate

conditional sensor output conditional sensor output

statisticsstatistics

SuccessivelySuccessively--refinablerefinable sensor models yield successivelysensor models yield successively--refinablerefinable decisionsdecisions

2121


System Design IssuesSystem Design Issues

•• ATR PerformanceATR Performance

•• RefinableRefinable ComputationsComputations

•• ParallelizableParallelizable

•• System Resource System Resource

ConstraintsConstraints

Result Quality vs. Complexity

2222


ATR as a ATR as a ParallelizableParallelizable OperationOperation

•• MaximizingMaximizing ppRR|| ,,AA is equivalent to maximizing the logis equivalent to maximizing the log--

likelihood,likelihood, ll((r|r| ,,aa)) == kk ++ lnln ppRR|| ,,AA

l r ,a ln i2,a

ri2

i2,ai

•• Each measured value, Each measured value, rrii , undergoes operations of the , undergoes operations of the

same form for all pixels, orientations, and target classessame form for all pixels, orientations, and target classes

2323


ATR as a ATR as a ParallelizableParallelizable OperationOperation

ATRATR aa11rr1

••

••

••

aa22rr2 ATRATR

aamm

rrm ATRATR

aamaxmax

ll((rr|| 1,, aa1))^

maxmax ll((rr|| ,, aa1))

••

••

••

maxmax ll((rr|| ,, aa2))

maxmax ll((rr|| ,, aat))

ll((rr|| 2,, aa2))^

ll((rr|| t,, aat))^

••

••

••

maxmax

ll((rr||355355 ,,aa))

ll((rr||55 ,,aa))

ll((rr||00 ,,aa))

ll((rr|| ,,aa))^

rr

22(( ,, aa))

gg gg gg

gg gg gg

gg gg gg

•• •• ••

•• •• ••

•• •• ••

••

••

••

ll((rr|| ,, aa))••

••

••

••

••

••

2424


ATR IllustrationATR Illustration

•• QualityQuality -- Probability of erroneous classificationProbability of erroneous classification

•• ThroughputThroughput -- Target images processed per secondTarget images processed per second

•• ResourcesResources -- Processors, memory and I/O bandwidth, etc.Processors, memory and I/O bandwidth, etc.

aa=T72

SARSAR

PlatformPlatform

rr

TargetTarget



EstimatorEstimator

ââ=T72=T72

=45=45°°^

For classification/estimation components we relate:

2525


ExampleExampleT2=T1 with prefetch 16 KB/SAR Image (4B floats)

1 GHz clock M=10 targets

Varying target model complexity

(L templates/target and N pixels/template)

1 Gb/s Interconnection Network 10 Gb/s Interconnection Network

2626


ConclusionsConclusions

•• ATR Performance Bounds ATR Performance Bounds Problem StatementProblem Statement

–– Information Rate Functions for DetectionInformation Rate Functions for Detection

–– Fisher Information for EstimationFisher Information for Estimation

–– Approximation ErrorApproximation Error——Estimation ErrorEstimation Error

•• ModelModel--Based Approaches:Based Approaches:

Known DistributionsKnown Distributions

•• Successive RefinementSuccessive Refinement

•• Implementation ConsiderationsImplementation Considerations

2727


Factor InterrelationshipsFactor Interrelationships

•• ATR systems are explicitly or implicitly based on models of ATR systems are explicitly or implicitly based on models of

targets with some complexity targets with some complexity CC

•• More complex target models require more computation but can More complex target models require more computation but can

yield better results; Pr(error)=yield better results; Pr(error)=ff((CC,, SARSAR))

•• Target model complexity and computational power determine Target model complexity and computational power determine

overall system throughput; overall system throughput; TTCHIPCHIP==hh((CC,, COMPCOMP))

•• Given an architecture, both result Given an architecture, both result qualityquality, Pr(error), Pr(error),, andand

throughputthroughput,, RR=1/=1/TTCHIPCHIP, are parameterized by target model , are parameterized by target model

complexitycomplexity

2828


Quality of Results and ComplexityQuality of Results and Complexity

Model complexity Model complexity

resolution in resolution in

approximation of approximation of 22(( ,,aa))

Coarse model of aT62 tank,

1 template with 16K floats

Fine model of a T72 tank (1/5 relative scale),

72 templates totaling 1.1M floats

2929


MotivationMotivation

OutlineOutline

1.1. ConditionallyConditionally GaussianGaussian Model for SAR imageryModel for SAR imagery

2.2. Likelihood Based Approach to RecognitionLikelihood Based Approach to Recognition

3.3. Target Model Estimation & SegmentationTarget Model Estimation & Segmentation

4.4. SuccessivelySuccessively--RefinableRefinable Sensor ModelsSensor Models

5.5. ExampleExample

ATR from CAD ModelsATR from CAD Models Template Based ATRTemplate Based ATR Model ExtractionModel Extraction

Combine sensor output prediction with training dataCombine sensor output prediction with training data

3030


Target Model EstimationTarget Model EstimationGiven N registered training images qj of a target with pose j , estimate

variances over Nw windows of width d

wk2 k

Nw

d

2,2 k

Nw

d

2ˆ 2 k ,a

1nk

q j2

j: j wkwhere

Variance estimate for an Variance estimate for an

unregisteredunregistered image image rr with pose with pose

formed by transforming the formed by transforming the

estimate from the closest estimate from the closest wwkk

Registered Variance ImagesRegistered Variance Images

Transformed EstimatesTransformed Estimates

TT00°° TT9090°° TT180180°° TT270270°°

3131


Target Model SegmentationTarget Model Segmentation•• Pixel information relative to nullPixel information relative to null--hypothesis used for target recognitionhypothesis used for target recognition

•• Retain pixels Retain pixels ii that are informative relative to the nullthat are informative relative to the null--hypothesis:hypothesis:

Sa i :1

NwD p ; ˆ i

2k ,a p ;

2

k

Top 5Top 5 Top 50Top 50 Top 100Top 100 Top 300Top 300

•• Segmentation of target models, not of imagesSegmentation of target models, not of images

•• Ordering of pixels by their empirical information relative to nuOrdering of pixels by their empirical information relative to nullll--hypothesis.hypothesis.

For nullFor null--hypothesishypothesis 22=0.0028=0.0028 -- approximate background variance approximate background variance -- pixels on illuminated pixels on illuminated

side of target are deemed most informative.side of target are deemed most informative.

3232


Computational ModelsComputational Models

Chip processing rate Chip processing rate RR=1/=1/TTCHIPCHIP

Assumptions:Assumptions:

•• Each CPU optimizes over a region of the search spaceEach CPU optimizes over a region of the search space

•• MultiMulti--issue CPU with 2 instructions/clock cycleissue CPU with 2 instructions/clock cycle

•• 6 instructions per pixel6 instructions per pixel

TCHIP sec/SAR Image L templates/target

T1 sec/clock cycle M targets

T2 sec/template memory read N pixels/template

T3 sec/SAR Image load P processors

TCHIP 3LMN

PT1

LMN

PT2 T3

Documents

ATR Theory and ATR Performance Analysis and Predictionjao/Talks/InvitedTalks/SPIE...ATR Theory and ATR Performance Analysis and Prediction Joseph A. O’Sullivan Electronic Systems