Status of AiTR/ATRin

Military Applications

James A. RatchesCERDEC NVESD

January 2007

UMDAROATR

Outline

• Definition • Importance & Scenarios• Performance Assessment• Problem Statement• Way Forward• Summary & Conclusions

• Generic term to describe automated/semi-automated functions carried out on imaging sensor data to perform operations ranging from cuing a human observer to complex fully autonomous object acquisition and identification• Machine function: - Detection - Classification - Recognition - Identification - Friend or Foe

• Aided Target Recognition (AiTR) - Machine makes some level of decision and annotates the image - Human makes higher level decision. e.g. to identify and fire

• ATR is fully autonomous - No human in-the-loop after weapon firing, e.g. fire-and-forget seeker

• ATR/AiTR may use information from other sensors to make decision by fusing information

AiTR (aided) ATR (autonomous)

Military Definition

Scenarios Where AiTR Essential

Urban terrain; 360 degree situationalawareness, short ranges, human intent,

transmission limitations

Rapid wide area search for close combat in high clutter, against difficult targets

(occlusion, defilade, CC&D) and variabletarget signatures

UAV & UGV transmissionsover limited bandwidthBDA

1830

road

road

Tree trunks

debris

New object1030 Detection of

Dismounts & intent,& bunkers

Scouts in Overwatch-Objects of interest and scene changes

Missile Scenarios Where ATR Essential

Fire UnitsFire UnitsFire UnitsFire Units

1st1stWaypointWaypoint

(Tower)(Tower)

1st1stWaypointWaypoint

(Tower)(Tower)

Field Of View

Engagement

EngagementAreaArea

1 K

m

- Power Up

- Computer Initialization

- Intelligence Preparation of

Battlefield

- Plan Missile Routes

if necessary

-Receive Target

Information

Through C2 Network

- Verify Target Selection

- Route & Salvo Selection-Launch Missile(s)

- In-Flight Intelligence

- Target Marking

- Target Reporting

to C2 Network

- Start Search (Wide FOV)

- Locate Target (Narrow FOV)

- Lock On

- Aimpoint Update

Warhead Function

On Impact

4 Km

Navigate To

Emplacement Site

Missile Auto-Navigate To

Target Search Point(Enroute Recon)

Detect, Recognize &

Identify Target(Engage Autotracker)

ObstacleObstacle 2ndWaypoint(Mountain)

2ndWaypoint(Mountain)

Target ofOpportunity

Target ofOpportunity

-Acquire GPS Satellites

-Update GPS Position

-Calibrate the Inertial System

- Navigate to Target Area

Which pixels in image correspond to targets?

AiTR Annotates Images – Not Maps

Hunter Ligget

Yuma

Grayling

Clutter levels:

High Hunter Ligget

Medium Yuma

Low Grayling

Effects of Clutter

WIDE AREA TARGET CUEING WITHIN 4 SECONDSCTRS

+

H MD VIS

H MD M ODE

H MD W P

NAV U PDT

NA V WP

NAV APLT

SCRN BRT

+

+

29

N

S E

W

NB 5200 2250NB 2320 445 6TFXY

AB1

TS D HO ME

T SD SCLE

TSD TAC

TSD NAV

TSD CNTR

T SD W NDW

SLAVE

Manual FLIR (300 FOR) Search Time > 60 Sec.

Aided Target Search Time less than 4 Sec.

ROC Curves

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

False Alarms per Square Degree

Pro

bab

ilit

y o

f D

etec

tio

n

Algo 1 - In Open

Algo 1 - Occluded

Algo 2 - In Open

Algo 2 - Occluded

Effects of Occlusion

Lab/Field Measured Performance

COMMON SENSORCOMMON SENSORLong Range Scout Surveillance SystemLong Range Scout Surveillance System22ndnd Gen FLIR Modified f/ Gimbaled Scan Gen FLIR Modified f/ Gimbaled Scan

AiTD/R•Assess Maturity Ground Based AiTD/Rs in Varied Environmental Conditions

Gimbaled Scan FLIR

•Long Range Target Detection• 2nd Gen B-Kit (LWIR)

SWIR CAMERA•Long Range Target Identification•Leverage ACT II LIVAR and CETS Program - EBCCD Technology •4.5” Aperture

MTI Radar•Utilize AN/PPS-5D

Laser Illumination/Designation

AiTR/ATR Continues to Be Tested in Realistic Environments

Target Acquisition Sensor Suite (TASS)

• Assess maturity of SOA AiTR in gimbal scanning mode in the field • SOA single color/ shape LWIR based algorithms from multiple sources• Include urban bkgds and man targets

Evaluations yieldROC curves

Overall Assessment

1. DOD investment in AiTR has resulted in quantifiable level of performance documented in ROC curves2. Performance measured under favorable conditions3. Order of magnitude improvement in search time with AiTR over human only4. Discrimination levels above detection have not been vigorously pursued*5. Detection performance can have degradation for sub- optimal conditions*

- high clutter - low contrast - obscuration - extended ranges

6. Training target sets have been typically for < 10 targets7. There are no human detection algorithms

SOA AiTR Algorithms Have Known Limitations

* Especially for ground-to-ground

Need for Robust AiTR/ATR

For future combat scenario must be robust - High false alarm rate renders aid useless and operator

will turn it off (AiTR)- Ground-to-ground presents high clutter - Target variability increases complexity- Low signature targets can be expected- Partial occlusion & defilade obscures the target - CC&D need to be mitigated- Detect human threats in urban terrain- Final ID can be man-in-the-loop (AiTR)

Robust AiTR/ATR Critical for Ground-to-GroundClose Fight Manned-shoot first Unmanned-autonomous

operation

The AiTR/ATR Problem

• ~$100M investment to realize SOA AiTR• Humans can still do better than SOA AiTR (Except for speed)

• Robust AiTR required - Potential target set is large with wide range of environmental and operational variations -AiTR for humans and urban terrain

• New university concepts have not migrated to industry and military developers

ARL-SEDDDATA

AiTR/ATR Cannot Do As Well As The Human

Alone-However, It Can Do It Faster

-Improvement that approaches human performance will be an enabling force multiplier

00.10.20.30.40.50.60.70.80.9

1

0 20 40 60Search Time (sec)

Aided

Manual

Pro

bab

i lit y

of

Det

e ct i

on

Perceived Impediments to ATR/AiTR

• Required computational power• High cost, power and size• Proprietary issues• Tactical scene complexity• Required to be better than human• New CONOPS will be needed to fully utilize benefits of ATR/AiTR

Real Limitation Is The Lack of An Image Science-What Is Important in An Image?

Possible Paths to Improvement

• 3D LADAR – if can cover/search field of regard- Otherwise, use for higher level discrimination

• Multi/hyper-spectral/look/mode sensor and Sensor Fusion• Untried University “New Ideas”

- Recognition by parts- Advanced eye-brain understanding- Gradient index flow and active contour analysis- Frame-to-frame correlations- Spatial contextual intelligence- Hierarchical imaging - Category theory

• Off-board sensor features data via low bandwidth tactical networks• Validated synthetic image generation to stress algorithms during formulation• Investment in Image Science

ARO/Duke University WorkshopComputational Sensors for Target Acquisition and Tracking

Beaufort, NC December 2-4, 2003

Representative Recommendations

• Different approach to applying eye-brain understanding to AiTR needed - Does not necessarily mean that we need to mimic that process

• Artists may give a unique insight into minimalist representation• Poor performance of AiTRs relative to humans suggest there are better features than have been found by AiTRs• The perspective of clutter rejection rather that object feature extraction may present a different set of opportunities

Eye-Brain Understanding Can Still Be A Fertile Groundof Investigation for AiTR Concepts

1. Statistical a. no range ΔT, size, perimeter, etc. Comanche - target in open & in center of FOV

- ~ 10 target set in low clutter - baseline performance

b. w/range same + target window size - reduce search time (10X) - reduce FAR (10X)

2. Template Matching comparison of ROI to stored SAIP - expand target set, e.g. aspect,target templates articulation, dirurnal/seasonal, etc.

2. Model Based comparison of ROI to stored MSTAR - increase target set with stored datatarget model set reduction

4. Multi-spectral pixel value=f(λ,Δλ) MFS3 - penetrate camouflage - reduce FAR

5. Multi-look target indications at GPS Dynamic - reduce FAR (~ 10-100X) by coords from off-board Variable by correlating target detectssensors Threshold - detect obscured, defilade targets

- missed target reduction (~2X)

6. Multi-mode non-imaging sensor indications ASM - mitigate CC&D(sound, vibration, magnetics) algorithms - reduce FAR

ALGORITHM FEATURE EXAMPLE CAPABILITY METRIC

*

*

*

*

*

*

* Classified data on false alarms and Pd exist for these algorithms

Progression of Algorithms

ALGORITHM FEATURE EXAMPLE CAPABILITY METRIC

7. Geographic Contour Maps terrain slope DTED - FAR reduction (potential ~ 75%)

8. Advanced Eye-Brain synapse maps NN, holographic NN, - intelligent search & detect Understanding & wavelets - FAR reduction Representation - reduce search timelines

9. Recognition-by-Parts target subelements - detect partially obscured targets detected - missed target reduction

10. Gradient Index Flow & 2D chips of humans - determine human intent Active Contour Analysis

11. Frame-to-Frame pixel change MTI - detect changes in scene Correlations correlations - reduce search times

12. Spatial Contextual target forbidden terrain - reduce search time Intelligence by reducing search area

13. Artists Insights hierarchical scene - reduce search time by characteristics focus on search area

14. Hierarchical Imaging activate/retard signals - bandwidth reduction by evaluating information before transmission

15. Category Theory sensor report & - geolocation accuracy locations improvement

Sf

S1

S2

ScSw

*

Progression of Algorithms (con’d)

• Different features have different ROC curves

– Range dependent

• Features from different sensors and platform can be passed over the network (low bandwidth information)

• Performance gain proportional to ROC curves

• Pick 2 features as example– local variation– wavelet

10X FAR reduction

Theoretical Basis for Multi-Look

00

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1ROC Curves for 2100m, 2500m, and Fused

Pd

Pfa

Fused Best: Pd Indep., Pfa Indep.

Fused: Pd Indep., Pfa Fully Correl.

Fused Worst: Pd, Pfa Fully Correl.2100m

2500m

Features from Off-Board Sensors Can Improve On-BoardSensor AiTR/ATR

R1

R3R2

V2

R4 R2

V1V2

False Alarms Uncorrelated between Sensors Ridge Sensor

Valley Sensor

Side View

Most false alarms

for LWIR-2Most false alarms

for LWIR-1

LWIR-1

LWIR-2

Plan ViewMost false alarms

for LWIR-1Most false alarms

for LWIR-2

LWIR-1LWIR-2

Category Theory is a mathematically sound framework:-Designed for network applications-Describes information fusion systems and processes in an elegant language - Captures commonality and relationships between objects

Specifications S: S1. S2 data from sensors 1 & 2 Sc real world stimuli Sw ground truth Sf registration transformation between S1 & S2Morphisms: arrowsFunctors: Relationships with other categories

Sf

S1 S2Sw

ScExample of a

category

Gunbarrel

Hot spot

Turretgeon1

geon2

geon3

Tracks & wheelsgeon4

Engine exhaust

geon5

FLIR Image

T-72 tank

Recognition is basedon recognition of critical sub-componentscalled geons

Biderman (USC)Kokar (Northeastern)

Recognition-by-Parts Category Theory

Library of Geons for targets of interestforms the basis for recognition

Network supplies opportunity for sophisticatedfusion techniques to be applied to AiTR

O1 O2 O3 O4a b c

cxaxc=(cxb)xa=cx(bxa)Composition operation that is associative

“Image Science” Based Algorithms

SOA algorithms attempt to recognize static targets in single frames: Need to consider more image-based, e.g.

parameters e.g., image temporal-spatial relationships.

Sensor-Scene Dynamics

Context

Algorithms Must Extract More Contextual Information

Change detection & MTI

Gradient of intensity (x, y)

Gradient Vector Flow (GVF)

• Higher level process or user initializes any curve close to the the object boundary (indication of a region of interest)• The parametric curves (snakes) then starts deforming and moving towards the desired object boundary• In the end it completely “shrink-wraps” around the object

Zucker (Yale) & Xu & Prince (JHU)

(Active Contour Analysis)

Eye-Brain Understanding Must Be Applied Faithfully

GVF field is defined to be a vector field X [x(s), y(s)]for s in [0,1]Solve Euler equation αx''(s) - β x''''(s) - Eext = 0to minimize energy functional E = ∫0

1 ½ (α│x'(s)│2 -β│x''(s)│2) + Eext (x(s))ds (α and β user defined constants)

Human intent

This painting shows how Van Gogh was able to transmit detailed Information about a person (20-year old woman) to the viewer Using Only ~10 brush strokes for her face.

Artists Unique Insight

From Falco (U of AZ)

Hierarchical Imaging &Target Representation

Elements of Network Make Localized Decisions Rather ThanSimply Sending Raw Data to A Central Processor

• Sensors Sample n Parameters• Network becomes large scale sensor• Hierarchical decisions - Local decisions determine relevant information - Global decisions develop global model - Each node is a virtual point detector at the next level - Algorithms determine what is to be shared/when

The Network Becomes The Sensor & AiTR

Conclusions

• SOA ATR/AiTR has attained a level of performance that has some level of military value

- Targets in the open- Low to medium clutter- Target set ~ 10-15- No obscuration or camouflage- No humans or human intent- No high value targets, e.g. bunkers

• Major new innovations are needed to get a leap ahead in performance under operational environments

- New university concepts- Network information

Back Up Slides

Aided Target Recognition for Intelligent Search

M35 @ 0°, scale = 1.0

ZSU @ 165°, scale = 0.85

M60 @ 180°, scale = 0.85

M35 @ 270°, scale = 1.0

M35 @ 195°, scale = 1.0

Prescreener FeatureExtraction Registration Recognition

NeuralNet

Inner

outer

M-35

Range x

Sensor

Representative Configuration of SOA AiTR

3D from Optical Flow

Crater

Above Surface Mound

• Subtle motion provides substantial depth information• Memory/ processing advances permit harvesting of depth information (Target/Sensor motion)• Algorithms have been developed that amplify motion vectors and present them in a binocular display in real time to create “hyperstereo” using advances in microlens technologies

Courtesy of FOR 3D, Santa Rosa, CA

Processing Motion Information Can Provide Depth (Range)

Passive Ranging with DTED Data

Lines of constant range superimposed on FLIR image where earth is flat at Yuma P.G.

DTED Data

Lines of constant rangesuperimposed on FLIR

image at Hunter-Liggett

Flat Earth Approximations Cannot Be Used for All ScenariosOf Interest

Work on passive ranging and imagery by Raytheon.

Passive Ranging• Accurate range estimation can reduce false alarm rates in AiTR• Permits estimation of target size• Active ranging potentially reveals position• Most AiTR algorithms make a flat earth approximation• Passive ranging may provide more adequate accuracy

“ Despite often heard claims to the contrary, without range data there is no wayof knowing if the target is 100- times smaller than a pixel or 1000 times largerthan the image as a whole. – Northrop-Grumman

Optical flow

Eye motion Near fieldobjects Far field

objects

DTED w/GPS

DTEDoverlaid

on imagery

GPS

El & Az of gun known

Lines ofconstant

range